Keep our listen points in our uplinks list, flagged, so we don't connect to them.