When closing down a connection, remove its poll ID.