For quite some time now (since sometime in 2003), I've had my border router keeping a blacklist of IPs I don't want to receive anything at all from. There are many reasons an IP can go on this list, such as trying to speak NetBIOS to me. When an IP goes on the list, conceptually, an expiration timer is started; when the timer expires, the IP is removed from the list. But if I receive anything from that IP during the timeout, the timer is restarted, so a host that keeps trying to send to me even though it's blocked, with an inter-packet interval shorter than the timeout, can stay blocked for as long as it's willing to keep trying. When I first implemented this, the timeout was 24 hours. Of course, I log everything, except that, after an IP goes on the list, or after its timer is restarted, there's a one-hour holddown during which I don't log timer restarts for that IP, otherwise my logs would get flooded by attacks that don't stop immediately upon blocking. (Of course, there are administrative interfaces I can use to manually list or delist IPs.)
In November 2020, I added code to keep track of how many packets got dropped due to each entry. On April 1st 2022, I raised the timeout from one day to one week (which, interestingly enough, didn't multiply the typical list size by seven; it rose from about 3000 to about 15000). All this has been puttering along just fine.
But I recently had occasion to look at those log entries for unrelated reasons and I noticed that, for the time sample I was looking at (about a day's data), all the delisted IPs were showing zero packets blocked. So I went through back data.
Turns out the rate of delisting (and, since the list size has to a first approximation been constant, listing) has been relatively stable, rising from about 1200 per day at the beginning of June to about 1500 per day now (late July). The interesting thing appears when I look at the rate of delistings showing nonzero blocked-packet counts.
That rate was cruising along at only a little below the total delisting count, right up until 2022-06-21 21:56:33 UTC. At that point they stopped, but that is meaningless, because at that point I lost the whole list (part of an internal software change that rendered the existing list unusable). It took one expiration interval, that is, a week, for it to build itself back up. Those expirations started again at 2022-06-29 00:07:19 UTC; they continued through 2022-06-30 00:08:25 UTC, at which point they stopped, staying stopped for most of July, specifically, until 2022-07-26 00:51:11.
I have nothing but speculation as to what was behind this odd behaviour. The list is still cruising at 17000-18000 IPs; based on looking at solely a count of blacklisted IPs, the peculiar gap from 06-30 to 07-26 is not visible.
If it weren't for the day or so of expirations on 2022-06-29, and the return of nonzero blocked-packet expirations on 2022-07-25, I'd speculate that the software change of 2022-06-21 broke something.