Handle time more correctly for decaying average.