Don't free mantissa buffer when a NUM has taken it over.