Factor out destroying a block - and free up the input and output vectors more thoroughly