Switch from private VM to an AVL tree.

We don't need private VM except for larger problems (W=7 and up), and,
for them, the nodefor[] array is very sparse in practice.  So, instead
of grabbing the VM for a 64-gigaentry table of 8-bit objects (half a
terabyte of memory) only to use only few thousand entries of it, use an
AVL tree, keyed by what would be the array indices, instead.