vm: fix basic VM tests

The tests now compile and won't panic for unexpected return values.

We need a recent-enough version of the mutan compiler because of
the new JUMPDEST requirements. Skip some tests if the installed mutan
version is too old. The debug VM test still fails, probably because
of an implementation bug.