* cmp to reference db if --reference_chaindata provided

* graceful shutdown

* add ./cmd/integration to makefile

* add interruption support for different stages

* don't compare errors directly

* Fix storageRangeAt

* Fix formatting

* Fix lint

* Fix lint

* Prevent racing by closing db after the blockchain

Integration add checkChangeSet into state_stages command (#728)

* after concurrency testing

* fixes for linters

* add an extra parameter

* change init order for protocolManager

* green tests

* Different chain generation

* More fixes

* Fix TestSideLogRebirth

* Fix the rest of core tests

* Fix accounts/abi/bind

* Fix compile errors for cmd/pics

* Fix consensus/clique tests

* More fixes

* Fix core/state tests

* Fix eth tests

* Fix eth/downloader tests

* Fix more tests

* Fix compile errors

* Some more fixes

* Save

* Exists

* Fix last tests

* Clean up

* Fix formatting

* Fix linter

* Fix formatting

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods

Add commands "compare_states", add command "stage9", fix unwind of stage9 (it walked over whole db) (#722)

* ./cmd/integration compare_states, ./cmd/integration stage9

* TxLookup fix unwind - it walked over all db

* move stages from hack.go to ./cmd/integration

* move stages from hack.go to ./cmd/integration

```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```

* fix case: if header stage is ehead of body stage

* fix case: if header stage is ehead of body stage

* fix case: if header stage is ehead of body stage

* allow stage3 to sync to given block

* allow stage3 to sync to given block

* Fixing history generation

* Try fixes

* Reset from 0

* Cleanup

* cmd for integration tests

* cmd for integration tests

Experiments: how much will take state in DupSort feature and how fast can delete large DupSort key (#709)

* Cherry-picked

* Reset3

* Fix formatting

* Print before commit

* Use ETL

* Fix linter

* Use different keys for ETL

* Fix formatting

* Update progress correctly

* Reset stage5

* Fix stage3 hack

* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test

* Kludge to remove duplicates

* Fix

* Fix formatting

* Fix in Len function

* removed unused actions in stage 2

* fixed stage 2

This reverts commit 63c998c9.

# Conflicts:
#	cmd/hack/hack.go
#	eth/stagedsync/stage_interhashes.go

* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup

* integration of insertBodies instead of insertChain

* remove seal verification from stage2

* fixed seals

* AbstractKV: defer friendly rollback

* AbstractKV: defer friendly rollback

* cursor which returns only useful IH

* cursor which returns only useful IH

* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter

* TestWatchNoDir not to be parallel

* Print ks and inc

* Print addrHash

* Change the buffer

* Print loading

* Skip

* More logging

* Error out earlier

* Handle empty codes

* Handle empty codes

* Remove logging

* Compare states

* Not do stage5

* compareBucket

* Preimage

* Clearer errors

* No need to clean up contract code

* Restore stage6

* Printing

* Skip the skipping

* Print all

* Change buffer type

* Add limit to stage5

* Always fail

* Remove exessive logging

* Restore buffer type

* Revert

* Print when exception

* Reenable skipping

* Skip storage items for deleted contracts

* not shortcut

* Remove removeAccount

* Re-enable state hashing

* Default to plain state

* Disable hashing state

* Reenable reset5

* Print unfurl list

* Enable removingAccount

* No printing

* Reenable stage5 commit

* Swap order of stages

* Prevent backwards promotion, reset tx lookup

* reset finish

* Introduce storage item replacement

* See if unwind works

* Restore removingAccount

* Don't do removeAccount for unwinding

* Possible fix

* Proper(er) fix

* Don't exclude unwinding

* Remove unwinding flag

* Fix formatting

* Fix lint

* Not to ignore blocks if they cause reorg

* Fix test, separate stages again

* Fix TestUnwind

* Fix stage

* Swap unwinding

* Revert to unwinding flag

* Print unfurl list

* Print

* Print inside receive

* Print after

* No printing

* Cleanup

* Not use blockCache when doing GetBlock

* reduce amount of c.Next calls

* add timings of gen IH to logs