This adds support for a new struct tag "optional". Using this tag, structs used
for RLP encoding/decoding can be extended in a backwards-compatible way,
by adding new fields at the end.

see geth commit https://github.com/ethereum/go-ethereum/commit/700df1442d714cb3c42a602c39c042ce88be463f



Co-authored-by: Felix Lange <fjl@twurst.com>

* done

* removed sleep

* f

* errors.Is(err, io.EOF) instead of err == io.EOF

Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>

* save

* log

* save

* Clean up test runners. Don't run legacy tests

* Cherry pick https://github.com/ethereum/go-ethereum/pull/22927



* Tests update 10.1: Transaction Tests

* Port decodeBigInt changes to decodeUint256

* Introduce (*Stream) Uint256Bytes

* Temporarily disable stTransactionTest/HighGasPrice

* linter

* ttWrongRLP transaction tests pass now

* Fix stTransactionTest/HighGasPrice

Co-authored-by: Felix Lange <fjl@twurst.com>

* begin 2021.10.2 release cycle

* Revert "rpcdaemon: (#2752)" (#2762)

This reverts commit 2afd028d.

* Pool v2: --txpool.pricelimit support (#2763)

* --txpoo.pricelimit support

* Pool v2: --txpool.accountslots flag support (#2765)

* Update signal_windows.go (#2767)

Trap os.interrupt instead of SIGINT and SIGTERM

* Update stage_finish.go : notifications to rpc daemon (#2755)

* Dockerfile: switch to go1.17 and alpine3.14 (#2766)

* add logs in recoverFromDb func (#2769)

* eip 1559 in miner (#2773)

* Inner errors (#2774)

* Clean up DEBUG category logs (#2776)

- move many DEBUG logs into TRACE category

* Decoding incarnation implemented (#2764)

* WIP decoding incarnation specifically

* Changed decodeIncarnation to be an external function

* added tests to for decoding incarnations

* ran gofmt -w -s

* changed test name, and changed incarnations to 4

* Created a test which tests if it returns an error when there is one

* ran gofmt

* Capitalized all tests and made breaking test

* added an error check

* changed decodingForStorage for decodingIncarnationFromStorage

* ran gofmt -w -s

* No senders is fine (#2775)

* IntermediateHash stage - switch from incremental to re-generate mode - if jump > 100K blocks (#2781)

* Enable "State stream" by default (#2780)

* No json rpc streaming (#2779)

* reduce_bach_concurrency_default

* RPCDaemon: reduce --rpc.batch.concurrency default from 50 to 2 (#2784)

* Integration to print right stage in logs (#2785)

* remove debug prints

* RemoteDB: don't spend time to close cursors on end of tx - server will cleanup everything well (#2786)

* Fermion genesis block (#2787)

* updated Fermion genesis block

* Updated Fermion genesis block: added precompiles

* Rpcdaemon: add  --tevm flag to enable experiment (#2788)

* Reworkings of state compression experiments (#2790)

* Changes

* Progress

* Another way

* More

* More

* Produce encoding

* Add uncoded characters

* cleanup

* Add sortdict

* Fixes

* Use patricia from erigon-lib

* Cleanup

* Switch to dynamic programming, optimise allocations in FindMatches

* Optimise allocations

* Reduce allocations

* Switch to main branch of erigon-lib, reduce allocations further

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>

* Ignore MaxPeers param for staticpeers (#2789)

Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>

* less warnings

* Update skip_analysis.go (#2792)

* Extend preverified hashes for mainnet and ropsten (#2793)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: Andrea Lanfranchi <andrea.lanfranchi@gmail.com>
Co-authored-by: Enrique Jose Avila Asapche <eavilaasapche@gmail.com>
Co-authored-by: e-danko <89912851+e-danko@users.noreply.github.com>
Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexandr Borodulin <sashaborodulin@gmail.com>
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>

Pool: switch cache to tx.ID(), add blockNum to tx info to drop from full pool old txs first  (#2688)

* Try to solve sentry hanging problem + p2p impovement from geth

* Fix import

* Finer stream lock

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* eip 1559_tx_constraints

* eip 1559_tx_constraints

* return error on uint256 overflow

* up uint256 version

* merge master

* fix master

Co-authored-by: ledgerwatch <akhounov@gmail.com>

* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"

* Where I am at

* Refactoring of transaction types

* More refactoring

* Use Homested signer in rpc daemon

* Unified signer

* Continue unified signer

* A bit more

* Fixes and down the rabbit hole...

* More tx pool fixes

* More refactoring fixes

* More fixes'

* more fixes

* More fixes

* More compile fixes

* More RLP hand-writing

* Finish RLP encoding/decoding of transactions

* Fixes to header encoding, start on protocol packets

* Transaction decoding

* Use DecodeTransaction function

* Decoding BlockBodyPacket

* Encode and decode for pool txs

* Start fixing tests

* Introduce SigningHash

* Fixes to SignHash

* RLP encoding fixes

* Fixes for encoding/decoding

* More test fixes

* Fix more tests

* More test fixes

* More test fixes

* Fix core tests

* More fixes for signer

* Fix for tx

* Fixes to string encoding/size

* Fix eip2930 test

* Fix rest of ./tests

* More fixes

* Fix compilation

* More test fixes

* More test fixes

* Test fixes

* More fixes

* Reuse EncodingSize in EncodeRLP for accessList

* Rearrange things in dynamic fee tx

* Add MarshalBinary

* More fixes

* Make V,R,S non-pointers

* More NPE fixes

* More fixes

* Receipt fixes

* Fix core/types

* Fix ./eth

* More compile fixes for tests

* More test fixes

* More test fixes

* Try to see lint errors better

* Try to see lint errors better

* Fix lint

* Debugging eip1559 test

* Fix TestEIP1559Transition test

* Fix NewBlockPacket encoding/decoding

* Fix calculation of TxHash

* Fix perf problem with senders

* Update aleut config values

* Try adding static peers

* Add staticpeers to defaul flags

* Change aleut networkID

* Fix test

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

io.Reader may return n > 0 and io.EOF at the end of the input stream.
readFull did not handle this correctly, looking only at the error. This fixes
it to check for n == len(buf) as well.

Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
# Conflicts:
#	cmd/utils/flags.go
#	console/bridge.go
#	crypto/bls12381/g2.go
#	les/benchmark.go
#	les/lespay/server/balance.go
#	les/lespay/server/balance_tracker.go
#	les/lespay/server/prioritypool.go
#	les/odr_requests.go
#	les/serverpool.go
#	les/serverpool_test.go
#	p2p/nodestate/nodestate_test.go
#	trie/committer.go

This PR contains a minor optimization in derivesha, by exposing the RLP
int-encoding and making use of it to write integers directly to a
buffer (an RLP integer is known to never require more than 9 bytes
total). rlp.AppendUint64 might be useful in other places too.

The code assumes, just as before, that the hasher (a trie) will copy the
key internally, which it does when doing keybytesToHex(key).

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	core/types/derive_sha.go

This can be useful when working with raw RLP data.

This change further improves the performance of RLP encoding by removing
allocations for big.Int and [...]byte types. I have added a new benchmark
that measures RLP encoding of types.Block to verify that performance is
improved.
# Conflicts:
#	core/types/block_test.go
#	rlp/encode.go
#	rlp/encode_test.go

List headers made up 11% of all allocations during sync. This change
removes most of those allocations by keeping the list header values
cached in the encoder buffer instead. Since encoder buffers are pooled,
list headers are no longer allocated in the common case where an
encoder buffer is available for reuse.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	rlp/encode.go

* no pools

* Remove excessive logging

* Revert

* Initial commit

* Cleanup

* Fix some compile errors

* Fix core tests

* Fix more tests

* Fix eth/downloader

* go mod tidy

* Fix core/vm

* Fix formatting

* Fix formatting

* Fix lint

* Fix lint

* Switch tests to StagedSync

* Cleanup

* Reuse cache in stage4

* Fix

* Fix formatting

* Try to fix test

* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray

* Misc GC optimisations

* unsafe experiment with writeByteArray

* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray

* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	consensus/clique/snapshot_test.go
#	core/bench_test.go
#	core/block_validator_test.go
#	core/blockchain.go
#	core/blockchain_test.go
#	core/chain_makers_test.go
#	core/dao_test.go
#	core/rawdb/accessors_indexes.go
#	core/rawdb/schema.go
#	eth/config.go
#	eth/helper_test.go
#	eth/sync.go
#	light/odr_test.go
#	light/trie_test.go
#	light/txpool_test.go
#	miner/worker_test.go
#	tests/block_test_util.go

* rlp: fix staticcheck warnings

* rlp: fix ExampleDecode test

* build: use golangci-lint

This changes build/ci.go to download and run golangci-lint instead
of gometalinter.

* core/state: fix unnecessary conversion

* p2p/simulations: fix lock copying (found by go vet)

* signer/core: fix unnecessary conversions

* crypto/ecies: remove unused function cmpPublic

* core/rawdb: remove unused function print

* core/state: remove unused function xTestFuzzCutter

* core/vm: disable TestWriteExpectedValues in a different way

* core/forkid: remove unused function checksum

* les: remove unused type proofsData

* cmd/utils: remove unused functions prefixedNames, prefixFor

* crypto/bn256: run goimports

* p2p/nat: fix goimports lint issue

* cmd/clef: avoid using unkeyed struct fields

* les: cancel context in testRequest

* rlp: delete unreachable code

* core: gofmt

* internal/build: simplify DownloadFile for Go 1.11 compatibility

* build: remove go test --short flag

* .travis.yml: disable build cache

* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement

* .golangci.yml: enable goconst and ineffassign linters

* build: print message when there are no lint issues

* internal/build: refactor download a bit

* rlp: improve nil pointer handling

In both encoder and decoder, the rules for encoding nil pointers were a
bit hard to understand, and didn't leave much choice. Since RLP allows
two empty values (empty list, empty string), any protocol built on RLP
must choose either of these values to represent the null value in a
certain context.

This change adds choice in the form of two new struct tags, "nilString"
and "nilList". These can be used to specify how a nil pointer value is
encoded. The "nil" tag still exists, but its implementation is now
explicit and defines exactly how nil pointers are handled in a single
place.

Another important change in this commit is how nil pointers and the
Encoder interface interact. The EncodeRLP method was previously called
even on nil values, which was supposed to give users a choice of how
their value would be handled when nil. It turns out this is a stupid
idea. If you create a network protocol containing an object defined in
another package, it's better to be able to say that the object should be
a list or string when nil in the definition of the protocol message
rather than defining the encoding of nil on the object itself.

As of this commit, the encoding rules for pointers now take precedence
over the Encoder interface rule. I think the "nil" tag will work fine
for most cases. For special kinds of objects which are a struct in Go
but strings in RLP, code using the object can specify the desired
encoding of nil using the "nilString" and "nilList" tags.

* rlp: propagate struct field type errors

If a struct contained fields of undecodable type, the encoder and
decoder would panic instead of returning an error. Fix this by
propagating type errors in makeStruct{Writer,Decoder} and add a test.

These changes fix two corner cases related to internal handling of types
in package rlp: The "tail" struct tag can only be applied to the last field.
The check for this was wrong and didn't allow for private fields after the
field with the tag. Unsupported types (e.g. structs containing int) which
implement either the Encoder or Decoder interface but not both 
couldn't be encoded/decoded.

Also fixes #19367

Prevents reallocation, improves performance

Issue found by @guidovranken