p2p: wait for listener goroutines on shutdown (#20569)

* p2p: wait for goroutine exit, fixes #20558

* p2p: wait for all slots on exit

Co-authored-by: Martin Holst Swende <martin@swende.se>