The net package is the next to move. It will be massaged a bit still to fix the Network / "NetworkBackend" conflict.
The crypto package moves into p2p. Nothing in it so far is ipfs specific; everything is p2p-general.
I think it's time to move a lot of the peer-to-peer networking but-not-ipfs-specific things into its own package: p2p. This could in the future be split off into its own library. The first thing to go is the peer.
Juan Batiz-Benet