status-im/nimbus-eth1
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth1.git synced 2025-01-13 05:44:40 +00:00
Code Issues Projects Releases Wiki Activity

Remove old portal_debug API for seeding data into the network (#2726)

Browse Source 
This was the first API created for this, it has been superseded
by the API that makes use of era1 files and/or the portal_bridge.

Only usage was still in test_portal_testnet which has now been
altered to make use if API calls from Portal specification.
This commit is contained in:
Kim De Mey 2024-10-10 16:42:57 +02:00 committed by GitHub
parent 59dde39d95
commit a5f0b12bd1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 34 additions and 324 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

86
fluffy/docs/the_fluffy_book/docs/history-content-bridging.md
View File

@ -2,7 +2,11 @@
## Seeding from content bridges
### Seeding history data with the `portal_bridge`
### Seeding history content with the `portal_bridge`
The `portal_bridge` requires `era1` files as source for the block content from before the merge.
It requires access to a full node with EL JSON-RPC API for seeding the latest (head of the chain) block content.
Any block content between the merge and the latest is currently not implemented, but will be implemented in the future by usage of `era` files as source.
#### Step 1: Run a Portal client
@ -45,7 +49,7 @@ WEB3_URL="http://127.0.0.1:8548" # Replace with your provider.
./build/portal_bridge history --latest:true --backfill:true --audit:true --era1-dir:/somedir/era1/ --web3-url:${WEB3_URL}
```
### Seeding post-merge history data with the `beacon_lc_bridge`
### Seeding post-merge history content with the `beacon_lc_bridge`
The `beacon_lc_bridge` is more of a standalone bridge that does not require access to a full node with its EL JSON-RPC API. However it is also more limited in the functions it provides.
It will start with the consensus light client sync and follow beacon block gossip. Once it is synced, the execution payload of new beacon blocks will be extracted and injected in the Portal network as execution headers
@ -72,82 +76,22 @@ TRUSTED_BLOCK_ROOT=0x12345678901234567890123456789012345678901234567890123456789
./build/beacon_lc_bridge --trusted-block-root=${TRUSTED_BLOCK_ROOT}
```
## From locally stored block data
## Seeding directly from the fluffy client
### Building and seeding epoch accumulators
This method currently only supports seeding block content from before the merge.
It uses `era1` files as source for the content.
#### Step 1: Building the epoch accumulators
1. Set-up access to an Ethereum JSON-RPC endpoint (e.g. local geth instance)
that can serve the data.
2. Use the `eth_data_exporter` tool to download and store all block headers into
*.e2s files arranged per epoch (8192 blocks):
```bash
make eth_data_exporter
./build/eth_data_exporter history exportEpochHeaders --data-dir:"./user_data_dir/"
```
This will store all block headers up till the merge block into *.e2s files in
the assigned `--data-dir`.
3. Build the master accumulator and the epoch accumulators:
```bash
./build/eth_data_exporter history exportAccumulatorData --write-epoch-records --data-dir:"./user_data_dir/"
```
#### Step 2: Seed the epoch accumulators into the Portal network
Run Fluffy and trigger the propagation of data with the
`portal_history_propagateEpochRecords` JSON-RPC API call:
```bash
./build/fluffy --rpc --rpc-api:portal,portal_debug
# From another terminal
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"portal_history_propagateEpochRecords","params":["./user_data_dir/"]}' http://localhost:8545 | jq
```
#### Step 3 (Optional): Verify that all epoch accumulators are available
Run Fluffy and run the `content_verifier` tool to verify that all epoch
accumulators are available on the history network:
Make sure you still have a fluffy instance running, if not run:
1. Run Fluffy and enable `portal_debug` JSON-RPC API:
```bash
./build/fluffy --rpc --rpc-api:portal,portal_debug
```
Run the `content_verifier` tool and see if all epoch accumulators are found:
```bash
make content_verifier
./build/content_verifier
```
### Downloading & seeding block data
1. Set-up access to an Ethereum JSON-RPC endpoint (e.g. local geth instance)
that can serve the data.
2. Use the `eth_data_exporter` tool to download history data through the
JSON-RPC endpoint into the format which is suitable for reading data into
Fluffy client and propagating into the network:
2. Trigger the seeding of the content with the `portal_debug_historyGossipHeaders` and `portal_debug_historyGossipBlockContent` JSON-RPC methods.
The first method will gossip in the block headers, the second method will gossip the block bodies and receipts. It is important to first trigger the gossip of the headers because these are required for the validation of the bodies and the receipts.
```bash
make eth_data_exporter
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1",
"method":"portal_debug_historyGossipHeaders","params":["/somedir/era1/"]}' http://localhost:8545 | jq
./build/eth_data_exporter history exportBlockData--initial-block:1 --end-block:10 --data-dir:"/user_data_dir/"
```
This will store blocks 1 to 10 into a json file located at
`./user_data_dir/eth-history-data.json`.
3. Run Fluffy and trigger the propagation of data with the
`portal_debug_history_propagate` JSON-RPC API call:
```bash
./build/fluffy --rpc --rpc-api:portal,portal_debug
# From another shell
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"portal_debug_history_propagate","params":["./user_data_dir/eth-history-data.json"]}' http://localhost:8545 | jq
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"portal_debug_historyGossipBlockContent","params":["/somedir/era1/"]}' http://localhost:8545 | jq
```

10
fluffy/eth_data/history_data_json_store.nim
View File

@ -100,16 +100,6 @@ iterator blocks*(
else:
error "Failed reading block from block data", error = res.error
iterator blocksContent*(
blockData: BlockDataTable, verify = false
): (ContentId, seq[byte], seq[byte]) =
for b in blocks(blockData, verify):
for value in b:
if len(value[1]) > 0:
let ckBytes = history_content.encode(value[0])
let contentId = history_content.toContentId(ckBytes)
yield (contentId, ckBytes.asSeq(), value[1])
func readBlockHeader*(blockData: BlockData): Result[Header, string] =
var rlp =
try:

193
fluffy/eth_data/history_data_seeding.nim
View File

@ -8,209 +8,18 @@
{.push raises: [].}
import
std/[strformat, os],
results,
chronos,
chronicles,
../network/wire/portal_protocol,
../network/history/
[history_content, history_network, validation/historical_hashes_accumulator],
"."/[era1, history_data_json_store, history_data_ssz_e2s]
"."/[era1, history_data_ssz_e2s]
from eth/common/eth_types_rlp import rlpHash
export results
### Helper calls to seed the local database and/or the network
proc historyStore*(
p: PortalProtocol, dataFile: string, verify = false
): Result[void, string] =
let blockData = ?readJsonType(dataFile, BlockDataTable)
for b in blocks(blockData, verify):
for value in b:
let encKey = history_content.encode(value[0])
# Note: This is the slowest part due to the hashing that takes place.
p.storeContent(encKey, history_content.toContentId(encKey), value[1])
ok()
proc historyPropagate*(
p: PortalProtocol, dataFile: string, verify = false
): Future[Result[void, string]] {.async.} =
const concurrentGossips = 20
var gossipQueue =
newAsyncQueue[(Opt[NodeId], ContentKeysList, seq[byte])](concurrentGossips)
var gossipWorkers: seq[Future[void]]
proc gossipWorker(p: PortalProtocol) {.async.} =
while true:
let (srcNodeId, keys, content) = await gossipQueue.popFirst()
discard await p.neighborhoodGossip(srcNodeId, keys, @[content])
for i in 0 ..< concurrentGossips:
gossipWorkers.add(gossipWorker(p))
let blockData = readJsonType(dataFile, BlockDataTable)
if blockData.isOk():
for b in blocks(blockData.get(), verify):
for i, value in b:
if i == 0:
# Note: Skipping propagation of headers here as they should be offered
# separately to be certain that bodies and receipts can be verified.
# TODO: Rename this chain of calls to be more clear about this and
# adjust the interator call.
continue
# Only sending non empty data, e.g. empty receipts are not send
# TODO: Could do a similar thing for a combination of empty
# txs and empty uncles, as then the serialization is always the same.
if value[1].len() > 0:
info "Seeding block content into the network", contentKey = value[0]
# Note: This is the slowest part due to the hashing that takes place.
let
encKey = history_content.encode(value[0])
contentId = history_content.toContentId(encKey)
p.storeContent(encKey, contentId, value[1])
await gossipQueue.addLast(
(Opt.none(NodeId), ContentKeysList(@[encode(value[0])]), value[1])
)
return ok()
else:
return err(blockData.error)
proc historyPropagateBlock*(
p: PortalProtocol, dataFile: string, blockHash: string, verify = false
): Future[Result[void, string]] {.async.} =
let blockDataTable = readJsonType(dataFile, BlockDataTable)
if blockDataTable.isOk():
let b =
try:
blockDataTable.get()[blockHash]
except KeyError:
return err("Block hash not found in block data file")
let blockDataRes = readBlockData(blockHash, b)
if blockDataRes.isErr:
return err(blockDataRes.error)
let blockData = blockDataRes.get()
for value in blockData:
info "Seeding block content into the network", contentKey = value[0]
let
encKey = history_content.encode(value[0])
contentId = history_content.toContentId(encKey)
p.storeContent(encKey, contentId, value[1])
discard await p.neighborhoodGossip(
Opt.none(NodeId), ContentKeysList(@[encode(value[0])]), @[value[1]]
)
return ok()
else:
return err(blockDataTable.error)
proc historyPropagateHeadersWithProof*(
p: PortalProtocol, epochHeadersFile: string, epochRecordFile: string
): Future[Result[void, string]] {.async.} =
let res = readBlockHeaders(epochHeadersFile)
if res.isErr():
return err(res.error)
let blockHeaders = res.get()
let epochRecordRes = readEpochRecordCached(epochRecordFile)
if epochRecordRes.isErr():
return err(res.error)
let epochRecord = epochRecordRes.get()
for header in blockHeaders:
if header.isPreMerge():
let headerWithProof = buildHeaderWithProof(header, epochRecord)
if headerWithProof.isErr:
return err(headerWithProof.error)
let
content = headerWithProof.get()
contentKey = ContentKey(
contentType: blockHeader,
blockHeaderKey: BlockKey(blockHash: header.rlpHash()),
)
encKey = history_content.encode(contentKey)
contentId = history_content.toContentId(encKey)
encodedContent = SSZ.encode(content)
p.storeContent(encKey, contentId, encodedContent)
let keys = ContentKeysList(@[encode(contentKey)])
discard await p.neighborhoodGossip(Opt.none(NodeId), keys, @[encodedContent])
return ok()
proc historyPropagateHeadersWithProof*(
p: PortalProtocol, dataDir: string
): Future[Result[void, string]] {.async.} =
for i in 0 ..< preMergeEpochs:
let
epochHeadersfile =
try:
dataDir / &"mainnet-headers-epoch-{i.uint64:05}.e2s"
except ValueError as e:
raiseAssert e.msg
epochRecordFile =
try:
dataDir / &"mainnet-epoch-record-{i.uint64:05}.ssz"
except ValueError as e:
raiseAssert e.msg
let res =
await p.historyPropagateHeadersWithProof(epochHeadersfile, epochRecordFile)
if res.isOk():
info "Finished gossiping 1 epoch of headers with proof", i
else:
return err(res.error)
return ok()
proc historyPropagateHeaders*(
p: PortalProtocol, dataFile: string, verify = false
): Future[Result[void, string]] {.async.} =
# TODO: Should perhaps be integrated with `historyPropagate` call.
const concurrentGossips = 20
var gossipQueue = newAsyncQueue[(ContentKeysList, seq[byte])](concurrentGossips)
var gossipWorkers: seq[Future[void]]
proc gossipWorker(p: PortalProtocol) {.async.} =
while true:
let (keys, content) = await gossipQueue.popFirst()
discard await p.neighborhoodGossip(Opt.none(NodeId), keys, @[content])
for i in 0 ..< concurrentGossips:
gossipWorkers.add(gossipWorker(p))
let blockData = readJsonType(dataFile, BlockDataTable)
if blockData.isOk():
for header in headers(blockData.get(), verify):
info "Seeding header content into the network", contentKey = header[0]
let
encKey = history_content.encode(header[0])
contentId = history_content.toContentId(encKey)
p.storeContent(encKey, contentId, header[1])
await gossipQueue.addLast((ContentKeysList(@[encode(header[0])]), header[1]))
return ok()
else:
return err(blockData.error)
##
## Era1 based iterators that encode to Portal content
##

4
fluffy/rpc/rpc_calls/rpc_portal_debug_calls.nim
View File

@ -21,7 +21,3 @@ createRpcSigsFromNim(RpcClient):
proc portal_debug_historyGossipHeaders(era1File: string): bool
proc portal_debug_historyGossipBlockContent(era1File: string): bool
proc portal_debug_history_storeContent(dataFile: string): bool
proc portal_debug_history_propagate(dataFile: string): bool
proc portal_debug_history_propagateHeaders(dataFile: string): bool
proc portal_debug_history_propagateBlock(dataFile: string, blockHash: string): bool

46
fluffy/rpc/rpc_portal_debug_history_api.nim
View File

@ -15,8 +15,8 @@ import
export rpcserver
# Non-spec-RPCs that are used for testing, debugging and seeding data without a
# bridge.
# Non-spec-RPCs that are used for seeding history content into the network without
# usage of the standalone portal_bridge. As source Era1 files are used.
proc installPortalDebugHistoryApiHandlers*(rpcServer: RpcServer, p: PortalProtocol) =
## Portal debug API calls related to storage and seeding from Era1 files.
rpcServer.rpc("portal_debug_historyGossipHeaders") do(
@ -34,45 +34,3 @@ proc installPortalDebugHistoryApiHandlers*(rpcServer: RpcServer, p: PortalProtoc
return true
else:
raise newException(ValueError, $res.error)
## Portal debug API calls related to storage and seeding
## TODO: To be removed/replaced with the Era1 versions where applicable.
rpcServer.rpc("portal_debug_history_storeContent") do(dataFile: string) -> bool:
let res = p.historyStore(dataFile)
if res.isOk():
return true
else:
raise newException(ValueError, $res.error)
rpcServer.rpc("portal_debug_history_propagate") do(dataFile: string) -> bool:
let res = await p.historyPropagate(dataFile)
if res.isOk():
return true
else:
raise newException(ValueError, $res.error)
rpcServer.rpc("portal_debug_history_propagateHeaders") do(dataDir: string) -> bool:
let res = await p.historyPropagateHeadersWithProof(dataDir)
if res.isOk():
return true
else:
raise newException(ValueError, $res.error)
rpcServer.rpc("portal_debug_history_propagateHeaders") do(
epochHeadersFile: string, epochRecordFile: string
) -> bool:
let res =
await p.historyPropagateHeadersWithProof(epochHeadersFile, epochRecordFile)
if res.isOk():
return true
else:
raise newException(ValueError, $res.error)
rpcServer.rpc("portal_debug_history_propagateBlock") do(
dataFile: string, blockHash: string
) -> bool:
let res = await p.historyPropagateBlock(dataFile, blockHash)
if res.isOk():
return true
else:
raise newException(ValueError, $res.error)

19
fluffy/scripts/test_portal_testnet.nim
View File

@ -265,9 +265,22 @@ procSuite "Portal testnet tests":
discard
(await clients[0].portal_historyGossip(content.toHex(), contentKey.toHex()))
# This will fill the first node its db with blocks from the data file. Next,
# this node wil offer all these blocks their headers one by one.
check (await clients[0].portal_debug_history_propagate(blockDataFile))
# Gossiping all block bodies and receipts.
for b in blocks(blockData, false):
for i, value in b:
if i == 0:
# Note: Skipping the headers, they are handled above already
continue
# Only sending non empty data, e.g. empty receipts are not send
# TODO: Could do a similar thing for a combination of empty
# txs and empty uncles, as then the serialization is always the same.
if value[1].len() > 0:
let
contentKey = history_content.encode(value[0]).asSeq().toHex()
contentValue = value[1].toHex()
discard (await clients[0].portal_historyGossip(contentKey, contentValue))
await clients[0].close()
for i, client in clients: