Commit Graph

112 Commits

Author SHA1 Message Date
Jordan Hrycaj 0d2a72d2a9
Flare sync (#2627)
* Cosmetics, small fixes, add stashed headers verifier

* Remove direct `Era1` support

why:
  Era1 is indirectly supported by using the import tool before syncing.

* Clarify database persistent save function.

why:
  Function relied on the last saved state block number which was wrong.
  It now relies on the tx-level. If it is 0, then data are saved directly.
  Otherwise the task that owns the tx will do it.

* Extracted configuration constants into separate file

* Enable single peer mode for debugging

* Fix peer losing issue in multi-mode

details:
  Running concurrent download peers was previously programmed as running
  a batch downloading and storing ~8k headers and then leaving the `async`
  function to be restarted by a scheduler.

  This was unfortunate because of occasionally occurring long waiting
  times for restart.

  While the time gap until restarting were typically observed a few
  millisecs, there were always a few outliers which well exceed several
  seconds. This seemed to let remote peers run into timeouts.

* Prefix function names `unprocXxx()` and `stagedYyy()` by `headers`

why:
  There will be other `unproc` and `staged` modules.

* Remove cruft, update logging

* Fix accounting issue

details:
  When staging after fetching headers from the network, there was an off
  by 1 error occurring when the result was by one smaller than requested.
  Also, a whole range was mis-accounted when a peer was terminating
  connection immediately after responding.

* Fix slow/error header accounting when fetching

why:
  Originally set for detecting slow headers in a row, the counter
  was wrongly extended to general errors.

* Ban peers for a while that respond with too few headers continuously

why:
  Some peers only returned one header at a time. If these peers sit on a
  farm, they might collectively slow down the download process.

* Update RPC beacon header updater

why:
  Old function hook has slightly changed its meaning since it was used
  for snap sync. Also, the old hook is used by other functions already.

* Limit number of peers or set to single peer mode

details:
  Merge several concepts, single peer mode being one of it.

* Some code clean up, fixings for removing of compiler warnings

* De-noise header fetch related sources

why:
  Header download looks relatively stable, so general debugging is not
  needed, anymore. This is the equivalent of removing the scaffold from
  the part of the building where work has completed.

* More clean up and code prettification for headers stuff

* Implement body fetch and block import

details:
  Available headers are used stage blocks by combining existing headers
  with newly fetched blocks. Then these blocks are imported/executed via
  `persistBlocks()`.

* Logger cosmetics and cleanup

* Remove staged block queue debugging

details:
  Feature still available, just not executed anymore

* Docu, logging update

* Update/simplify `runDaemon()`

* Re-calibrate block body requests and soft config for import blocks batch

why:
* For fetching, larger fetch requests are mostly truncated anyway on
  MainNet.
* For executing, smaller batch sizes reduce the memory needed for the
  price of longer execution times.

* Update metrics counters

* Docu update

* Some fixes, formatting updates, etc.

* Update `borrowed` type: uint -. uint64

also:
  Always convert to `uint64` rather than `uint` where appropriate
2024-09-27 15:07:42 +00:00
tersec 35d4529ef2
rm outdated development information (#2646) 2024-09-23 09:22:58 +00:00
tersec 1b173d420d
small cleanups (#2598)
* small cleanups

* stop hiding ConvFromXtoItselfNotNeeded hints

* lowmem optimization flag is no-op
2024-09-10 05:24:45 +00:00
Jordan Hrycaj 42a08cfba9
Coredb and sync maintenance update (#2583)
* bump metrics

* Remove cruft

* Cosmetics, update some logging, noise control

* Renamed `CoreDb` function `hasKey` => `hasKeyRc` and provided `hasKey`

why:
  Currently, `hasKey` returns a `Result[]` rather than a `bool` which
  is what one would expect from a function prototype of this name.

  This was a bit of an annoyance and cost unnecessary attention.
2024-08-30 11:18:36 +00:00
Jordan Hrycaj 4db9c5c2d5
Small updates and fixes for rlpx suite (#2571)
* Remove redundant `eth/68` message and clean up docu

details:
  There is only eth/68 available at the moment

* Allow to turn on chronicles line number logging in `Makefile`

* Accept (and forget) tx hashes announcements

why:
  Does no harm to just ignore it at the moment

* Bump nim-eth (rlp fix)
2024-08-19 14:00:10 +00:00
web3-developer 9699293bfc
Update nim-rocksdb to latest version and cleanup outdated RocksDb install instructions in readme. (#2566) 2024-08-16 08:23:04 +02:00
Jordan Hrycaj d148de5b1c
Remove chunked rlpx (#2562)
* bump nim-eth

* Update make environment
2024-08-14 10:56:49 +00:00
web3-developer 060c759b01
Use vendor RocksDb dll on Windows. (#2341)
* Bump rocksdb version and update Makefile.

* Update readme.
2024-06-13 10:10:53 +08:00
tersec f932c8df22
rm some proof of work mining support (#2227) 2024-05-27 21:49:07 +02:00
andri lim 70be982259
Fix CI badge URL (#2150) 2024-04-24 08:17:13 +07:00
Jordan Hrycaj e8eb3268f5
Generalise prune mode option 4 different db models (#2139)
* Update README

* Nimbus-main: replaced `PruneMode` options by `ChainDbMode` options

details:
  For the legacy database, this changes the phrase
  - `conf.pruneMode == PruneMode.Full` to the expression
  + `conf.chainDbMode == ChainDbMode.Prune`.

* Fix issues moaned about by NIM compiler

* Fix copyright year
2024-04-17 18:09:55 +00:00
Jordan Hrycaj dac7a2cbe1
Providing eth68 stubs while setting eth67 as default (#2129)
* Suspend `snap` sync tests

why:
  Snap needs to be updated. While this is not done, tests are obsolete
  and eat up memory resources only

* Suspend rocks DB timing tests

why:
  Currently of no use for general test suite.

* Mothballed unused evm code for opportunistic DB handling

why:
  Needs to be refactored anyway. Uses hard coded protocol dependencies.

* Update `eth` protocol configuration

why:
 + new upcoming version `eth68`
 + prepare for eth-multi protocol stack support

* Strip the `legacy_` prefix from eth66 compiler flag variable

why:
  Being it is an oddity having `eth67_enabled`, `eth68_enabled`, and
  `legacy_eth_66_enabled`.

* Providing eth68 stubs

* Fix copyright year

* Fix eth68 stub method name
2024-04-12 11:21:17 +00:00
Kim De Mey 77d8e97823
Update README.md to clarify nimbus update (#1943) 2023-12-13 13:58:58 +01:00
Vivek Kumar 00659bc181
Update README.md to fix link to portal specs (#1812)
* Update README.md to fix link to portal specs

---------

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-10-10 11:01:18 +02:00
jangko a7c088843d
Remove VM2 mention in README.md 2023-07-02 09:20:58 +07:00
A. F. Dudley beecc48374 Fixed Android install instructions 2023-07-02 08:38:28 +07:00
Jordan Hrycaj f20f20f962
Prepare snap server client test scenario (#1483)
* Enable `snap/1` accounts range service

* Allow to change the garbage collector to `boehm` as a Makefile option.

why:
  There is still an unsolved memory corruption problem that might be
  related to the standard `gc`. It seemingly goes away if the `gc` is
  changed to `boehm`.

  Specifying another `gc` on the make level simplifies debugging and
  development.

* Code cosmetics

details:
* updated exception annotations
* extracted `worker_desc.nim` from `full/worker.nim`
* etc.

* Implement option to state a sync modifier file

why:
  This allows to specify extra sync type specific options which might
  change over time. This file is regularly checked for updates.

* Implement a threshold when to suspend full syncing

why:
  For a test scenario, a full sync beep may work as a local snap server.
  There is no need to download the full block chain.

details:
  The file containing the pivot specs is specified by the
  `--sync-ctrl-file` option. It is regularly parsed for updates.
2023-03-02 09:57:58 +00:00
Kim De Mey 96776deed6
Update Fluffy docs on how to seed data (#1408) 2023-01-09 17:57:55 +01:00
Jacek Sieka 51f3124717
Remove stability badge
There's no ABI to speak of
2022-09-21 11:32:15 +02:00
Marius Kjærstad 952ac647a6
Changed http:// to https:// on some links (#1221)
Changed http:// to https:// on some links in README.md
2022-09-10 18:51:34 +02:00
Ivan Yonchovski 7031ef88fc Fix windows build instructions 2022-09-03 20:44:07 +07:00
Jordan Hrycaj 4696a53302
Enable optional chunked RLPx messages (#1010)
* Enable optional chunked RLPx messages

why:
  Legacy feature used by Nethermind

details:
  Disable with make flag: ENABLE_CHUNKED_RLPX=0

* Rebase & bump nim-eth

* Fix default behaviour

why:
  Got lost somehow. Comments do not match GNU make code directives.
2022-03-29 10:19:32 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
Jordan Hrycaj ed0e882387
Exported blobs and some scripts to parallel project nimbus-eth1-blobs (#995)
why:
  TDD data and test script that are not needed for CI are externally held.
  This saves space.

also:
  Added support for test-custom_networks.nim to run import Devnet5 dump.
2022-03-16 09:13:17 +00:00
Kim De Mey 1d74ef4b98
Fix typo in README.md (#850) 2021-09-29 13:04:51 +02:00
sacha 749069da5c
Update README.md
Add recent highlights
2021-09-14 00:48:25 +02:00
sacha b16c16f3ea
Update README.md 2021-09-14 00:04:05 +02:00
Kim De Mey a05f366ca2
Add monthly development updates link (#776)
* Update readme with adjusted intro and dev updates link

* Remove pcre from prerequisites as it no longer is

* Change to hackmd doc as status notes seem not accessible
2021-08-09 16:10:58 +02:00
Kim De Mey f8b3922eb5
Change name from nlpn to fluffy (#734) 2021-06-28 17:53:13 +02:00
jangko 59595b6485
update badges section in README.md [skip ci]
- remove azure pipelines and travis badges because we don't need them anymore
- fix github action badge
- add nlpn CI badge
2021-06-24 08:48:55 +07:00
Jordan Hrycaj bca6e791aa provide experimental op handler switch -d:lowmem:1 for low memory C compiler
why:
  on 32bit windows 7, there seems to be a 64k memory ceiling for the gcc
  compiler which was exceeded on some test platform.

details:
  compiling VM2 for low memory C compiler can be triggered with
  "make ENABLE_VM2LOWMEM". this comes with a ~24% longer execution time
  of the test suite against old VM and optimised VM2.
2021-04-28 15:24:14 +03:00
Jacek Sieka 3147df0dcd
switch to chronos metrics, remove insecure (#580)
* switch to chronos metrics, remove insecure

See https://github.com/status-im/nimbus-eth2/pull/2468

also fixes pcre linking for real, and adds some random build flags that
help nimbus-eth2 stay afloat

* fix help

* don't omit frame pointers on windows
2021-04-09 09:26:06 +02:00
jangko b22828806f add github action script 2021-01-15 14:03:10 +07:00
zah 4e77dc220c
Update README.md 2020-10-07 21:48:46 +03:00
Ștefan Talpalaru 6fcf7f4ef4
add link to nimbus-build-system docs 2020-06-13 02:03:02 +02:00
Ștefan Talpalaru 73e9199ebf
new initial submodule update strategy (#494)
* new initial submodule update strategy

* Azure: increase timeout

* Makefile: change comment [skip ci]
2020-04-18 14:56:40 +02:00
Ștefan Talpalaru 9c0922cc16
Azure test (#492)
* Azure test

* can't build libnimbus on Windows
2020-04-16 03:31:00 +02:00
acolytec3 b4dc8105d1 Add readme updates 2020-02-21 23:45:21 +02:00
Oskar Thorén 5a978089d6 Update README.md (#434)
Without `pcre-devel` I get the following error when running `make`:

```
> make
Building: build/premix

Error: execution of an external compiler program 'gcc -c  -w -pthread -I/home/oskarth/git/status-im/nimbus/vendor/nim-secp256k1/secp256k1_wrapper -I/home/oskarth/git/status-im/nimbus/vendor/nim-secp256k1/secp256k1_wrapper/secp256k1 -I/home/oskarth/git/status-im/nimbus/vendor/nim-secp256k1/secp256k1_wrapper/secp256k1/src -DHAVE_CONFIG_H -I/home/oskarth/git/status-im/nimbus/vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc -I/home/oskarth/git/status-im/nimbus/vendor/nim-nat-traversal/vendor/libnatpmp -DENABLE_STRNATPMPERR -g3 -Og -O3 -fno-strict-aliasing  -I/home/oskarth/git/status-im/nimbus/vendor/nimbus-build-system/vendor/Nim/lib -I/home/oskarth/git/status-im/nimbus/premix -o nimcache/debug/premix/stdlib_re.nim.c.o nimcache/debug/premix/stdlib_re.nim.c' failed with exit code: 1

nimcache/debug/premix/stdlib_re.nim.c:10:10: fatal error: pcre.h: No such file or directory
 #include <pcre.h>
          ^~~~~~~~
compilation terminated.
make: *** [Makefile:48: premix] Error 1
```
2019-12-09 09:44:59 +01:00
Jacek Sieka f4ff53364e
Update README.md 2019-10-25 14:11:02 +02:00
Jacek Sieka 03d033cc95
Update README.md 2019-10-25 13:47:14 +02:00
Jacek Sieka 39bc73b7e0
Update README.md 2019-10-25 13:46:06 +02:00
Jacek Sieka 4e010676cc
Update README.md 2019-10-25 13:44:58 +02:00
Dustin Brody 3a71ad202d Under Debian and probably Ubuntu, simply running prometheus does not use local config 2019-10-25 11:16:19 +00:00
Ștefan Talpalaru 642691aa30
link screenshot [skip ci] 2019-10-02 22:50:40 +02:00
Ștefan Talpalaru 7dd2c657fe
metric visualisation instructions 2019-10-02 18:14:21 +02:00
Ștefan Talpalaru 013688c44b
README: minimum Git version 2019-09-12 18:29:14 +02:00
Ștefan Talpalaru b086ad9c5a
it's "libpcre3-dev" in Ubuntu too 2019-09-04 13:29:36 +02:00
Ștefan Talpalaru 0602dae2f0
spelling 2019-09-04 13:11:22 +02:00
Ștefan Talpalaru c120ec2e49
more env.sh info 2019-09-04 13:09:42 +02:00