fec507e755
Add peers back to gossipsub table, slow down heartbeat ( #256 )
...
* Add peers back to gossipsub table, slow down heartbeat
* exclude on unsub from mesh and fanout
2020-07-08 11:06:26 -06:00
f2e7035a17
fix gossip 11 upstream
2020-07-08 19:22:02 +09:00
853250918c
wip
2020-07-08 15:45:11 +09:00
be98a478b4
Fix merge oops
2020-07-08 12:58:39 +09:00
a7d90214ea
Merge branch 'gossip-one-one' of github.com:status-im/nim-libp2p into gossip-one-one
2020-07-08 11:52:54 +09:00
0fbc0ec4b2
fix multistream 1.1 multi proto
2020-07-08 11:52:44 +09:00
38422ff4cb
Merge branch 'master' into gossip-one-one
2020-07-08 11:39:33 +09:00
a52763cc6d
fix publishing ( #250 )
...
* use var semantics to optimize table access
* wip... lvalues don't work properly sadly...
* big publish refactor, replenish and balance
* fix internal tests
* use g.peers for fanout (todo: don't include flood peers)
* exclude non gossip from fanout
* internal test fixes
* fix flood tests
* fix test's trypublish
* test interop fixes
* make sure to not remove peers from gossip table
* restore old replenishFanout
* cleanups
* Cleanup resources (#246 )
* consolidate reading in lpstream
* remove debug echo
* tune log level
* add channel cleanup and cancelation handling
* cancelation handling
* cancelation handling
* cancelation handling
* cancelation handling
* cleanup and cancelation handling
* cancelation handling
* cancelation
* tests
* rename isConnected to connected
* remove testing trace
* comment out debug stacktraces
* explicit raises
* restore trace vs debug in gossip
* improve fanout replenish behavior further
* cleanup stale peers more eaguerly
* synchronize connection cleanup and small refactor
* close client first and call parent second
* disconnect failed peers on publish
* check for publish result
* fix tests
* fix tests
* always call close
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-07-07 18:33:05 -06:00
775cab414a
Remove SHA1 from crypto and crypto tests. ( #251 )
...
* Remove SHA1 from crypto and crypto tests.
* Simplify RSA comparison procedure.
Refactor some procedures in crypto.nim.
2020-07-07 15:48:15 +02:00
57e6a95b9c
Merge branch 'publish-backporting' into gossip-one-one
2020-07-07 21:54:07 +09:00
2d0e285810
Merge branch 'master' into publish-backporting
2020-07-07 20:58:58 +09:00
d522537b19
reuse single RNG instance for all crypto key generation ( #249 )
...
* reuse single RNG instance for all crypto key generation
* use foolproof rng
* initRng -> newRng (because it's ref)
* fix test
* imports/exports, chat fix
* fix rsa
* imports and exports
* work around threadvar issue
* fixup
* mac workaround test
2020-07-07 13:14:11 +02:00
73cefce798
remove assertion in publish
2020-07-07 19:42:36 +09:00
69a92aebe6
merge 1.0
2020-07-04 20:53:40 +09:00
2f39cf6f42
Merge branch 'publish-backporting' into gossip-one-one
2020-07-04 19:37:31 +09:00
479c83c1df
with sugar
2020-07-04 19:36:15 +09:00
5e0870d4a4
minor fixes
2020-07-04 14:22:58 +09:00
aed8dcadaf
actually lock the heartbeat
2020-07-04 14:09:42 +09:00
ad7db1ca26
add more heartbeat locking to prevent races
2020-07-04 14:04:44 +09:00
466f3c9171
add back topics in peers, re-enable flood publish
2020-07-04 12:15:35 +09:00
028b390c7f
conditional enable gossip11 module
2020-07-04 11:58:35 +09:00
be0375f840
fix up merge
2020-07-04 11:51:09 +09:00
afb5cb88f9
Merge remote-tracking branch 'origin/publish-backporting' into gossip-one-one
2020-07-04 11:42:16 +09:00
8d0709ce5d
merge new pubsubpeer, copy 1.1 to new module
2020-07-04 11:29:37 +09:00
0d0309a601
better diagnostics
2020-07-03 09:28:20 +09:00
27d22d534d
simplify rebalanceMesh (following spec) and make it finally reach D_high
2020-07-02 23:07:17 +09:00
5e4934bcce
Merge branch 'master' into publish-backporting
2020-07-02 17:44:23 +09:00
4f5c97e27c
remove topics from pubsubpeer (was unused)
2020-07-01 17:44:26 +09:00
d9e0ca6091
getGossipPeers fixes
2020-07-01 17:26:42 +09:00
a976c22dae
triage publish nil peers (issue is on master too but just hidden behind a if/in)
2020-07-01 16:54:28 +09:00
b49c619ca8
export public field types ( #248 )
...
* export public field types
* one more
2020-07-01 09:22:29 +02:00
ec00c7fc50
Peer resultification and defect only ( #245 )
...
* Peer resultification and defect only
* Fixing some tests
* test fixes
* Rename peer into peerid
* better result error message in identify
* further merge fixes
2020-07-01 08:25:09 +02:00
f49e59cb4e
improve fanout replenish behavior further
2020-06-30 11:36:47 +09:00
509d84f03d
restore trace vs debug in gossip
2020-06-30 11:28:45 +09:00
9d33390239
restore utility module import
2020-06-30 00:17:58 +09:00
c788a6a3c0
Cleanup resources ( #246 )
...
* consolidate reading in lpstream
* remove debug echo
* tune log level
* add channel cleanup and cancelation handling
* cancelation handling
* cancelation handling
* cancelation handling
* cancelation handling
* cleanup and cancelation handling
* cancelation handling
* cancelation
* tests
* rename isConnected to connected
* remove testing trace
* comment out debug stacktraces
* explicit raises
2020-06-29 09:15:31 -06:00
680dcfd9cb
cleanups
2020-06-30 00:10:11 +09:00
3a657135dc
Merge branch 'master' into publish-backporting
2020-06-29 22:02:58 +09:00
3dd28adc9f
restore old replenishFanout
2020-06-29 10:46:59 +09:00
2356f6ab68
make sure to not remove peers from gossip table
2020-06-29 10:45:06 +09:00
aa6756dfe0
allow message id provider to be specified ( #243 )
...
* don't send public key in message when not signing (information leak)
* don't run rebalance if there are peers in gossip (see #242 )
* don't crash randomly on bad peer id from remote
2020-06-28 09:56:38 -06:00
b7382eaf25
exclude non gossip from fanout
2020-06-28 10:55:44 +09:00
8e2c9301a0
use g.peers for fanout (todo: don't include flood peers)
2020-06-28 10:53:43 +09:00
902880ef1f
consolidate reading in lpstream ( #241 )
...
* consolidate reading in lpstream
* remove debug echo
* throw if not enough bytes where read
* tune log level
* set eof flag
* test readExactly to fail on not enough bytes
2020-06-27 11:33:34 -06:00
ccb323ad0f
big publish refactor, replenish and balance
2020-06-27 20:24:29 +09:00
dc3c568dc0
wip... lvalues don't work properly sadly...
2020-06-27 14:59:48 +09:00
1a2f336eb5
use var semantics to optimize table access
2020-06-27 13:35:07 +09:00
ab8d7cc968
make sure to waitsub in sparse test
2020-06-26 16:24:56 +09:00
03801fcc52
small fixes, publish still half broken
2020-06-26 16:06:08 +09:00
ec5c0b867d
add floodPublish logic
2020-06-26 12:39:45 +09:00
cf51da00b2
take into account explicit direct peers in publish
2020-06-26 12:31:08 +09:00
84988a147c
standard setup fixes
2020-06-26 10:16:01 +09:00
98e4bb6769
add a way to pass parameters to gossip
2020-06-25 21:14:22 +09:00
7a95f1844b
Concurrent dials ( #238 )
...
* count published messages
* don't call `switch.dial` in `subscribeToPeer`
* add secureconn constructor
* close in the correct order
* concurent dial lock and track in/out conns better
* make tests pass
* add todo comment
* disconect peers that open too many connections
* wip
* do connection and muxer tracking in one place
* prevent nil pointer in observers
* drop connections when peers is over max
* prevent channel leaks
* don't use closure to handle channel
2020-06-24 09:08:44 -06:00
08ed2a7974
prune by score (score is stub still)
2020-06-24 12:43:35 +09:00
cc3a0c29c0
validated always, even explicit peers
2020-06-24 12:19:59 +09:00
eea48f8bd5
fix encodePrune
2020-06-23 17:05:13 +09:00
7748b8c5c9
PeerInfo and ControlPrune protocols
2020-06-23 17:01:21 +09:00
99b982c21d
explicit peering part 3
2020-06-23 17:01:05 +09:00
f8292f7086
explicit peering part 2
2020-06-23 15:56:07 +09:00
b3aebb18e9
explicit peering part 1
2020-06-23 14:23:00 +09:00
9eb240fb7d
add 1.1 protocol to gossip
2020-06-23 11:49:14 +09:00
3568b073fd
allow multiple codecs per protocol (without breaking things)
2020-06-23 11:48:11 +09:00
ee6e545878
multistream select make sure to not report NA ( #235 )
...
* multistream select make sure to not report NA but rather empty string if all fails
Also re-enable tests
* avoid using bad constructs, make multistream.select flow crystal clear
2020-06-22 15:38:48 -06:00
6331b04cb4
secp: requiresInit updates ( #237 )
...
* secp: requiresInit updates
* fix
2020-06-22 19:03:15 +02:00
b99fd88deb
logging fixes
2020-06-21 11:14:19 +02:00
7852c6dd0f
Noise and eth2/nbc fixes ( #226 )
...
* Remove noise padding payload (spec removed it)
* add log scope in secure
* avoid defect array out of range in switch secure when "na"
* improve identify traces
* wip noise fixes
* noise protobuf adjustments (trying)
* add more debugging messages/traces, improve their actual contents
* re-enable ID check in noise
* bump go daemon tag version
* bump go daemon tag version
* enable noise in daemonapi
* interop testing, (both secio and noise will be tested)
* azure cache bump (p2pd)
* CI changes
- Travis: use Go 1.14
- azure-pipelines.yml: big cleanup
- Azure: bump cache keys
- build 64-bit p2pd on 32-bit Windows
- install both Mingw-w64 architectures
* noise logging fixes
* alternate testing between noise and secio
* increase timeout to avoid VM errors in CI (multistream tests)
* refactor heartbeat management in gossipsub
* remove locking within heartbeat
* refactor heartbeat management in gossipsub
* remove locking within heartbeat
Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2020-06-20 19:56:55 +09:00
7a1c1c2ea6
fixing some key not found exceptions ( #231 )
2020-06-19 15:19:07 -06:00
5b28e8c488
Cleanup lpstream, Connection and BufferStream ( #228 )
...
* count published messages
* don't call `switch.dial` in `subscribeToPeer`
* don't use delegation in connection
* move connection out to own file
* don't breakout on reset
* make sure to call close on secured conn
* add lpstream tracing
* don't breackdown by conn id
* fix import
* remove unused lable
* reset connection on exception
* add additional metrics for skipped messages
* check for nil in secure.close
2020-06-19 11:29:43 -06:00
719744f46a
Small fixes ( #230 )
...
* count published messages
* don't call `switch.dial` in `subscribeToPeer`
* make sure sending doesn't fail
* add `contains`
* review comment from prev pr
2020-06-19 11:29:25 -06:00
5f75e5adc6
don't use `switch.dial` in `subscribeToPeer` ( #227 )
...
* count published messages
* don't call `switch.dial` in `subscribeToPeer`
2020-06-17 20:12:32 -06:00
fe828d87d8
count published messages ( #224 )
2020-06-16 22:14:02 -06:00
224c2613e3
Remove noise padding payload (spec removed it) ( #222 )
2020-06-16 17:34:02 -06:00
523eed65ce
metrics: use eth2 name for peers metric ( #223 )
2020-06-16 14:43:45 -06:00
0d81acafa8
Fix multistream select trace proto logging
2020-06-16 12:02:47 +09:00
9d9f793b4f
add metrics for sent messages by topic and peer ( #220 )
2020-06-15 17:39:03 -06:00
3c4c10d871
con't crash on nil ptr
2020-06-15 13:37:25 -06:00
7cb6c81159
Don't modify iterables while iterating them ( #219 )
...
* don't modify iterables while iterating
* assert handlers to properly close connections
2020-06-15 12:30:09 -06:00
9cd47fe816
add a logScope to make tracing less ugly ( #218 )
...
* add a logScope to make tracing less ugly
* don't crash on nil pointer
2020-06-15 12:10:41 -06:00
85b56d0b3a
Observedaddr ( #217 )
...
* send correct observerAddr
* run tests with info log level
* set observedaddr for channels
2020-06-12 19:56:17 -06:00
f97e2deec7
Make methods in FloodSub, GossipSub public ( #216 )
...
Similar to https://github.com/status-im/nim-libp2p/pull/193 but doing it for all
methods to avoid this issue in PubSub in the future. Got hit by this behavior
again in rpcHandler and took me a while to figure out.
See https://github.com/oskarth/nim-private-method-skipping-case/ for minimal
repro
2020-06-12 17:54:12 -06:00
ac04ca6e31
make sure keys exist and more metrics ( #215 )
2020-06-11 20:20:58 -06:00
55a294a5c9
better pubsub metrics ( #214 )
2020-06-11 12:09:34 -06:00
6b196ad7b4
remove pubsub peer on disconnect ( #212 )
...
* remove pubsub peer on disconnect
* make sure lock is aquired
* add $
* count upgrades/dials/disconnects
2020-06-11 08:45:59 -06:00
92579435b6
secio then noise ( #213 )
...
* secio then noise
much fewer peers on witti with noise first
* comment
2020-06-11 08:38:47 +02:00
1afec627c2
proper name for topics so that we can filter dynamically using chronicles ( #210 )
...
* proper name for topics so that we can filter dynamically using chronicles
* lowercase
2020-06-10 10:48:01 +02:00
8d9e231a74
grab agentversion/protoversion ( #211 )
2020-06-09 12:42:52 -06:00
35ff99829e
close streams ( #208 )
...
* close streams
* don't warn on missing proto
2020-06-08 17:42:27 -06:00
ee281310c0
move trace log
2020-06-08 10:40:08 -06:00
82b4ed8f44
use declareCounter rather then gauge for certain metrics
2020-06-07 16:41:23 +09:00
a6a2a81711
Start adding some metrics to pubsub ( #192 )
...
* Start adding some metrics to pubsub
In order to visualize it's functionality
Still WIP
* more metrics
* add per topic metrics
* finishup with requested metrics
* add a metrisServer define to start local server
* PR fixes and cleanup
2020-06-07 09:15:21 +02:00
130c64f33a
don't return nil in dial ( #205 )
...
* dont return nil in dial
* don't crash on pubsub send
2020-06-05 18:17:05 -06:00
2aebae56c0
Don't rely on the side-effects from doAssert
2020-06-05 19:12:10 +03:00
828a80ec8f
Make the MultiAddress.init function used in NBC non-failing
2020-06-05 17:51:22 +03:00
5960d42c50
remove casts from ( #203 )
2020-06-02 20:21:11 -06:00
bb8bff2195
add sparse message propagation tests to gossipsub ( #202 )
...
* add sparce tests to gossipsub
* add send hooks
* remove `all`
2020-06-02 17:53:38 -06:00
285884c20c
Close peers ( #201 )
...
* wip
* exceptions and resource cleanup
* correct peerlifetime on disconnect
* emulate defered
* remove comment
2020-06-02 11:32:42 -06:00
c7298f34f4
additional comments
2020-06-02 09:10:27 -06:00
71640da8f2
remove close from read/write methods
2020-06-02 09:10:27 -06:00
1b4876d26d
emulate `defered`
2020-06-02 09:10:27 -06:00
abf659a01a
more consistent dialing proto selecting logic
2020-06-02 09:10:27 -06:00
86e1c8169c
decorate observers hooks with {.raises: [Defect].}
...
move hooks logic out into standalone procs
License: MIT
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-06-02 09:10:27 -06:00
4df151a3a3
typos
2020-06-02 09:10:27 -06:00
27a7f8c948
move EOF flag after local close and comments
2020-06-02 09:10:27 -06:00
815282a5da
remove all()
2020-06-02 09:10:27 -06:00
293b7da295
typo
2020-06-02 09:10:27 -06:00
6e0eb93d4f
remove readloops
2020-06-02 09:10:27 -06:00
abc12d0fb5
move stram close to a better location
2020-06-02 09:10:27 -06:00
daef00fc7b
don't crash schlesi-dev
2020-06-02 09:10:27 -06:00
640c3bdc45
better exception handling and resource cleanup
2020-06-02 09:10:27 -06:00
7ff76d76b6
better exceptions
2020-06-02 09:10:27 -06:00
95774b2b81
better cleanup
2020-06-02 09:10:27 -06:00
9c2f31262e
wip: try handling child stream exceptions
2020-06-02 09:10:27 -06:00
e3f8f53620
initStream method and better exceptions handling
2020-06-02 09:10:27 -06:00
d3b79b002e
better exceptions and don't fail writes
2020-06-02 09:10:27 -06:00
74f8b5b5f1
resource cleanup
2020-06-02 09:10:27 -06:00
93e5805c01
better exception handling
2020-06-02 09:10:27 -06:00
d83ce4c932
exceptions and resource cleanup
2020-06-02 09:10:27 -06:00
d4bdb42046
gossipsub fixes
2020-06-02 09:10:27 -06:00
9d3cc9647b
fix merge
2020-06-02 09:10:27 -06:00
3d9c0bffba
read from stream
2020-06-02 09:10:27 -06:00
048b1db1ad
revert back allread
2020-06-02 09:10:27 -06:00
8e9716f5c3
remove on transport close cleanup
2020-06-02 09:10:27 -06:00
cf76edb0dd
make noise work again
2020-06-02 09:10:27 -06:00
5158d96eaf
close connection on chronos close
2020-06-02 09:10:27 -06:00
46daed9a38
wip
2020-06-02 09:10:27 -06:00
0f691cbafd
add eof and closed handling
2020-06-02 09:10:27 -06:00
62da2a05c3
wip: rework with proper half-closed
2020-06-02 09:10:27 -06:00
98117a3068
call write until all is written out
2020-06-02 09:10:27 -06:00
802299e69a
breakout from publish loop
2020-06-02 09:10:27 -06:00
7e31210455
cover missing case in MultiAddress.init ( #198 )
...
* cover missing case in MultiAddress.init
* raise assert on marker in protocol
* unify inits for markers / non-markers
* fix string
2020-06-01 14:41:45 +02:00
88dbeebf17
add side effect annotations ( #197 )
2020-06-01 09:25:16 +02:00
37b98ad45c
Secure managers are now sorted, giving priority to noise ( #191 )
...
* Secure managers are now sorted, giving priority to noise
* fix nimble test command
* Fix native tests
* fix directchat sample
* Could not write to connection - reduce verbosity
* fix interop testing
* Remove more tables
* test interop fixes
* directchat fix
* fix interop/remove some deprecation
2020-06-01 08:41:32 +02:00
6affcda937
Less exceptions more results ( #188 )
...
* Less exceptions more results
* Fix daemonapi and interop tests
* Add multibase
* wip multiaddress
* fix the build, consuming new result types
* fix standard setup
* Simplify match, rename into MaError, add more exaustive err text
* Fix the CI issues
* Fix directchat build
* daemon api fixes
* better err messages formatting
Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-05-31 16:22:49 +02:00
7c9e5c2f7a
"Could not write to connection" message split between trace and debug due to log size
2020-05-30 23:47:56 +09:00
b88bfc05f8
Make GossipSub initPubSub method public ( #193 )
...
This means we can use it from other protocols that inherit GossipSub. Otherwise,
a lot of internal state (heartbeat lock etc) doesn't get initialized properly.
2020-05-29 09:35:03 -06:00
7b6e1c0688
Gossipsub interop ( #189 )
...
* interop fixes
* add custom messageid provider and fix seqno
* use ECDSA for speed
* adding messageid tests
* breakout from publish loop
* addressing review comments
* remove unneded var
* dont stop broadcasting on failed peers
2020-05-27 12:33:49 -06:00
536555138c
fix "future" typo
2020-05-26 16:08:09 +09:00
4447d97234
add back forgotten message in tryAndWarn
2020-05-26 15:21:59 +09:00
9132f16927
gossipsub fixes ( #186 )
2020-05-21 14:24:20 -06:00
ba53c08b3c
Track incoming connections ( #181 )
...
* call write until all is written out
* wip: rework with proper half-closed
* add eof and closed handling
* wip
* close connection on chronos close
* don't use read
* make noise work again
* don't reraise just yet
* fixes after backporting
* remove on transport close cleanup
* revert back allread
* rust interop fixes
* read from stream
* inc count before closing
* rebasing master
* store incomming connections
* fix merge
* remove unneeded changes
* use internal close flag to indicate disposal
2020-05-21 11:33:48 -06:00
7900fd9f61
Half closed ( #174 )
...
* call write until all is written out
* add comments to lpchannel fields
* add an eof flag to signal which end closed
* wip: rework with proper half-closed
* add eof and closed handling
* propagate closes to piped
* call parent close
* moving bufferstream trackers out
* move writeLock to bufferstream
* move writeLock out
* remove unused call
* wip
* rebasing master
* fix mplex tests
* wip
* fix bufferstream after backport
* wip
* rename to differentiate from chronos tracker
* close connection on chronos close
* make reset request asyncCheck
* fix channel cleanup
* misc
* don't use read
* fix backports
* make noise work again
* proper exception handling
* don't reraise just yet
* add convenience templates
* dont double wrap
* use async pragma
* fixes after backporting
* muxer owns connection
* remove on transport close cleanup
* revert back allread
* adding some todos
* read from stream
* inc count before closing
* rebasing master
* rebase master
* use correct exception type
* use try/finally insted of defer
* fix compile in trace mode
* reset channels on mplex close
2020-05-19 18:14:15 -06:00
681991ae48
reduce buffer size in lpchannel ( #184 )
2020-05-19 16:15:36 -06:00
01339c991f
Don't use and expose directly secp types ( #183 )
...
* Don't use and expose directly secp types
* Reuse same secp type names
2020-05-19 14:48:55 +02:00
c219100e64
Use results and no exceptions in chacha and curve25519 ( #182 )
2020-05-19 10:22:49 +02:00
f8029e7359
use sha256 digest as cache keys ( #135 )
...
* use sha256 digest as cache keys
* rebasing master
2020-05-18 14:49:49 -06:00
9cf1fd0216
remove generic constructor and expose serverflags ( #176 )
...
* remove generic constructor and expose serverflags
* fix transport constructor
* fix merge issues
2020-05-18 13:04:05 -06:00
773b738c12
don't track Connection, track StreamTransport ( #177 )
...
* don't track Connection, track StreamTransport
* make tests more deterministic
2020-05-18 11:05:34 -06:00
5583168965
Track all connections ( #175 )
...
* adding to do for future refactor
* no mix conn and transport stopping or it will race
* don't use intermediary vars
2020-05-18 09:08:10 -06:00
1819502fb5
Cleanup - tests and logging ( #178 )
...
* make async for proper exception handling
* tryAndWarn msg messes up Exception msg
* misc: comment out tracker dumps
* cleanup mplex tests
* more informative errors
* give CI time to run
* revert change, bacause it causes races
2020-05-18 07:49:49 -06:00
Giovanni Petrantoni
Giovanni Petrantoni
Dmitriy Ryajov
Eugene Kabanov
Jacek Sieka
Oskar Thorén
Viktor Kirilov
Zahary Karadjov