Commit Graph

72 Commits

Author SHA1 Message Date
Jacek Sieka f5ff9e32ca
introduce asyncraises in transports/asyncsync (#470)
With these fixes, `transports`/`asyncsync` correctly propagate and document their raises information - generally, most transport functions (send etc) raise `TransportError` and `CancelledError` - `closeWait` is special in that it generally doesn't fail.

This PR introduces the syntax `Future[void].Raises([types])` to create the `InternalRaisesFuture` type with the correct encoding for the types - this allows it to be used in user code while retaining the possibility to change the internal representation down the line.

* introduce raising constraints on stream callbacks - these constraints now give a warning when called with a callback that can raise exceptions (raising callbacks would crash 
* fix fail and its tests, which wasn't always given a good generic match
* work around nim bugs related to macro expansion of generic types
* make sure transports raise only `TransportError`-derived exceptions (and `CancelledError`)
2023-11-15 09:38:48 +01:00
Eugene Kabanov be2edab3ac
Consider ERROR_NETNAME_DELETED as ConnectionAbortedError. (#460) 2023-10-31 03:43:58 +02:00
Eugene Kabanov a70b145964
IPv4/IPv6 dualstack (#456)
* Initial commit.

* Fix tests.

* Fix linux compilation issue.

* Add getDomain() implementation.
Add getDomain() tests.
Add datagram tests.

* Fix style errors.

* Deprecate NetFlag.
Deprecate new flags in ServerFlags.
Add isAvailable().
Fix setDualstack() to ignore errors on `Auto`.
Updatetests.

* Deprecate some old procedures.
Improve datagram transport a bit.

* Address review comments, and fix tests.

* Fix setDescriptorBlocking() issue.
Recover connect() dualstack behavior.
Add test for connect() IPv6-[IPv4 mapped] addresses.

* Fix alignment code issue.
Fix TcpNoDelay was not available on Windows.

* Add dualstack support to HTTP/HTTPS client/server.
2023-10-30 15:27:50 +02:00
Eugene Kabanov 2e8551b0d9
Cancellation fixes and tests. (#445)
* Add callTick and stream cancellation tests.

* Fix stepsAsync() test.

* Cancellation changes.

* Update and add more cancellation tests.

* Fix Posix shutdown call to handle ENOTCONN error.

* With new changes to to cancellation its now possible.

* Refactor testsoon.nim to not produce artifacts after tests are finished.

* Debugging MacOS issue.

* Adjust flaky test times.

* Fix issue.

* Add test for issue #334 which was also addressed in this PR.
Avoid `break` in problematic test.

* Add noCancelWait() call which prohibits cancellation.
Fix closeWait() calls to use noCancelWait() predicate.
Adding sleep to flaky MacOS test.

* Remove all debugging echoes.

* Fix cancelAndWait() which now could perform multiple attempts to cancel target Future (mustCancel behavior).

* Fix issues revealed by switch to different cancelAndWait().

* Address review comments.

* Fix testutils compilation warning.

* Rename callTick() to internalCallTick().

* Add some documentation comments.

* Disable flaky ratelimit test.

* Rename noCancelWait() to noCancel().
Address review comments.
2023-09-15 19:38:39 +03:00
Jacek Sieka a1eb30360b
fix invalid protocol casts (#430) 2023-08-04 08:08:34 +02:00
Eugene Kabanov d214bcfb4f
Increase backlog defaults to maximum possible values. (#428) 2023-07-31 22:40:00 +03:00
Eugene Kabanov e04c042e8a
Add cross-platform shutdown() call and use it for HTTP server. (#420)
* Add cross-platform shutdown() call and use it for httpserver connection processing.

* Fix Posix compilation issues and warnings.
2023-07-19 20:33:28 +03:00
Eugene Kabanov 155d89450e
Trackers refactoring. (#416)
* Refactor chronos trackers to be more simple.

* Refactor trackers.
Add HTTP server trackers.
Refactor HTTP main processing loop.

* Compatibility fixes.
Add checkLeaks().

* Fix posix test issue.

* Add httpdebug module which introduces HTTP connection dumping helpers.
Add tests for it.

* Recover and deprecate old version of Trackers.

* Make public iterators to iterate over all tracker counters available.
Fix asynctests to use public iterators instead private one.
2023-07-14 13:35:08 +03:00
Eugene Kabanov 94ca0c3847
Fix remoteAddress() should raise TransportAbortedError too. (#409)
Add ENOTCONN as TransportAbortedError.
Refactor remoteAddress()/localAddress().
2023-06-26 16:31:09 +03:00
Jacek Sieka 1d6c309d77
clean up `Defect` (#404) 2023-06-05 22:21:50 +02:00
Eugene Kabanov 02fda01bf2
Add facility to emulate signals on Windows. (#357)
* Initial Windows asyncproc part.
Deprecate usage of net/nativesockets in handles, asyncloop.
Introduce primitives with inheritable flag.

* Some posix fixes.

* More duplicate fixes.

* Fix AsyncProcessOption.EchoCommand option.
Remove isMainModule code.
Fix execCommand to use AsyncProcessOption.EvalCommand.

* Initial asyncproc tests commit.

* Some Posix fixes.

* Fix Posix crash.

* Add testsuite.
Fix osnet.nim to use proper declarations.
Fix timer.nim to use proper declarations.
Add Windows environment handling procedures.
Fix createAsyncPipe.
Add leaks tracking for AsyncProcessRef.

* Fix O_CLOEXEC constant value.

* Add file descriptors leak test.

* Remove commented code.
Refactor exceptions.
Fix compilation warnings.

* No exception ioselectors_kqueue initial commit.

* Some BSD fixes.
Linux refactoring of selectors.nim.

* Some fixes to move further.

* Last Linux fixes.

* Fixes for asyncloop to use 2nd version of selectors api.

* Add osutils.nim.

* Some fixes.

* Hardening resumeRead(), resumeWrite() and consumers.
Add ESRCH handling.
Introduce no-exception fromPipe2.

* Make Windows part exception-free and fix zombie race issue.

* createStreamServer() fixes.

* Upgrade asyncproc to use non-exception primitives.
Fix ioselectors_kqueue to use closeFd().

* Refactor accept() and acceptLoop() to be exception free.

* Deprecated some `result` usage.
Last fixes to make stream.nim exception free.
Use closeFd().
Refactor some loops to use handleEintr().

* Fix connect() forgot to unregister socket on error.

* All createAsyncSocket() sockets should be closed with unregisterAndCloseFd().

* Attempt to fix posix bug with incomplete output.

* Change number of runs in big test.

* Refactoring pipes creation. Attempt to fix "write error: Resource temporarily unavailable".

* Fix waitForExit(duration) code.
Fix test exit code.

* Fix Windows missing SIGTERM.

* Fix mistype.

* Fix compilation error.

* Attempt to fix Nim 1.6 issue.

* Eliminate Nim's WideCString usage to avoid any problems in future.

* Deprecate posix usage in osnet.

* Eliminate unused imports.

* Some debugging statements for investigation.

* Remove all the debugging statements.

* Remove testhelpers in favor of unittest2/asynctests.

* Fix flaky test.

* Make test more resilient to timings.

* Add memory size dump to CI.

* Refactor some blocks to use valueOr.
Make waitForExit to use kill() instead of terminate().

* Remove memory dumps.

* Fix peekProcessExitCode() blocks issue.

* Fix Windows issue.

* Add some debugging echoes.

* Fix compilation error.

* Add some debugging echoes.

* Add more helpers.

* Fix compilation error.

* waitForExit() is primary suspect.

* Remove fast-path for waitForExit.
Remove loop for reading signal events.

* Remove all debugging echoes.

* Return one debugging echo.

* Fix futures tests.

* Add test for multiple processes waiting to attempt stress test.

* Refactor ioselectors_epoll for better signalfd and process monitoring.
Add more race condition fixes to waitForExit.
Fix some Nim 1.6 warnings.

* Fix after rebase issues and warnings.

* Fix style issues.
Fix different Nim versions issues.
Workaround `signalfd` style issues.

* Add one more Linux signals workaround.
Add one more multiple processes test.

* Windows fixes.

* Remove unixPlatform define.
Fix WSAECONNABORTED for devel.

* Temporarily disable rate limit tests.
Fix more devel issues.

* Deprecate `hasThreadSupport` for ioselectors_kqueue.
Fix verifySelectParams issue.
Add exit codes test for multiple processes.
Fix osnet PtrToCstringConv warning.

* ioselectors_kqueue refactoring.

* Initial commit.

* Fix 1.2-1.4 compilation issue.

* Fix unused warning for testCtrlC() test.

* Post-rebase fixes.

* Restore master files.

* More fixes.

* Remove duplicates.

* Fix style mistake.

* Add more flexible pragmas.
2023-06-02 01:53:20 +03:00
Eugene Kabanov 3118f8c1b2
Fix cast[pointer] issues on nim-devel. (#381)
* Fix cast[pointer] issues on nim-devel.

* More cast[T] fixes.

* Fix compilation error.

* Add oserrno.nim
Further fixes for Windows and Linux.

* MacOS fixes.

* More Windows fixes and attempt to fix 1.2, 1.4 branches.

* Implicitly import/export oserrno.

* Replace oserrno with osdefs.

* Return back oserrno.

* epoll to oserrno.

* datagram/stream to oserrno.

* common to oserrno.

* test to oserrno.
2023-04-30 08:20:08 +02:00
Eugene Kabanov e05d2f8e96
Fix some compilation warnings and expose tracker names. (#376)
* Fix asyncsync compilation warnings.

* Fix tracker names should be public.
2023-04-08 19:34:57 +03:00
Tanguy ab5a8c2e0f
Add `localAddress` support to `stream.connect` (#362)
* Add `localAddress` support to `stream.connect`

* fix windows

* TransportAddress() instead of AnyAddress

* tweak flags

* Better flags

* try to workaround nim 1.2 issue

* Handle ReusePort in createStreamServer and improve tests

* Rename ClientFlags to SocketFlags

---------

Co-authored-by: Diego <diego@status.im>
2023-04-03 15:34:35 +03:00
Eugene Kabanov b0af576c7c
Address #320 issue. (#372) 2023-03-24 17:34:45 +02:00
Eugene Kabanov 0f70a6b8ee
OS definitions and exceptions changes. (AsyncProc part 1). (#361)
* Initial commit.

* Finalize Windows part.

* Finalize Posix part.

* Fix style issues.

* Move osnet declarations to osdefs.

* Move osnet declarations to osdefs 2.

* Fix Nim 1.2 issues.

* Fix 1.2 compilation issues 2.

* Address review comments.
Fix SignalFdInfo issue.
2023-02-21 11:48:36 +01:00
Tanguy 3d532b6d4a
Enable TCP client to enable TCP_NODELAY (#340)
* Enable TCP client to enable TCP_NODELAY

* add missing return

* fix
2023-01-23 11:18:12 +02:00
Bung e9f8baa6ee
fix int set (#341) 2022-12-29 12:12:12 +01:00
Bung 5d0c725463
fix int set construction (#335) 2022-12-16 10:40:33 +01:00
Eugene Kabanov 6525f4ce1d
Fix some warnings. (#310)
* Fix some warnings messages.

* More warning fixes.

* Address review comments.
Fix MacOS issues.

* More warning fixes.

* More Windows specific fixes.

* Fix macos and windows warnings.

* Fix warnings in timer.nim
Refactor to remove `result`.
Improve performance and behavior of timer to string procedure.
Add tests.
2022-11-02 08:09:15 +01:00
Miran 93800879fd
make it more Nim 1.4+ compatible (#296) 2022-08-06 13:56:06 +03:00
Eugene Kabanov e9d0e2d208
Apply fix for recent Nim devel changes. (#300) 2022-07-30 12:48:58 +03:00
Jacek Sieka bb4c3298f5
fix crash after reading from fd > 1024 (#267)
The socket selector holds a `seq` of per-descriptor data. When a reader
is registered, a pointer to a seq item is stored - when the `seq` grows,
this pointer becomes dangling and causes crashes like
https://github.com/status-im/nimbus-eth2/issues/3521.

It turns out that there already exist two mechanisms for passing user
data around - this PR simply removes one of them, saving on memory usage
and removing the need to store pointers to the `seq` data that become
dangling on resize.
2022-04-11 11:56:30 +02:00
Eugene Kabanov 519ca463df
Fix transport.write() unable to send data through OS pipes. (#256)
* Fix transport.write() unable to send data through pipe.
Add test for pipes.

* Fix flaky test.

* Add workaround for Nim's issue #19425.
2022-01-20 18:38:41 +02:00
Ștefan Talpalaru 17fed89c99
style fixes (#253)
* style fixes

and disable the cron CI job
2022-01-04 23:14:30 +01:00
Tanguy d926415f42
Fixes for nimdoc (#252) 2022-01-04 18:00:17 +01:00
Zahary Karadjov 7ca85ddadc
Fix compilation errors with --styleCheck:usages --styleCheck:error 2021-12-08 16:58:24 +02:00
Jacek Sieka c25fa1f6cd
posix: fast path for write (#244)
When `write` is called on a `StreamTransport`, the current sequence of
operations is:

* copy data to queue
* register for "write" event notification
* return unfinished future to `write` caller
* wait for "write" notification (in `poll`)
* perform one `send`
* wait for notification again if there's more data to write
* complete the future

In this PR, we introduce a fast path for writing:

* If the queue is empty, try to send as much data as possible
  * If all data is sent, return completed future without `poll` round
  * If there's more data to send than can be sent in one go, add the
rest to queue
* If the queue is not empty, enqueue as above
* When notified that write is possible, keep writing until OS buffer is
full before waiting for event again

The fast path provides significant performance benefits when there are
many small writes, such as when sending gossip to many peers, by
avoiding the poll loop and data copy on each send.

Also fixes an issue where the socket would not be removed from the
writer set if there were pending writes on close.
2021-12-08 11:35:27 +01:00
Ștefan Talpalaru 4aaaa3379a
Nim-1.6 fixes (#239) 2021-11-20 01:30:23 +01:00
Eugene Kabanov 0c7a0bd0a5
Handle accept() errors properly. (#234)
* Add more accept() call error handlers.
Fix issue when HTTP server silently stops accepting new connections.
Remove unneeded MacOS syscall to disable SIG_PIPE on socket, we already mask this signal in process.

* User `if` instead `case` because constants are actually variables.

* Fix mistypes.

* Do not use case for posix constants.

* Fix Linux compilation error.
2021-11-12 18:13:56 +02:00
Eugene Kabanov 5034f0a5a6
Fix issue #219 (#220)
* Address issue #219 and add tests for it.
Some cosmetic refactoring.

* Fix *nix tests.
2021-09-05 00:53:27 +03:00
Eugene Kabanov 14ebf269e9
Fix accept defect (#206)
* Fix accept() FutureDefect on cancellation.

* Do not close pipe twice.

* Check for retFuture state before changing it state.
2021-06-30 18:22:37 +03:00
Eugene Kabanov d270dba8a3
Fix unused warnings, result, asyncCheck and 80 cpl (#185)
* Fix sources to follow 80 characters per line.
Fix unused compilation warnings.
Refactor (remove result) handles.nim.
Fix tests to use asyncSpawn instead of asyncCheck.

* Fix handles for Unix platforms.
2021-05-07 23:52:24 +03:00
Eugene Kabanov 39f4060e07
Refactor and optimization of BoundedStream. (#180)
Documentation for BoundedStream.
Fix HttpServer bounding.
2021-04-26 14:05:37 +03:00
Jacek Sieka 895fc53193
add explicit gcsafe annotations (fixes #174) (#175)
Seems like a variation of https://github.com/nim-lang/Nim/issues/17369
2021-04-03 10:05:08 +02:00
Jacek Sieka 4abd7a5645
exception tracking (#166)
* exception tracking

This PR adds minimal exception tracking to chronos, moving the goalpost
one step further.

In particular, it becomes invalid to raise exceptions from `callSoon`
callbacks: this is critical for writing correct error handling because
there's no reasonable way that a user of chronos can possibly _reason_
about exceptions coming out of there: the event loop will be in an
indeterminite state when the loop is executing an _random_ callback.

As expected, there are several issues in the error handling of chronos:
in particular, it will end up in an inconsistent internal state whenever
the selector loop operations fail, because the internal state update
functions are not written in an exception-safe way. This PR turns this
into a Defect, which probably is not the optimal way of handling things
- expect more work to be done here.

Some API have no way of reporting back errors to callers - for example,
when something fails in the accept loop, there's not much it can do, and
no way to report it back to the user of the API - this has been fixed
with the new accept flow - the old one should be deprecated.

Finally, there is information loss in the API: in composite operations
like `poll` and `waitFor` there's no way to differentiate internal
errors from user-level errors originating from callbacks.

* store `CatchableError` in future
* annotate proc's with correct raises information
* `selectors2` to avoid non-CatchableError IOSelectorsException
* `$` should never raise
* remove unnecessary gcsafe annotations
* fix exceptions leaking out of timer waits
* fix some imports
* functions must signal raising the union of all exceptions across all
platforms to enable cross-platform code
* switch to unittest2
* add `selectors2` which supercedes the std library version and fixes
several exception handling issues in there

* fixes

* docs, platform-independent eh specifiers for some functions

* add feature flag for strict exception mode

also bump version to 3.0.0 - _most_ existing code should be compatible
with this version of exception handling but some things might need
fixing - callbacks, existing raises specifications etc.

* fix AsyncCheck for non-void T
2021-03-24 10:08:33 +01:00
cheatfate fb137112be Remove debugging changes. 2021-02-18 22:16:04 +02:00
cheatfate 13eddf382d Simplification and fixes for TLSStream state machine. 2021-02-18 22:16:04 +02:00
cheatfate 49fd70f504 Attempt #4. 2021-02-18 22:16:04 +02:00
cheatfate f1b43aeb04 Investigation of Linux freezes #3. 2021-02-18 22:16:04 +02:00
Eugene Kabanov 0933feaa35
Fix callSoon to raise Defect only. (#145)
* Initial commit.

* Move pragmas to forward declaration.
Add raises to callSoon declaration.

* Fix for IOSelectorsException.
2021-01-11 18:15:23 +01:00
Eugene Kabanov ac9b3e304f
Fix deadlock for pending write() calls on transport close. (#139)
Add tests for read() and write() deadlocks.
2020-11-18 11:30:33 +02:00
Eugene Kabanov 57ebe84d17
Add Defect check for arguments `pbytes` and `nbytes`. (#128)
* Add Defect check for arguments `pbytes` and `nbytes`.

* Allow some edge cases to be not Defect.
2020-09-10 03:50:06 +03:00
Eugene Kabanov 31fec25063
Fix accept races. (#110)
* Fix accept races.
2020-07-15 11:09:34 +03:00
cheatfate 783f84aa4b
Fix EAGAIN issue in acceptLoop(). 2020-07-15 01:07:03 +03:00
Ștefan Talpalaru 0ad7321aa0
ServerFlags.ReusePort support on POSIX (#95) 2020-07-12 18:13:22 +02:00
Eugene Kabanov 528688d01e
Add sanity check. (#105) 2020-06-25 09:32:41 +03:00
Eugene Kabanov 02b8da986b
Add accept() call (#103)
* Add accept() call and tests.
* Fix rare fd leaks on Windows.
* Fix compilation warnings.
* Add fd leak test.
* Bump version to 2.4.0.
2020-06-24 11:21:52 +03:00
Eugene Kabanov bedd1ded5e
Properly fix cancellation race and not introduce FD leaks. (#102) 2020-06-05 19:11:51 +03:00
Eugene Kabanov d6d0084333
Fix cancellation race when connect() call is getting cancelled. (#101) 2020-06-04 13:07:52 +03:00