Commit Graph

128 Commits

Author SHA1 Message Date
Eugene Kabanov 03f4328de6
Fix possible issues with byte order in ipnet. (#562)
* Fix issues with byte order.

* Some polishing.
2024-11-28 10:10:12 +02:00
Giuliano Mega 7630f39471
Fixes compilation issues in v3 compatibility mode (`-d:chronosHandleException`) (#545)
* add missing calls to await

* add test run in v3 compatibility

* fix semantics for chronosHandleException so it does not override local raises/handleException annotations

* distinguish between explicit override and default setting; fix test

* re-enable wrongly disabled check

* make implementation simpler/clearer

* update docs

* reflow long line

* word swap
2024-06-10 10:18:42 +02:00
Eugene Kabanov 7a3eaffa4f
Fix English spelling for `readed` variable. (#534) 2024-04-17 23:08:19 +00:00
Eugene Kabanov 0d050d5823
Add automatic constructors for TCP and UDP transports. (#512)
* Add automatic constructors for TCP and UDP transports.

* Add port number argument.
Add some documentation comments.
Fix tests.

* Make datagram test use request/response scheme.

* Add helper.

* Fix issue with non-zero port setups.
Add test.

* Fix tests to probe ports.

* Attempt to fix MacOS issue.

* Add Opt[IpAddress].
Make IPv4 mapping to IPv6 space automatic.

* Add tests.

* Add stream capabilities.

* Fix Linux issues.

* Make getTransportFlags() available for all OSes.

* Fix one more compilation issue.

* Workaround weird compiler bug.

* Fix forgotten typed version of constructor.

* Make single source for addresses calculation.

* Add one more check into tests.

* Fix flags not being set in transport constructor.

* Fix post-rebase issues with flags not being set.

* Address review comments.
2024-04-13 03:04:42 +03:00
Eugene Kabanov 8e49df1400
Ensure that all buffers used inside HTTP client will follow original buffer size. (#530)
Ensure that buffer size cannot be lower than default size.
2024-04-07 07:03:12 +03:00
Eugene Kabanov 2d85229dce
Add `join()` operation to wait for future completion. (#525)
* Add `join()` operation to wait for future completion without cancelling it when `join()` got cancelled.

* Start using join() operation.
2024-04-04 00:30:01 +03:00
Eugene Kabanov 402914f4cf
Add custom ring buffer into chronos streams and transports. (#485)
* Add custom ring buffer into chronos stream transport.

* Rename BipBuffer.decommit() to BipBuffer.consume()
Make asyncstream's using BipBuffer.

* Address review comments part 1.

* Address review comments part 2.

* Address review comments.

* Remove unused import results.

* Address review comments.
2024-03-26 22:33:19 +02:00
Eugene Kabanov 035288f3f0
Remove `sink` and `chronosMoveSink()` usage. (#524) 2024-03-20 07:47:59 +01:00
Etan Kissling 17b7a76c7e
Ensure `transp.reader` is reset to `nil` on error (#508)
In `stream.readLoop`, a finished `Future` was left in `transp.reader`
if there was an error in `resumeRead`. Set it to `nil` as well.

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-03-07 08:09:16 +01:00
Eugene Kabanov 2e37a6e26c
Increase AsyncStream and Transport default buffer size from 4096 to 16384 bytes. (#506)
Make buffer sizes configurable at compile time.
2024-02-14 19:23:01 +02:00
Eugene Kabanov a81961a3c6
Fix HTTP server accept() loop exiting under heavy load. (#502)
* Add more specific accept() exceptions raised.
Add some refactoring to HTTP server code.

* Refactor acceptLoop.

* Print GC statistics in every failing test.

* Try to disable failing tests.
2024-02-14 14:05:19 +02:00
Jacek Sieka 3ca2c5e6b5
deprecate `callback=`, UDP fixes (fixes #491) (#492)
Using the callback setter may lead to callbacks owned by others being
reset, which is unexpected.

* don't crash on zero-length UDP writes
2024-01-19 09:21:10 +01:00
Jacek Sieka e15dc3b41f
prevent http `closeWait` future from being cancelled (#486)
* simplify `closeWait` implementations
  * remove redundant cancellation callbacks
  * use `noCancel` to avoid forgetting the right future flags
* add a few missing raises trackers
* enforce `OwnCancelSchedule` on manually created futures that don't raise `CancelledError`
* ensure cancellations don't reach internal futures
2024-01-04 16:17:42 +01:00
Jacek Sieka e38ceb5378
fix v3 backwards compatibility for callbacks (#481)
Because the callback types were used explicitly in some consumers of
chronos, the change of type introduces a backwards incompatibility
preventing a smooth transition to v4 for code that doesn't uses
`raises`.

This PR restores backwards compatibility at the expense of introducing a
new type with a potentially ugly name - that said, there is already
precedence for using numbered names to provide new error handling
strategy in chronos.
2023-12-04 14:19:29 +01:00
Eugene Kabanov b18d471629
Asyncraises HTTP client/server. (#476)
* Fixes.

* Make httpcommon no-raises.

* Make httpbodyrw no-raises.

* Make multipart no-raises.

* Make httpdebug no-raises.

* Make httpagent no-raises.

* Make httpclient no-raises.

* Make httpserver/shttpserver no-raises.

* fix prepend/remove when E is noraises

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-11-21 11:01:44 +01:00
Jacek Sieka f03cdfcc40
futures: sinkify (#475)
This avoids copies here and there throughout the pipeline - ie
`copyString` and friends can often be avoided when moving things into
and out of futures

Annoyingly, one has to sprinkle the codebase liberally with `sink` and
`move` for the pipeline to work well - sink stuff _generally_ works
better in orc/arc

Looking at nim 1.6/refc, sink + local variable + move generates the best
code:

msg directly:
```nim
	T1_ = (*colonenv_).msg1; (*colonenv_).msg1 = copyStringRC1(msg);
```

local copy without move:
```nim
	T60_ = (*colonenv_).localCopy1; (*colonenv_).localCopy1 =
copyStringRC1(msg);
```

local copy with move:
```nim
	asgnRef((void**) (&(*colonenv_).localCopy1), msg);
```

Annoyingly, sink is also broken for refc+literals as it tries to
changes the refcount of the literal as part of the move (which shouldn't
be happening, but here we are), so we have to use a hack to find
literals and avoid moving them.
2023-11-19 18:29:09 +01:00
Eugene Kabanov 0b136b33c8
Asyncstreams asyncraises. (#472)
* Fix transports addresses functions should not return so many exceptions.

* Add raising `Defect` functions to AsyncQueue.

* Add raises/asyncraises into async streams.

* Remove `Safe` primitives.
Make AsyncStreamError to be ancestor of AsyncError.
Make AsyncStreamReader/Writer loops requirement to not raise any exceptions

* Remove `par` fields.

* Remove `par` fields from TLSStream.

* Attempt to lower memory usage.
2023-11-17 23:18:09 +01:00
Jacek Sieka f5ff9e32ca
introduce asyncraises in transports/asyncsync (#470)
With these fixes, `transports`/`asyncsync` correctly propagate and document their raises information - generally, most transport functions (send etc) raise `TransportError` and `CancelledError` - `closeWait` is special in that it generally doesn't fail.

This PR introduces the syntax `Future[void].Raises([types])` to create the `InternalRaisesFuture` type with the correct encoding for the types - this allows it to be used in user code while retaining the possibility to change the internal representation down the line.

* introduce raising constraints on stream callbacks - these constraints now give a warning when called with a callback that can raise exceptions (raising callbacks would crash 
* fix fail and its tests, which wasn't always given a good generic match
* work around nim bugs related to macro expansion of generic types
* make sure transports raise only `TransportError`-derived exceptions (and `CancelledError`)
2023-11-15 09:38:48 +01:00
Jacek Sieka 9759f01016
doc generation fixes (#464)
* doc generation fixes

* fix
2023-11-08 21:20:24 +01:00
Eugene Kabanov be2edab3ac
Consider ERROR_NETNAME_DELETED as ConnectionAbortedError. (#460) 2023-10-31 03:43:58 +02:00
Eugene Kabanov a70b145964
IPv4/IPv6 dualstack (#456)
* Initial commit.

* Fix tests.

* Fix linux compilation issue.

* Add getDomain() implementation.
Add getDomain() tests.
Add datagram tests.

* Fix style errors.

* Deprecate NetFlag.
Deprecate new flags in ServerFlags.
Add isAvailable().
Fix setDualstack() to ignore errors on `Auto`.
Updatetests.

* Deprecate some old procedures.
Improve datagram transport a bit.

* Address review comments, and fix tests.

* Fix setDescriptorBlocking() issue.
Recover connect() dualstack behavior.
Add test for connect() IPv6-[IPv4 mapped] addresses.

* Fix alignment code issue.
Fix TcpNoDelay was not available on Windows.

* Add dualstack support to HTTP/HTTPS client/server.
2023-10-30 15:27:50 +02:00
Eugene Kabanov 8375770fe5
Fix unreachable code places. (#459)
* Fix unreachable code.

* Use implicit returns instead.
2023-10-30 15:27:25 +02:00
Eugene Kabanov 2e8551b0d9
Cancellation fixes and tests. (#445)
* Add callTick and stream cancellation tests.

* Fix stepsAsync() test.

* Cancellation changes.

* Update and add more cancellation tests.

* Fix Posix shutdown call to handle ENOTCONN error.

* With new changes to to cancellation its now possible.

* Refactor testsoon.nim to not produce artifacts after tests are finished.

* Debugging MacOS issue.

* Adjust flaky test times.

* Fix issue.

* Add test for issue #334 which was also addressed in this PR.
Avoid `break` in problematic test.

* Add noCancelWait() call which prohibits cancellation.
Fix closeWait() calls to use noCancelWait() predicate.
Adding sleep to flaky MacOS test.

* Remove all debugging echoes.

* Fix cancelAndWait() which now could perform multiple attempts to cancel target Future (mustCancel behavior).

* Fix issues revealed by switch to different cancelAndWait().

* Address review comments.

* Fix testutils compilation warning.

* Rename callTick() to internalCallTick().

* Add some documentation comments.

* Disable flaky ratelimit test.

* Rename noCancelWait() to noCancel().
Address review comments.
2023-09-15 19:38:39 +03:00
Jacek Sieka a1eb30360b
fix invalid protocol casts (#430) 2023-08-04 08:08:34 +02:00
rockcavera 5c39bf47be
fixing unfreed memory leak with `freeAddrInfo()` (#425)
* fixing unfreed memory leak with `freeAddrInfo()`

* `freeaddrinfo` to `freeAddrInfo()`
2023-08-01 01:28:34 +03:00
Eugene Kabanov d214bcfb4f
Increase backlog defaults to maximum possible values. (#428) 2023-07-31 22:40:00 +03:00
Eugene Kabanov e04c042e8a
Add cross-platform shutdown() call and use it for HTTP server. (#420)
* Add cross-platform shutdown() call and use it for httpserver connection processing.

* Fix Posix compilation issues and warnings.
2023-07-19 20:33:28 +03:00
Eugene Kabanov 155d89450e
Trackers refactoring. (#416)
* Refactor chronos trackers to be more simple.

* Refactor trackers.
Add HTTP server trackers.
Refactor HTTP main processing loop.

* Compatibility fixes.
Add checkLeaks().

* Fix posix test issue.

* Add httpdebug module which introduces HTTP connection dumping helpers.
Add tests for it.

* Recover and deprecate old version of Trackers.

* Make public iterators to iterate over all tracker counters available.
Fix asynctests to use public iterators instead private one.
2023-07-14 13:35:08 +03:00
Eugene Kabanov 94ca0c3847
Fix remoteAddress() should raise TransportAbortedError too. (#409)
Add ENOTCONN as TransportAbortedError.
Refactor remoteAddress()/localAddress().
2023-06-26 16:31:09 +03:00
Jacek Sieka 1d6c309d77
clean up `Defect` (#404) 2023-06-05 22:21:50 +02:00
Jacek Sieka a6ac5f2213
exception warning fixes (#403) 2023-06-05 13:03:38 +02:00
Eugene Kabanov 02fda01bf2
Add facility to emulate signals on Windows. (#357)
* Initial Windows asyncproc part.
Deprecate usage of net/nativesockets in handles, asyncloop.
Introduce primitives with inheritable flag.

* Some posix fixes.

* More duplicate fixes.

* Fix AsyncProcessOption.EchoCommand option.
Remove isMainModule code.
Fix execCommand to use AsyncProcessOption.EvalCommand.

* Initial asyncproc tests commit.

* Some Posix fixes.

* Fix Posix crash.

* Add testsuite.
Fix osnet.nim to use proper declarations.
Fix timer.nim to use proper declarations.
Add Windows environment handling procedures.
Fix createAsyncPipe.
Add leaks tracking for AsyncProcessRef.

* Fix O_CLOEXEC constant value.

* Add file descriptors leak test.

* Remove commented code.
Refactor exceptions.
Fix compilation warnings.

* No exception ioselectors_kqueue initial commit.

* Some BSD fixes.
Linux refactoring of selectors.nim.

* Some fixes to move further.

* Last Linux fixes.

* Fixes for asyncloop to use 2nd version of selectors api.

* Add osutils.nim.

* Some fixes.

* Hardening resumeRead(), resumeWrite() and consumers.
Add ESRCH handling.
Introduce no-exception fromPipe2.

* Make Windows part exception-free and fix zombie race issue.

* createStreamServer() fixes.

* Upgrade asyncproc to use non-exception primitives.
Fix ioselectors_kqueue to use closeFd().

* Refactor accept() and acceptLoop() to be exception free.

* Deprecated some `result` usage.
Last fixes to make stream.nim exception free.
Use closeFd().
Refactor some loops to use handleEintr().

* Fix connect() forgot to unregister socket on error.

* All createAsyncSocket() sockets should be closed with unregisterAndCloseFd().

* Attempt to fix posix bug with incomplete output.

* Change number of runs in big test.

* Refactoring pipes creation. Attempt to fix "write error: Resource temporarily unavailable".

* Fix waitForExit(duration) code.
Fix test exit code.

* Fix Windows missing SIGTERM.

* Fix mistype.

* Fix compilation error.

* Attempt to fix Nim 1.6 issue.

* Eliminate Nim's WideCString usage to avoid any problems in future.

* Deprecate posix usage in osnet.

* Eliminate unused imports.

* Some debugging statements for investigation.

* Remove all the debugging statements.

* Remove testhelpers in favor of unittest2/asynctests.

* Fix flaky test.

* Make test more resilient to timings.

* Add memory size dump to CI.

* Refactor some blocks to use valueOr.
Make waitForExit to use kill() instead of terminate().

* Remove memory dumps.

* Fix peekProcessExitCode() blocks issue.

* Fix Windows issue.

* Add some debugging echoes.

* Fix compilation error.

* Add some debugging echoes.

* Add more helpers.

* Fix compilation error.

* waitForExit() is primary suspect.

* Remove fast-path for waitForExit.
Remove loop for reading signal events.

* Remove all debugging echoes.

* Return one debugging echo.

* Fix futures tests.

* Add test for multiple processes waiting to attempt stress test.

* Refactor ioselectors_epoll for better signalfd and process monitoring.
Add more race condition fixes to waitForExit.
Fix some Nim 1.6 warnings.

* Fix after rebase issues and warnings.

* Fix style issues.
Fix different Nim versions issues.
Workaround `signalfd` style issues.

* Add one more Linux signals workaround.
Add one more multiple processes test.

* Windows fixes.

* Remove unixPlatform define.
Fix WSAECONNABORTED for devel.

* Temporarily disable rate limit tests.
Fix more devel issues.

* Deprecate `hasThreadSupport` for ioselectors_kqueue.
Fix verifySelectParams issue.
Add exit codes test for multiple processes.
Fix osnet PtrToCstringConv warning.

* ioselectors_kqueue refactoring.

* Initial commit.

* Fix 1.2-1.4 compilation issue.

* Fix unused warning for testCtrlC() test.

* Post-rebase fixes.

* Restore master files.

* More fixes.

* Remove duplicates.

* Fix style mistake.

* Add more flexible pragmas.
2023-06-02 01:53:20 +03:00
gpicron 5be223e92c
allow reuse of port for UDP (#379) 2023-06-01 17:02:33 +03:00
Eugene Kabanov 956ae5af55
Fix import poison regression. (#388)
* Initial commit.

* One more linux fix.

* Eliminate posix from asyncloop.

* Fix Linux warnings.

* Fix MacOS issues.

* Fix BSD issues.

* Fix comma.

* Fix Windows issues.
2023-05-15 19:45:26 +03:00
Eugene Kabanov 8563c93673
Reexport osdefs where Sockaddr_storage was re-declared. (#385) 2023-05-12 07:11:05 +03:00
Eugene Kabanov ef94d75d61
Eliminate last 2 macos warnings. (#384) 2023-05-01 16:02:59 +03:00
Eugene Kabanov 5755d79b67
Eliminate warnings (#383)
* Eliminate warnings on Windows.

* Deprecated toHex() in osnet.
2023-05-01 00:46:15 +03:00
Eugene Kabanov 3118f8c1b2
Fix cast[pointer] issues on nim-devel. (#381)
* Fix cast[pointer] issues on nim-devel.

* More cast[T] fixes.

* Fix compilation error.

* Add oserrno.nim
Further fixes for Windows and Linux.

* MacOS fixes.

* More Windows fixes and attempt to fix 1.2, 1.4 branches.

* Implicitly import/export oserrno.

* Replace oserrno with osdefs.

* Return back oserrno.

* epoll to oserrno.

* datagram/stream to oserrno.

* common to oserrno.

* test to oserrno.
2023-04-30 08:20:08 +02:00
Eugene Kabanov e05d2f8e96
Fix some compilation warnings and expose tracker names. (#376)
* Fix asyncsync compilation warnings.

* Fix tracker names should be public.
2023-04-08 19:34:57 +03:00
Tanguy ab5a8c2e0f
Add `localAddress` support to `stream.connect` (#362)
* Add `localAddress` support to `stream.connect`

* fix windows

* TransportAddress() instead of AnyAddress

* tweak flags

* Better flags

* try to workaround nim 1.2 issue

* Handle ReusePort in createStreamServer and improve tests

* Rename ClientFlags to SocketFlags

---------

Co-authored-by: Diego <diego@status.im>
2023-04-03 15:34:35 +03:00
Eugene Kabanov b0af576c7c
Address #320 issue. (#372) 2023-03-24 17:34:45 +02:00
Eugene Kabanov f7835a192b
Fix osnet to not raise defect in ipnet. (#364)
* Debugging commit.

* Fix compilation issue.

* Remove debugging echoes.
2023-02-23 09:58:58 +02:00
Eugene Kabanov 0f70a6b8ee
OS definitions and exceptions changes. (AsyncProc part 1). (#361)
* Initial commit.

* Finalize Windows part.

* Finalize Posix part.

* Fix style issues.

* Move osnet declarations to osdefs.

* Move osnet declarations to osdefs 2.

* Fix Nim 1.2 issues.

* Fix 1.2 compilation issues 2.

* Address review comments.
Fix SignalFdInfo issue.
2023-02-21 11:48:36 +01:00
Eugene Kabanov 1e743dc415
IpNet refactoring (#354)
Remove `result` usage.
Remove cast[T] usage.
Add more helpers for TransportAddress checks.
Add isGlobal() helper to check if TransportAddress is globally accessed IP address.
Add tests.
Add more tests for sub.
Add TransportAddress.toHex().
2023-02-16 17:18:05 +01:00
Tanguy 3d532b6d4a
Enable TCP client to enable TCP_NODELAY (#340)
* Enable TCP client to enable TCP_NODELAY

* add missing return

* fix
2023-01-23 11:18:12 +02:00
Bung e9f8baa6ee
fix int set (#341) 2022-12-29 12:12:12 +01:00
Bung 5d0c725463
fix int set construction (#335) 2022-12-16 10:40:33 +01:00
Eugene Kabanov 6525f4ce1d
Fix some warnings. (#310)
* Fix some warnings messages.

* More warning fixes.

* Address review comments.
Fix MacOS issues.

* More warning fixes.

* More Windows specific fixes.

* Fix macos and windows warnings.

* Fix warnings in timer.nim
Refactor to remove `result`.
Improve performance and behavior of timer to string procedure.
Add tests.
2022-11-02 08:09:15 +01:00
Miran 93800879fd
make it more Nim 1.4+ compatible (#296) 2022-08-06 13:56:06 +03:00
Eugene Kabanov e9d0e2d208
Apply fix for recent Nim devel changes. (#300) 2022-07-30 12:48:58 +03:00