Commit Graph

358 Commits

Author SHA1 Message Date
Eugene Kabanov 8375770fe5
Fix unreachable code places. (#459)
* Fix unreachable code.

* Use implicit returns instead.
2023-10-30 15:27:25 +02:00
Jacek Sieka f56d286687
introduce `asyncraises` to core future utilities (#454)
* introduce `asyncraises` to core future utilities

Similar to the introduction of `raises` into a codebase, `asyncraises`
needs to be introduced gradually across all functionality before
deriving benefit.

This is a first introduction along with utilities to manage raises lists
and transform them at compile time.

Several scenarios ensue:

* for trivial cases, adding `asyncraises` is enough and the framework
deduces the rest
* some functions "add" new asyncraises (similar to what `raise` does in
"normal" code) - for example `wait` may raise all exceptions of the
future passed to it and additionally a few of its own - this requires
extending the raises list
* som functions "remove" raises (similar to what `try/except` does) such
as `nocancel` with blocks cancellations and therefore reduce the raising
set

Both of the above cases are currently handled by a macro, but depending
on the situation lead to code organisation issues around return types
and pragma limitations - in particular, to keep `asyncraises`
backwards-compatibility, some code needs to exist in two versions which
somewhat complicates the implementation.

* add `asyncraises` versions for several `asyncfutures` utilities
* when assigning exceptions to a `Future` via `fail`, check at compile
time if possible and at runtime if not that the exception matches
constraints
* fix `waitFor` comments
* move async raises to separate module, implement `or`
2023-10-24 16:21:07 +02:00
Jacek Sieka e3c5a86a14
Introduce chronos/internals, move some code (#453)
* Introduce chronos/internals, move some code

This PR breaks the include dependencies between `asyncfutures2` and
`asyncmacros2` by moving the dispatcher and some other code to a new
module.

This step makes it easier to implement `asyncraises` support for future
utilities like `allFutures` etc avoiding the need to play tricks with
include order etc.

Future PR:s may further articulate the difference between "internal"
stuff subject to API breakage and regular public API intended for end
users (rather than advanced integrators).

* names

* windows fix
2023-10-17 20:25:25 +02:00
Jacek Sieka be9eef7a09
move test data to c file (#448)
* move test data to c file

allows compiling with nlvm

* more nlvm compat
2023-10-17 14:19:20 +02:00
Tanguy a759c11ce4
Raise tracking (#251)
* Exception tracking v2

* some fixes

* Nim 1.2 compat

* simpler things

* Fixes for libp2p

* Fixes for strictException

* better await exception check

* Fix for template async proc

* make async work with procTy

* FuturEx is now a ref object type

* add tests

* update test

* update readme

* Switch to asyncraises pragma

* Address tests review comments

* Rename FuturEx to RaiseTrackingFuture

* Fix typo

* Split asyncraises into async, asyncraises

* Add -d:chronosWarnMissingRaises

* Add comment to RaiseTrackingFuture

* Allow standalone asyncraises

* CheckedFuture.fail type checking

* First cleanup

* Remove useless line

* Review comments

* nimble: Remove #head from unittest2

* Remove implict raises: CancelledError

* Move checkFutureExceptions to asyncfutures2

* Small refacto

* small cleanup

* Complete in closure finally

* cleanup tests, add comment

* bump

* chronos is not compatible with nim 1.2 anymore

* re-add readme modifications

* fix special exception handlers

* also propagate excetion type in `read`

* `RaiseTrackingFuture` -> `InternalRaisesFuture`

Use internal naming scheme for RTF (this type should only be accessed
via asyncraises)

* use `internalError` for error reading

* oops

* 2.0 workarounds

* again

* remove try/finally for non-raising functions

* Revert "remove try/finally for non-raising functions"

This reverts commit 86bfeb5c97.

`finally` is needed if code returns early :/

* fixes

* avoid exposing `newInternalRaisesFuture` in manual macro code
* avoid unnecessary codegen for `Future[void]`
* avoid reduntant block around async proc body
* simplify body generation for forward declarations with comment but no
body
* avoid duplicate `gcsafe` annotiations
* line info for return at end of async proc

* expand tests

* fix comments, add defer test

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-10-17 14:18:14 +02:00
Tanguy 253bc3cfc0
Complete futures in closure finally (fix #415) (#449)
* Complete in closure finally

* cleanup tests, add comment

* handle defects

* don't complete future on defect

* complete future in test to avoid failure

* fix with strict exceptions

* fix regressions

* fix nim 1.6
2023-10-16 10:38:11 +02:00
Eugene Kabanov 2e8551b0d9
Cancellation fixes and tests. (#445)
* Add callTick and stream cancellation tests.

* Fix stepsAsync() test.

* Cancellation changes.

* Update and add more cancellation tests.

* Fix Posix shutdown call to handle ENOTCONN error.

* With new changes to to cancellation its now possible.

* Refactor testsoon.nim to not produce artifacts after tests are finished.

* Debugging MacOS issue.

* Adjust flaky test times.

* Fix issue.

* Add test for issue #334 which was also addressed in this PR.
Avoid `break` in problematic test.

* Add noCancelWait() call which prohibits cancellation.
Fix closeWait() calls to use noCancelWait() predicate.
Adding sleep to flaky MacOS test.

* Remove all debugging echoes.

* Fix cancelAndWait() which now could perform multiple attempts to cancel target Future (mustCancel behavior).

* Fix issues revealed by switch to different cancelAndWait().

* Address review comments.

* Fix testutils compilation warning.

* Rename callTick() to internalCallTick().

* Add some documentation comments.

* Disable flaky ratelimit test.

* Rename noCancelWait() to noCancel().
Address review comments.
2023-09-15 19:38:39 +03:00
Eugene Kabanov 00614476c6
Address issue #443. (#447)
* Address issue #443.

* Address review comments.
2023-09-07 16:25:25 +03:00
Jacek Sieka e706167a53
add connect cancellation test (#444) 2023-09-05 13:41:52 +03:00
Eugene Kabanov 300fbaaf09
HttpAddress errors should be not only critical. (#446)
* Distinguish between resolve errors and check errors.

* Fix issues and add test for getHttpAddress() procedure.

* Address review comments.
2023-09-04 21:49:45 +03:00
Eugene Kabanov 60e6fc55bf
Fix #431. (#441) 2023-08-11 00:31:47 +03:00
Jacek Sieka a7f708bea8
futures: lentify (#413)
sometimes avoid copies when reading from `Future`
2023-08-09 17:27:17 +03:00
Eugene Kabanov 6c2ea67512
Unroll `defer`s and remove `break`s. (#440)
* Unpack `finally/defer` blocks and introduce explicit cleaning of objects.
Add request query to debug information.

* Unroll one more loop to avoid `break`.
Add test for query debug string.

* Fix cancellation behavior.

* Address review comments.
2023-08-09 10:57:49 +03:00
andri lim 38c31e21d3
fix type mismatch error in asyncstream join (#433) 2023-08-04 09:27:01 +02:00
Jacek Sieka a1eb30360b
fix invalid protocol casts (#430) 2023-08-04 08:08:34 +02:00
Eugene Kabanov 6b4f5a1d23
Recover `poll` engine and add tests. (#421)
* Initial commit.

* Fix one more place with deprecated constant.

* Fix testall and nimble file.

* Fix poll issue.

* Workaround Nim's faulty declaration of `poll()` and types on MacOS.

* Fix syntax errors.

* Fix MacOS post-rebase issue.

* Add more conditionals.

* Address review comments.

* Fix Nim 1.2 configuration defaults.
2023-08-01 12:56:08 +03:00
rockcavera 5c39bf47be
fixing unfreed memory leak with `freeAddrInfo()` (#425)
* fixing unfreed memory leak with `freeAddrInfo()`

* `freeaddrinfo` to `freeAddrInfo()`
2023-08-01 01:28:34 +03:00
Eugene Kabanov d214bcfb4f
Increase backlog defaults to maximum possible values. (#428) 2023-07-31 22:40:00 +03:00
Eugene Kabanov 926956bcbe
Add time used to establish HTTP client connection. (#427) 2023-07-30 12:43:25 +03:00
Eugene Kabanov 53e9f75735
Add some helpers for asyncproc. (#424)
* Initial commit.

* Adjust posix tests.

* Fix compilation issue.

* Attempt to fix flaky addProcess() test.
2023-07-28 11:54:53 +03:00
Eugene Kabanov f91ac169dc
Fix `NoVerifyServerName` do not actually disables SNI extension. (#423)
Fix HTTP client SSL/TLS error information is now part of connection error exception.
2023-07-23 19:40:57 +03:00
Eugene Kabanov 0277b65be2
Asynchronous thread notification mechanism. (#406)
* Initial commit.

* Some fixes.

* More fixes.

* Add first test.

* Further fixes for MacOS/BSD.

* Fixes for Linux.

* Add proper tests.

* Lower number of tests.

* Add threadsync tests to test suite.

* There is no need to run tests when threads are off.

* Address review comments.
Fix the issue with multiple signals.
Add tests.

* Switch to use socketpair() instead of pipes.
Fix semaphoring issue on MacOS/BSD.
Add tests.

* Add single threaded fire/wait tests.
2023-07-21 15:51:36 +03:00
Eugene Kabanov e04c042e8a
Add cross-platform shutdown() call and use it for HTTP server. (#420)
* Add cross-platform shutdown() call and use it for httpserver connection processing.

* Fix Posix compilation issues and warnings.
2023-07-19 20:33:28 +03:00
Eugene Kabanov d652c52142
Fix regression introduced by #416. (#419) 2023-07-17 16:39:41 +03:00
Eugene Kabanov 155d89450e
Trackers refactoring. (#416)
* Refactor chronos trackers to be more simple.

* Refactor trackers.
Add HTTP server trackers.
Refactor HTTP main processing loop.

* Compatibility fixes.
Add checkLeaks().

* Fix posix test issue.

* Add httpdebug module which introduces HTTP connection dumping helpers.
Add tests for it.

* Recover and deprecate old version of Trackers.

* Make public iterators to iterate over all tracker counters available.
Fix asynctests to use public iterators instead private one.
2023-07-14 13:35:08 +03:00
Jacek Sieka 3d80ea9fc7
fix EpollEvent memory layout (#411)
* fix EpollEvent memory layout

on x86_64, `EpollEvent` is packed - the upstream version is correct here

* copy-paste upstream instead

* Adjust lines to be 80 characters per line.

---------

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2023-07-06 19:51:59 +03:00
Jacek Sieka 0a6f5854a7
fix unused result warning on implict return (fixes #414) (#417)
* fix unused result warning on implict return

* describe `{.used.}` tradeoffs

* oops
2023-07-04 01:11:08 +03:00
Eugene Kabanov 94ca0c3847
Fix remoteAddress() should raise TransportAbortedError too. (#409)
Add ENOTCONN as TransportAbortedError.
Refactor remoteAddress()/localAddress().
2023-06-26 16:31:09 +03:00
Eugene Kabanov 73fd1206ab
Disable HTTP/1.1 pipeline support for HTTP/S server by default. (#410)
* Disable HTTP/1.1 pipeline support for HTTP/S server by default.
Add tests.

* Fix tests according to new behavior.
Fix tests to use random assigned OS ports instead of predefined.

* Fix flaky tests in testasyncstream.
2023-06-26 16:28:33 +03:00
Tanguy 47016961f5
Less flaky rate limit tests (#408) 2023-06-23 10:11:14 +02:00
Jacek Sieka 0035f4fa66
Introduce `chronos/futures` (#405)
* move `Future[T]` into its own module along with some basic accessors
* mark all fields internal, exposing only read-only versions under the
old names
* introduce `init`/`completed`/etc as a way of creating a future (vs
newFuture)
* introduce `LocationKind` for `SrcLoc` access
* don't expose `FutureList` unless future tracking is enabled
* introduce `chronosStrictFutureAccess` which controls a number of
additional `Defect` being raised when accessing Future fields in the
wrong state - this will become true in a future version

In this version, `Future[T]` backwards compatibility code remains in
`asyncfutures2` meaning that if only `chronos/futures` is imported, only
"new" API is available.

This branch is a refinement / less invasive / minimal version of
https://github.com/status-im/nim-chronos/pull/373.
2023-06-07 20:04:07 +02:00
Jacek Sieka 1d6c309d77
clean up `Defect` (#404) 2023-06-05 22:21:50 +02:00
Jacek Sieka 2ef34c7339
ratelimit: set cancellation earlier (#402)
future may be completed by worker before cancellation is set
2023-06-05 13:47:38 +02:00
Jacek Sieka a6ac5f2213
exception warning fixes (#403) 2023-06-05 13:03:38 +02:00
Jacek Sieka 157ca4fea5
Support implicit returns (#401)
* based on https://github.com/nim-lang/Nim/pull/21898
* also fixes generic Future[T] where T ends up being `void`
2023-06-05 13:02:13 +02:00
Eugene Kabanov 6e80f16e0c
Fix recently introduced imports poisoning. (#400)
* Fix import poison introduced.

* Move export to handles, and annotate it.
2023-06-03 19:36:06 +03:00
Eugene Kabanov 02fda01bf2
Add facility to emulate signals on Windows. (#357)
* Initial Windows asyncproc part.
Deprecate usage of net/nativesockets in handles, asyncloop.
Introduce primitives with inheritable flag.

* Some posix fixes.

* More duplicate fixes.

* Fix AsyncProcessOption.EchoCommand option.
Remove isMainModule code.
Fix execCommand to use AsyncProcessOption.EvalCommand.

* Initial asyncproc tests commit.

* Some Posix fixes.

* Fix Posix crash.

* Add testsuite.
Fix osnet.nim to use proper declarations.
Fix timer.nim to use proper declarations.
Add Windows environment handling procedures.
Fix createAsyncPipe.
Add leaks tracking for AsyncProcessRef.

* Fix O_CLOEXEC constant value.

* Add file descriptors leak test.

* Remove commented code.
Refactor exceptions.
Fix compilation warnings.

* No exception ioselectors_kqueue initial commit.

* Some BSD fixes.
Linux refactoring of selectors.nim.

* Some fixes to move further.

* Last Linux fixes.

* Fixes for asyncloop to use 2nd version of selectors api.

* Add osutils.nim.

* Some fixes.

* Hardening resumeRead(), resumeWrite() and consumers.
Add ESRCH handling.
Introduce no-exception fromPipe2.

* Make Windows part exception-free and fix zombie race issue.

* createStreamServer() fixes.

* Upgrade asyncproc to use non-exception primitives.
Fix ioselectors_kqueue to use closeFd().

* Refactor accept() and acceptLoop() to be exception free.

* Deprecated some `result` usage.
Last fixes to make stream.nim exception free.
Use closeFd().
Refactor some loops to use handleEintr().

* Fix connect() forgot to unregister socket on error.

* All createAsyncSocket() sockets should be closed with unregisterAndCloseFd().

* Attempt to fix posix bug with incomplete output.

* Change number of runs in big test.

* Refactoring pipes creation. Attempt to fix "write error: Resource temporarily unavailable".

* Fix waitForExit(duration) code.
Fix test exit code.

* Fix Windows missing SIGTERM.

* Fix mistype.

* Fix compilation error.

* Attempt to fix Nim 1.6 issue.

* Eliminate Nim's WideCString usage to avoid any problems in future.

* Deprecate posix usage in osnet.

* Eliminate unused imports.

* Some debugging statements for investigation.

* Remove all the debugging statements.

* Remove testhelpers in favor of unittest2/asynctests.

* Fix flaky test.

* Make test more resilient to timings.

* Add memory size dump to CI.

* Refactor some blocks to use valueOr.
Make waitForExit to use kill() instead of terminate().

* Remove memory dumps.

* Fix peekProcessExitCode() blocks issue.

* Fix Windows issue.

* Add some debugging echoes.

* Fix compilation error.

* Add some debugging echoes.

* Add more helpers.

* Fix compilation error.

* waitForExit() is primary suspect.

* Remove fast-path for waitForExit.
Remove loop for reading signal events.

* Remove all debugging echoes.

* Return one debugging echo.

* Fix futures tests.

* Add test for multiple processes waiting to attempt stress test.

* Refactor ioselectors_epoll for better signalfd and process monitoring.
Add more race condition fixes to waitForExit.
Fix some Nim 1.6 warnings.

* Fix after rebase issues and warnings.

* Fix style issues.
Fix different Nim versions issues.
Workaround `signalfd` style issues.

* Add one more Linux signals workaround.
Add one more multiple processes test.

* Windows fixes.

* Remove unixPlatform define.
Fix WSAECONNABORTED for devel.

* Temporarily disable rate limit tests.
Fix more devel issues.

* Deprecate `hasThreadSupport` for ioselectors_kqueue.
Fix verifySelectParams issue.
Add exit codes test for multiple processes.
Fix osnet PtrToCstringConv warning.

* ioselectors_kqueue refactoring.

* Initial commit.

* Fix 1.2-1.4 compilation issue.

* Fix unused warning for testCtrlC() test.

* Post-rebase fixes.

* Restore master files.

* More fixes.

* Remove duplicates.

* Fix style mistake.

* Add more flexible pragmas.
2023-06-02 01:53:20 +03:00
Eugene Kabanov 315a27236c
Recover cancellation cleanup for AsyncEvent wait(). (#398)
* Recover cancellation cleanup for AsyncEvent wait().

* Address review comments.
2023-06-01 18:09:49 +03:00
gpicron 5be223e92c
allow reuse of port for UDP (#379) 2023-06-01 17:02:33 +03:00
Jacek Sieka e436f20b33
Memory cleanups (#395)
* Avoid `FutureContinue` bloat by moving type punning into iterator
(which is typed anyway)
* clear closure iterator from future when iteration is done also on
cancellation / exception
* remove some redundant local variables in `await`
* document `futureContinue` flow
2023-05-31 07:24:25 +02:00
Eugene Kabanov 6523f741a6
Assertion and bugfix. (#397) 2023-05-30 17:45:16 +02:00
Eugene Kabanov 36ab9c8e4a
Fix sigprocmask declaration missing when threads are disabled. (#396) 2023-05-30 07:57:57 +02:00
Jacek Sieka b65b85533a
Future cleanup (#393)
* FutureState.Finished -> FutureState.Completed (to avoid name clash
with `proc finished` which means not-pending)
* deprecate `done` - to avoid additional confusion over completed vs
finished
* remove ad leftovers in stack trace formatting
* avoid some generic bloat
* avoid unnecessary allocations in `race`/`one`
2023-05-25 17:31:35 +02:00
Jacek Sieka 2fa6df0880
asyncmacro: code cleanups (#392)
* prefer `let`
* prefer expressions
* renames
2023-05-23 19:45:12 +02:00
Eugene Kabanov 148ddf49c2
Asyncproc (Part 3/3) (#374)
* Initial commit.

* Some Linux fixes.

* Address review comments on Windows.

* Fix issues on Linux.

* Fix 1.2 issue and Windows warnings.

* Fix posix compilation issues.
2023-05-23 12:39:35 +02:00
Eugene Kabanov f748387462
Add HTTP client helper to read `text/event-stream` streaming response. (#375) 2023-05-19 17:25:22 +03:00
Bung 38cc233700
fix missing sigprocmask import (#390) 2023-05-19 13:43:33 +03:00
Eugene Kabanov 61d52b1ef8
Export SocketFlags for HTTP client. (#389) 2023-05-18 10:10:01 +03:00
Eugene Kabanov 956ae5af55
Fix import poison regression. (#388)
* Initial commit.

* One more linux fix.

* Eliminate posix from asyncloop.

* Fix Linux warnings.

* Fix MacOS issues.

* Fix BSD issues.

* Fix comma.

* Fix Windows issues.
2023-05-15 19:45:26 +03:00
Eugene Kabanov 8aa8ee8822
Add flag to completely disable HTTP/1.1 pipelining support. (#387)
* Add flag for HTTP/1.1 pipeline disable.

* Switch from NoHttp11Pipeline to Http11Pipeline.
2023-05-12 17:00:37 +03:00