Reentrancy causes event reordering and stack explosions, in addition to
leading to hard-to-debug scenarios.
Since none of chronos is actually tested under reentrant conditions (ie
that all features such as exceptions, cancellations, buffer operations,
timers etc work reliably when the loop is reentered), this is hard to
support over time and prevents useful optimizations - this PR simply
detects and disallows the behaviour to remove the uncertainty,
simplifying reasoning about the event loop in general.
With these fixes, `transports`/`asyncsync` correctly propagate and document their raises information - generally, most transport functions (send etc) raise `TransportError` and `CancelledError` - `closeWait` is special in that it generally doesn't fail.
This PR introduces the syntax `Future[void].Raises([types])` to create the `InternalRaisesFuture` type with the correct encoding for the types - this allows it to be used in user code while retaining the possibility to change the internal representation down the line.
* introduce raising constraints on stream callbacks - these constraints now give a warning when called with a callback that can raise exceptions (raising callbacks would crash
* fix fail and its tests, which wasn't always given a good generic match
* work around nim bugs related to macro expansion of generic types
* make sure transports raise only `TransportError`-derived exceptions (and `CancelledError`)
* Fix waitFor() should not exit earlier last callback will be scheduled.
* Tune tests to use less memory.
* Fix `testutils`. There is no more last poll() needed.
* Update chronos/internal/asyncfutures.nim
---------
Co-authored-by: Jacek Sieka <jacek@status.im>
This PR replaces the global strict exception mode with an option to
handle `Exception` per function while at the same time enabling strict
exception checking globally by default as has been planned for v4.
`handleException` mode raises `AsyncExceptionError` to distinguish it
from `ValueError` which may originate from user code.
* remove obsolete 1.2 config options
Per discussion in
https://github.com/status-im/nim-chronos/pull/251#issuecomment-1559233139,
`async: (parameters..)` is introduced as a way to customize the async
transformation instead of relying on separate keywords (like
asyncraises).
Two parameters are available as of now:
`raises`: controls the exception effect tracking
`raw`: disables body transformation
Parameters are added to `async` as a tuple allowing more params to be
added easily in the future:
```nim:
proc f() {.async: (name: value, ...).}`
```
* introduce `asyncraises` to core future utilities
Similar to the introduction of `raises` into a codebase, `asyncraises`
needs to be introduced gradually across all functionality before
deriving benefit.
This is a first introduction along with utilities to manage raises lists
and transform them at compile time.
Several scenarios ensue:
* for trivial cases, adding `asyncraises` is enough and the framework
deduces the rest
* some functions "add" new asyncraises (similar to what `raise` does in
"normal" code) - for example `wait` may raise all exceptions of the
future passed to it and additionally a few of its own - this requires
extending the raises list
* som functions "remove" raises (similar to what `try/except` does) such
as `nocancel` with blocks cancellations and therefore reduce the raising
set
Both of the above cases are currently handled by a macro, but depending
on the situation lead to code organisation issues around return types
and pragma limitations - in particular, to keep `asyncraises`
backwards-compatibility, some code needs to exist in two versions which
somewhat complicates the implementation.
* add `asyncraises` versions for several `asyncfutures` utilities
* when assigning exceptions to a `Future` via `fail`, check at compile
time if possible and at runtime if not that the exception matches
constraints
* fix `waitFor` comments
* move async raises to separate module, implement `or`
* Exception tracking v2
* some fixes
* Nim 1.2 compat
* simpler things
* Fixes for libp2p
* Fixes for strictException
* better await exception check
* Fix for template async proc
* make async work with procTy
* FuturEx is now a ref object type
* add tests
* update test
* update readme
* Switch to asyncraises pragma
* Address tests review comments
* Rename FuturEx to RaiseTrackingFuture
* Fix typo
* Split asyncraises into async, asyncraises
* Add -d:chronosWarnMissingRaises
* Add comment to RaiseTrackingFuture
* Allow standalone asyncraises
* CheckedFuture.fail type checking
* First cleanup
* Remove useless line
* Review comments
* nimble: Remove #head from unittest2
* Remove implict raises: CancelledError
* Move checkFutureExceptions to asyncfutures2
* Small refacto
* small cleanup
* Complete in closure finally
* cleanup tests, add comment
* bump
* chronos is not compatible with nim 1.2 anymore
* re-add readme modifications
* fix special exception handlers
* also propagate excetion type in `read`
* `RaiseTrackingFuture` -> `InternalRaisesFuture`
Use internal naming scheme for RTF (this type should only be accessed
via asyncraises)
* use `internalError` for error reading
* oops
* 2.0 workarounds
* again
* remove try/finally for non-raising functions
* Revert "remove try/finally for non-raising functions"
This reverts commit 86bfeb5c97.
`finally` is needed if code returns early :/
* fixes
* avoid exposing `newInternalRaisesFuture` in manual macro code
* avoid unnecessary codegen for `Future[void]`
* avoid reduntant block around async proc body
* simplify body generation for forward declarations with comment but no
body
* avoid duplicate `gcsafe` annotiations
* line info for return at end of async proc
* expand tests
* fix comments, add defer test
---------
Co-authored-by: Jacek Sieka <jacek@status.im>
* Add callTick and stream cancellation tests.
* Fix stepsAsync() test.
* Cancellation changes.
* Update and add more cancellation tests.
* Fix Posix shutdown call to handle ENOTCONN error.
* With new changes to to cancellation its now possible.
* Refactor testsoon.nim to not produce artifacts after tests are finished.
* Debugging MacOS issue.
* Adjust flaky test times.
* Fix issue.
* Add test for issue #334 which was also addressed in this PR.
Avoid `break` in problematic test.
* Add noCancelWait() call which prohibits cancellation.
Fix closeWait() calls to use noCancelWait() predicate.
Adding sleep to flaky MacOS test.
* Remove all debugging echoes.
* Fix cancelAndWait() which now could perform multiple attempts to cancel target Future (mustCancel behavior).
* Fix issues revealed by switch to different cancelAndWait().
* Address review comments.
* Fix testutils compilation warning.
* Rename callTick() to internalCallTick().
* Add some documentation comments.
* Disable flaky ratelimit test.
* Rename noCancelWait() to noCancel().
Address review comments.
* Unpack `finally/defer` blocks and introduce explicit cleaning of objects.
Add request query to debug information.
* Unroll one more loop to avoid `break`.
Add test for query debug string.
* Fix cancellation behavior.
* Address review comments.
* Initial commit.
* Some fixes.
* More fixes.
* Add first test.
* Further fixes for MacOS/BSD.
* Fixes for Linux.
* Add proper tests.
* Lower number of tests.
* Add threadsync tests to test suite.
* There is no need to run tests when threads are off.
* Address review comments.
Fix the issue with multiple signals.
Add tests.
* Switch to use socketpair() instead of pipes.
Fix semaphoring issue on MacOS/BSD.
Add tests.
* Add single threaded fire/wait tests.
* Refactor chronos trackers to be more simple.
* Refactor trackers.
Add HTTP server trackers.
Refactor HTTP main processing loop.
* Compatibility fixes.
Add checkLeaks().
* Fix posix test issue.
* Add httpdebug module which introduces HTTP connection dumping helpers.
Add tests for it.
* Recover and deprecate old version of Trackers.
* Make public iterators to iterate over all tracker counters available.
Fix asynctests to use public iterators instead private one.
* Disable HTTP/1.1 pipeline support for HTTP/S server by default.
Add tests.
* Fix tests according to new behavior.
Fix tests to use random assigned OS ports instead of predefined.
* Fix flaky tests in testasyncstream.
* move `Future[T]` into its own module along with some basic accessors
* mark all fields internal, exposing only read-only versions under the
old names
* introduce `init`/`completed`/etc as a way of creating a future (vs
newFuture)
* introduce `LocationKind` for `SrcLoc` access
* don't expose `FutureList` unless future tracking is enabled
* introduce `chronosStrictFutureAccess` which controls a number of
additional `Defect` being raised when accessing Future fields in the
wrong state - this will become true in a future version
In this version, `Future[T]` backwards compatibility code remains in
`asyncfutures2` meaning that if only `chronos/futures` is imported, only
"new" API is available.
This branch is a refinement / less invasive / minimal version of
https://github.com/status-im/nim-chronos/pull/373.
* Initial Windows asyncproc part.
Deprecate usage of net/nativesockets in handles, asyncloop.
Introduce primitives with inheritable flag.
* Some posix fixes.
* More duplicate fixes.
* Fix AsyncProcessOption.EchoCommand option.
Remove isMainModule code.
Fix execCommand to use AsyncProcessOption.EvalCommand.
* Initial asyncproc tests commit.
* Some Posix fixes.
* Fix Posix crash.
* Add testsuite.
Fix osnet.nim to use proper declarations.
Fix timer.nim to use proper declarations.
Add Windows environment handling procedures.
Fix createAsyncPipe.
Add leaks tracking for AsyncProcessRef.
* Fix O_CLOEXEC constant value.
* Add file descriptors leak test.
* Remove commented code.
Refactor exceptions.
Fix compilation warnings.
* No exception ioselectors_kqueue initial commit.
* Some BSD fixes.
Linux refactoring of selectors.nim.
* Some fixes to move further.
* Last Linux fixes.
* Fixes for asyncloop to use 2nd version of selectors api.
* Add osutils.nim.
* Some fixes.
* Hardening resumeRead(), resumeWrite() and consumers.
Add ESRCH handling.
Introduce no-exception fromPipe2.
* Make Windows part exception-free and fix zombie race issue.
* createStreamServer() fixes.
* Upgrade asyncproc to use non-exception primitives.
Fix ioselectors_kqueue to use closeFd().
* Refactor accept() and acceptLoop() to be exception free.
* Deprecated some `result` usage.
Last fixes to make stream.nim exception free.
Use closeFd().
Refactor some loops to use handleEintr().
* Fix connect() forgot to unregister socket on error.
* All createAsyncSocket() sockets should be closed with unregisterAndCloseFd().
* Attempt to fix posix bug with incomplete output.
* Change number of runs in big test.
* Refactoring pipes creation. Attempt to fix "write error: Resource temporarily unavailable".
* Fix waitForExit(duration) code.
Fix test exit code.
* Fix Windows missing SIGTERM.
* Fix mistype.
* Fix compilation error.
* Attempt to fix Nim 1.6 issue.
* Eliminate Nim's WideCString usage to avoid any problems in future.
* Deprecate posix usage in osnet.
* Eliminate unused imports.
* Some debugging statements for investigation.
* Remove all the debugging statements.
* Remove testhelpers in favor of unittest2/asynctests.
* Fix flaky test.
* Make test more resilient to timings.
* Add memory size dump to CI.
* Refactor some blocks to use valueOr.
Make waitForExit to use kill() instead of terminate().
* Remove memory dumps.
* Fix peekProcessExitCode() blocks issue.
* Fix Windows issue.
* Add some debugging echoes.
* Fix compilation error.
* Add some debugging echoes.
* Add more helpers.
* Fix compilation error.
* waitForExit() is primary suspect.
* Remove fast-path for waitForExit.
Remove loop for reading signal events.
* Remove all debugging echoes.
* Return one debugging echo.
* Fix futures tests.
* Add test for multiple processes waiting to attempt stress test.
* Refactor ioselectors_epoll for better signalfd and process monitoring.
Add more race condition fixes to waitForExit.
Fix some Nim 1.6 warnings.
* Fix after rebase issues and warnings.
* Fix style issues.
Fix different Nim versions issues.
Workaround `signalfd` style issues.
* Add one more Linux signals workaround.
Add one more multiple processes test.
* Windows fixes.
* Remove unixPlatform define.
Fix WSAECONNABORTED for devel.
* Temporarily disable rate limit tests.
Fix more devel issues.
* Deprecate `hasThreadSupport` for ioselectors_kqueue.
Fix verifySelectParams issue.
Add exit codes test for multiple processes.
Fix osnet PtrToCstringConv warning.
* ioselectors_kqueue refactoring.
* Initial commit.
* Fix 1.2-1.4 compilation issue.
* Fix unused warning for testCtrlC() test.
* Post-rebase fixes.
* Restore master files.
* More fixes.
* Remove duplicates.
* Fix style mistake.
* Add more flexible pragmas.
* FutureState.Finished -> FutureState.Completed (to avoid name clash
with `proc finished` which means not-pending)
* deprecate `done` - to avoid additional confusion over completed vs
finished
* remove ad leftovers in stack trace formatting
* avoid some generic bloat
* avoid unnecessary allocations in `race`/`one`
* Initial commit.
* Some Linux fixes.
* Address review comments on Windows.
* Fix issues on Linux.
* Fix 1.2 issue and Windows warnings.
* Fix posix compilation issues.
* Fix cast[pointer] issues on nim-devel.
* More cast[T] fixes.
* Fix compilation error.
* Add oserrno.nim
Further fixes for Windows and Linux.
* MacOS fixes.
* More Windows fixes and attempt to fix 1.2, 1.4 branches.
* Implicitly import/export oserrno.
* Replace oserrno with osdefs.
* Return back oserrno.
* epoll to oserrno.
* datagram/stream to oserrno.
* common to oserrno.
* test to oserrno.
* Add `localAddress` support to `stream.connect`
* fix windows
* TransportAddress() instead of AnyAddress
* tweak flags
* Better flags
* try to workaround nim 1.2 issue
* Handle ReusePort in createStreamServer and improve tests
* Rename ClientFlags to SocketFlags
---------
Co-authored-by: Diego <diego@status.im>
This PR moves all compile-time configuration to a single module,
simplifying documentation and access to these features.
Upcomfing features may be enabled either individually, or through a new
`chronosPreviewV4` catch-all designed to allow code to be prepared for
increased strictness in future chronos releases.
`-d:chronosDebug` may be used to enable the existing debugging helpers
together.
* Allow passing in trustAnchors to newTLSClientAsyncStream
* Store X509TrustAnchors in a memory-secure, but memory-friendly way
* Remove forgotten import
* Change some waitFor to await in tests; add comment about trustAnchors reuse
* Remove use of result in newTrustAnchorStore
Remove `result` usage.
Remove cast[T] usage.
Add more helpers for TransportAddress checks.
Add isGlobal() helper to check if TransportAddress is globally accessed IP address.
Add tests.
Add more tests for sub.
Add TransportAddress.toHex().
Allow `type = proc(...) {.async.}` for pre-declaring async callback
types - in this case, the macro applies the correct transformation to
the return type and adds the appropriate exception / gcsafe annotations
* Address #329.
* Refactor AsyncStreamReader/Writer `state` helper functions.
Fix some compilation warnings in `debugutils`.
Add multiple writes into test to check consistency.
* Fix Linux issue.
Fix warnings.
Jacek Sieka
Eugene Kabanov
Tanguy
diegomrsantos
Matt Haggard