* Introduce chronos/internals, move some code
This PR breaks the include dependencies between `asyncfutures2` and
`asyncmacros2` by moving the dispatcher and some other code to a new
module.
This step makes it easier to implement `asyncraises` support for future
utilities like `allFutures` etc avoiding the need to play tricks with
include order etc.
Future PR:s may further articulate the difference between "internal"
stuff subject to API breakage and regular public API intended for end
users (rather than advanced integrators).
* names
* windows fix
* Add callTick and stream cancellation tests.
* Fix stepsAsync() test.
* Cancellation changes.
* Update and add more cancellation tests.
* Fix Posix shutdown call to handle ENOTCONN error.
* With new changes to to cancellation its now possible.
* Refactor testsoon.nim to not produce artifacts after tests are finished.
* Debugging MacOS issue.
* Adjust flaky test times.
* Fix issue.
* Add test for issue #334 which was also addressed in this PR.
Avoid `break` in problematic test.
* Add noCancelWait() call which prohibits cancellation.
Fix closeWait() calls to use noCancelWait() predicate.
Adding sleep to flaky MacOS test.
* Remove all debugging echoes.
* Fix cancelAndWait() which now could perform multiple attempts to cancel target Future (mustCancel behavior).
* Fix issues revealed by switch to different cancelAndWait().
* Address review comments.
* Fix testutils compilation warning.
* Rename callTick() to internalCallTick().
* Add some documentation comments.
* Disable flaky ratelimit test.
* Rename noCancelWait() to noCancel().
Address review comments.
* Refactor chronos trackers to be more simple.
* Refactor trackers.
Add HTTP server trackers.
Refactor HTTP main processing loop.
* Compatibility fixes.
Add checkLeaks().
* Fix posix test issue.
* Add httpdebug module which introduces HTTP connection dumping helpers.
Add tests for it.
* Recover and deprecate old version of Trackers.
* Make public iterators to iterate over all tracker counters available.
Fix asynctests to use public iterators instead private one.
* move `Future[T]` into its own module along with some basic accessors
* mark all fields internal, exposing only read-only versions under the
old names
* introduce `init`/`completed`/etc as a way of creating a future (vs
newFuture)
* introduce `LocationKind` for `SrcLoc` access
* don't expose `FutureList` unless future tracking is enabled
* introduce `chronosStrictFutureAccess` which controls a number of
additional `Defect` being raised when accessing Future fields in the
wrong state - this will become true in a future version
In this version, `Future[T]` backwards compatibility code remains in
`asyncfutures2` meaning that if only `chronos/futures` is imported, only
"new" API is available.
This branch is a refinement / less invasive / minimal version of
https://github.com/status-im/nim-chronos/pull/373.
* Initial Windows asyncproc part.
Deprecate usage of net/nativesockets in handles, asyncloop.
Introduce primitives with inheritable flag.
* Some posix fixes.
* More duplicate fixes.
* Fix AsyncProcessOption.EchoCommand option.
Remove isMainModule code.
Fix execCommand to use AsyncProcessOption.EvalCommand.
* Initial asyncproc tests commit.
* Some Posix fixes.
* Fix Posix crash.
* Add testsuite.
Fix osnet.nim to use proper declarations.
Fix timer.nim to use proper declarations.
Add Windows environment handling procedures.
Fix createAsyncPipe.
Add leaks tracking for AsyncProcessRef.
* Fix O_CLOEXEC constant value.
* Add file descriptors leak test.
* Remove commented code.
Refactor exceptions.
Fix compilation warnings.
* No exception ioselectors_kqueue initial commit.
* Some BSD fixes.
Linux refactoring of selectors.nim.
* Some fixes to move further.
* Last Linux fixes.
* Fixes for asyncloop to use 2nd version of selectors api.
* Add osutils.nim.
* Some fixes.
* Hardening resumeRead(), resumeWrite() and consumers.
Add ESRCH handling.
Introduce no-exception fromPipe2.
* Make Windows part exception-free and fix zombie race issue.
* createStreamServer() fixes.
* Upgrade asyncproc to use non-exception primitives.
Fix ioselectors_kqueue to use closeFd().
* Refactor accept() and acceptLoop() to be exception free.
* Deprecated some `result` usage.
Last fixes to make stream.nim exception free.
Use closeFd().
Refactor some loops to use handleEintr().
* Fix connect() forgot to unregister socket on error.
* All createAsyncSocket() sockets should be closed with unregisterAndCloseFd().
* Attempt to fix posix bug with incomplete output.
* Change number of runs in big test.
* Refactoring pipes creation. Attempt to fix "write error: Resource temporarily unavailable".
* Fix waitForExit(duration) code.
Fix test exit code.
* Fix Windows missing SIGTERM.
* Fix mistype.
* Fix compilation error.
* Attempt to fix Nim 1.6 issue.
* Eliminate Nim's WideCString usage to avoid any problems in future.
* Deprecate posix usage in osnet.
* Eliminate unused imports.
* Some debugging statements for investigation.
* Remove all the debugging statements.
* Remove testhelpers in favor of unittest2/asynctests.
* Fix flaky test.
* Make test more resilient to timings.
* Add memory size dump to CI.
* Refactor some blocks to use valueOr.
Make waitForExit to use kill() instead of terminate().
* Remove memory dumps.
* Fix peekProcessExitCode() blocks issue.
* Fix Windows issue.
* Add some debugging echoes.
* Fix compilation error.
* Add some debugging echoes.
* Add more helpers.
* Fix compilation error.
* waitForExit() is primary suspect.
* Remove fast-path for waitForExit.
Remove loop for reading signal events.
* Remove all debugging echoes.
* Return one debugging echo.
* Fix futures tests.
* Add test for multiple processes waiting to attempt stress test.
* Refactor ioselectors_epoll for better signalfd and process monitoring.
Add more race condition fixes to waitForExit.
Fix some Nim 1.6 warnings.
* Fix after rebase issues and warnings.
* Fix style issues.
Fix different Nim versions issues.
Workaround `signalfd` style issues.
* Add one more Linux signals workaround.
Add one more multiple processes test.
* Windows fixes.
* Remove unixPlatform define.
Fix WSAECONNABORTED for devel.
* Temporarily disable rate limit tests.
Fix more devel issues.
* Deprecate `hasThreadSupport` for ioselectors_kqueue.
Fix verifySelectParams issue.
Add exit codes test for multiple processes.
Fix osnet PtrToCstringConv warning.
* ioselectors_kqueue refactoring.
* Initial commit.
* Fix 1.2-1.4 compilation issue.
* Fix unused warning for testCtrlC() test.
* Post-rebase fixes.
* Restore master files.
* More fixes.
* Remove duplicates.
* Fix style mistake.
* Add more flexible pragmas.
* Initial commit.
* Some Linux fixes.
* Address review comments on Windows.
* Fix issues on Linux.
* Fix 1.2 issue and Windows warnings.
* Fix posix compilation issues.
* Fix cast[pointer] issues on nim-devel.
* More cast[T] fixes.
* Fix compilation error.
* Add oserrno.nim
Further fixes for Windows and Linux.
* MacOS fixes.
* More Windows fixes and attempt to fix 1.2, 1.4 branches.
* Implicitly import/export oserrno.
* Replace oserrno with osdefs.
* Return back oserrno.
* epoll to oserrno.
* datagram/stream to oserrno.
* common to oserrno.
* test to oserrno.
This PR moves all compile-time configuration to a single module,
simplifying documentation and access to these features.
Upcomfing features may be enabled either individually, or through a new
`chronosPreviewV4` catch-all designed to allow code to be prepared for
increased strictness in future chronos releases.
`-d:chronosDebug` may be used to enable the existing debugging helpers
together.
* ioselectors_epoll() refactoring.
* ioselectors_kqueue() refactoring.
* ioselectors_poll() initial refactor.
* Remove `s.count` because it inconsistent and not used in `chronos`.
* Remove Windows version of select() engine.
* Add ability to switch event queue engine via `asyncEventEngine` command line option.
* Make it possible to switch between engines.
* Fix epoll regression.
* Fix poll() engine issues.
* Address review comments.
* Add proper trick.
* Address review comments.
* Bump version to 3.1.0.
Allow `type = proc(...) {.async.}` for pre-declaring async callback
types - in this case, the macro applies the correct transformation to
the return type and adds the appropriate exception / gcsafe annotations
* Fix some warnings messages.
* More warning fixes.
* Address review comments.
Fix MacOS issues.
* More warning fixes.
* More Windows specific fixes.
* Fix macos and windows warnings.
* Fix warnings in timer.nim
Refactor to remove `result`.
Improve performance and behavior of timer to string procedure.
Add tests.
Add IndexError crash test for Linux/MacOS.
Add Sentinel version of fix.
Add GetQueuedCompletionStatusEx() support for Windows, which allows to capture more then one event in single `poll()` call.
The socket selector holds a `seq` of per-descriptor data. When a reader
is registered, a pointer to a seq item is stored - when the `seq` grows,
this pointer becomes dangling and causes crashes like
https://github.com/status-im/nimbus-eth2/issues/3521.
It turns out that there already exist two mechanisms for passing user
data around - this PR simply removes one of them, saving on memory usage
and removing the need to store pointers to the `seq` data that become
dangling on resize.
* Change Future identifier type from `int` to `uint64`.
* Fix compilation error and tests.
* Add integer overflow test, to avoid confusion with next Nim versions, we expect that unsigned integers do not raise any exceptions if overflow happens.
* Switch from `uint64` to `uint` type.
* Add `uint` overflow tests.
* exception tracking
This PR adds minimal exception tracking to chronos, moving the goalpost
one step further.
In particular, it becomes invalid to raise exceptions from `callSoon`
callbacks: this is critical for writing correct error handling because
there's no reasonable way that a user of chronos can possibly _reason_
about exceptions coming out of there: the event loop will be in an
indeterminite state when the loop is executing an _random_ callback.
As expected, there are several issues in the error handling of chronos:
in particular, it will end up in an inconsistent internal state whenever
the selector loop operations fail, because the internal state update
functions are not written in an exception-safe way. This PR turns this
into a Defect, which probably is not the optimal way of handling things
- expect more work to be done here.
Some API have no way of reporting back errors to callers - for example,
when something fails in the accept loop, there's not much it can do, and
no way to report it back to the user of the API - this has been fixed
with the new accept flow - the old one should be deprecated.
Finally, there is information loss in the API: in composite operations
like `poll` and `waitFor` there's no way to differentiate internal
errors from user-level errors originating from callbacks.
* store `CatchableError` in future
* annotate proc's with correct raises information
* `selectors2` to avoid non-CatchableError IOSelectorsException
* `$` should never raise
* remove unnecessary gcsafe annotations
* fix exceptions leaking out of timer waits
* fix some imports
* functions must signal raising the union of all exceptions across all
platforms to enable cross-platform code
* switch to unittest2
* add `selectors2` which supercedes the std library version and fixes
several exception handling issues in there
* fixes
* docs, platform-independent eh specifiers for some functions
* add feature flag for strict exception mode
also bump version to 3.0.0 - _most_ existing code should be compatible
with this version of exception handling but some things might need
fixing - callbacks, existing raises specifications etc.
* fix AsyncCheck for non-void T
* Add callIdle primitive.
* Make single idle callback to be processed by single poll() step.
Add idleAsync() primitive to allow wait for "idle" time.
Refactor some `result` usage.
it seems that due to a naming conflict from asyncdispatch, callSoon is
deduced to raise exceptions even if it doesn't in modules that import
both, even indirectly - this patch randomly works around the issue with
some more overloads
* Fix cancellation behavior.
Fix some compilation warnings.
* Fix cancelAndWait() and add proper documentation.
* Add completed(Future) and done(Future) calls to check if Future[T] completed without an error.
* Add stepsAsync() and tests.
* Fix comments.
* Fix new primitive comment, to avoid usage for task switches.
* release callback memory early
this fixes a memory leak where a deleted callback may keep references
alive until the future is finished.
In particular, when using helpers like `or` which try to remove
themselves from the callback list when a dependent future is completed,
create a reference chain between all futures in the expression - in the
pathological case where one of the futures is completes only rarely (for
example a timeout or a cancellation task), the buildup will be
significant.
* Removing unnecessary asserts, and place comments instead.
This reverts commit e45ef32b5b.
Metrics implemented this way, with a lock inside the otherwise tight
event loop are not consistent with the chronos architecture that for
good or bad uses thread local variables to avoid them - the solution
does not have rough consensus behind it, and other avenues should be
explored for this generally useful functionality.
This reverts commit 284d677815.
There is no rough consensus for how this should be implemented and
whether it's needed indeed - a use case that cannot be solved with
a queue or a lock/event should be identified before pursuing this
functionality.
* Make Future tracking optional via -d:chronosDutureTracking compilation flag.
* Stack traces is now optional, use -d:chronosStackTraces.
* Fix mistypes and add test for chronosStackTrace option.
Jacek Sieka
Eugene Kabanov
Tanguy
Miran
Ștefan Talpalaru
Zahary Karadjov
markspanbroek
Kim De Mey