Commit Graph

669 Commits

Author SHA1 Message Date
Eugene Kabanov 1eb834a2f9
Fix `or` deadlock issue. (#517)
* Fix `or` should not create future with OwnCancelSchedule flag set.

* Fix `CancelledError` missing from raises list when both futures has empty raises list.

* Fix macros tests.
2024-03-05 17:33:46 +01:00
Etan Kissling 5dfa3fd7fa
fix conversion error with `or` on futures with `{.async: (raises: []).}` (#515)
```nim
import chronos

proc f(): Future[void] {.async: (raises: []).} =
  discard

discard f() or f() or f()
```

```
/Users/etan/Documents/Repos/nimbus-eth2/vendor/nim-chronos/chronos/internal/raisesfutures.nim(145, 44) union
/Users/etan/Documents/Repos/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/core/macros.nim(185, 28) []
/Users/etan/Documents/Repos/nimbus-eth2/test.nim(6, 13) template/generic instantiation of `or` from here
/Users/etan/Documents/Repos/nimbus-eth2/vendor/nim-chronos/chronos/internal/asyncfutures.nim(1668, 39) template/generic instantiation of `union` from here
/Users/etan/Documents/Repos/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/core/macros.nim(185, 28) Error: illegal conversion from '-1' to '[0..9223372036854775807]'
```

Fix by checking for `void` before trying to access `raises`
2024-03-05 13:53:12 +01:00
Eugene Kabanov 7b02247ce7
Add `--mm:refc` to `libbacktrace` test. (#505)
* Add `--mm:refc` to `libbacktrace` test.

* Make tests with `refc` to run before tests with default memory manager.
2024-02-14 19:23:15 +02:00
Eugene Kabanov 2e37a6e26c
Increase AsyncStream and Transport default buffer size from 4096 to 16384 bytes. (#506)
Make buffer sizes configurable at compile time.
2024-02-14 19:23:01 +02:00
cheatfate be4923be19
Strip debugging echo in threadsync tests. 2024-02-14 14:09:01 +02:00
Eugene Kabanov a81961a3c6
Fix HTTP server accept() loop exiting under heavy load. (#502)
* Add more specific accept() exceptions raised.
Add some refactoring to HTTP server code.

* Refactor acceptLoop.

* Print GC statistics in every failing test.

* Try to disable failing tests.
2024-02-14 14:05:19 +02:00
Jacek Sieka 8cf2d69aaa
Minimal threading docs (#493)
* Minimal threading docs

* compile examples with threads

* links
2024-02-14 08:27:09 +01:00
Eugene Kabanov 08db79fe63
Disable memory hungry tests in 32bit tests. (#503)
* Disable memory hungry tests in 32bit tests.

* Limit threadsync tests for 32bit.
2024-02-14 00:03:12 +02:00
Jacek Sieka 672db137b7
v4.0.0 (#494)
Features:

* Exception effects / raises for async procedures helping you write more
efficient leak-free code
* Cross-thread notification mechanism for suitable building channels,
queues and other multithreaded primitives
* Async process I/O
* IPv6 dual stack support
* HTTP middleware support alloing multiple services to share a single
http server
* A new [documentation web
site](https://status-im.github.io/nim-chronos/) covering the basics,
with several simple examples for getting started
* Implicit returns, support for `results.?` and other conveniences
* Rate limiter
* Revamped cancellation support with more control over the cancellation
process
* Efficiency improvements with `lent` and `sink`

See the [porting](https://status-im.github.io/nim-chronos/porting.html)
guides for porting code from earlier chronos releases (as well as
asyncdispatch)
2024-01-24 19:33:13 +02:00
Eugene Kabanov 09a0b11719
Make asyncproc use asyncraises. (#497)
* Make asyncproc use asyncraises.

* Fix missing asyncraises for waitForExit().
2024-01-23 08:34:10 +01:00
Jacek Sieka e296ae30c8
asyncraises for threadsync (#495)
* asyncraises for threadsync

* missing bracket

* missing exception
2024-01-20 16:56:57 +01:00
Jacek Sieka 3ca2c5e6b5
deprecate `callback=`, UDP fixes (fixes #491) (#492)
Using the callback setter may lead to callbacks owned by others being
reset, which is unexpected.

* don't crash on zero-length UDP writes
2024-01-19 09:21:10 +01:00
Jacek Sieka 1021a7d294
check leaks after every test (#487) 2024-01-18 14:34:16 +02:00
cheatfate 92acf68b04
Fix examples documentation. 2024-01-12 15:39:45 +02:00
Eugene Kabanov b02b9608c3
HTTP server middleware implementation. (#483)
* HTTP server middleware implementation and test.

* Address review comments.

* Address review comments.
2024-01-12 15:27:36 +02:00
Jacek Sieka f0a2d4df61
Feature flag for raises support (#488)
Feature flags allow consumers of chronos to target versions with and
without certain features via compile-time selection. The first feature
flag added is for raise tracking support.
2024-01-08 14:54:50 +01:00
Jacek Sieka e15dc3b41f
prevent http `closeWait` future from being cancelled (#486)
* simplify `closeWait` implementations
  * remove redundant cancellation callbacks
  * use `noCancel` to avoid forgetting the right future flags
* add a few missing raises trackers
* enforce `OwnCancelSchedule` on manually created futures that don't raise `CancelledError`
* ensure cancellations don't reach internal futures
2024-01-04 16:17:42 +01:00
Jacek Sieka 41f77d261e
Better line information on effect violation
We can capture the line info from the original future source and direct
violation errors there
2023-12-27 20:57:39 +01:00
Jacek Sieka 1598471ed2
add a test for `results.?` compatibility (#484)
Finally! (haha)
2023-12-21 15:52:16 +01:00
Eugene Kabanov c41599a6d6
Asyncraises HTTP layer V3 (#482)
* No Critical and Recoverable errors anymore.

* Recover raiseHttpCriticalError()

* Post-rebase fixes.

* Remove deprecated ResponseFence and getResponseFence().

* HttpProcessCallback and 2.

* Fix callback holder.

* Fix test issue.

* Fix backwards compatibility of `HttpResponse.state` field.
2023-12-09 06:50:35 +02:00
Jacek Sieka e38ceb5378
fix v3 backwards compatibility for callbacks (#481)
Because the callback types were used explicitly in some consumers of
chronos, the change of type introduces a backwards incompatibility
preventing a smooth transition to v4 for code that doesn't uses
`raises`.

This PR restores backwards compatibility at the expense of introducing a
new type with a potentially ugly name - that said, there is already
precedence for using numbered names to provide new error handling
strategy in chronos.
2023-12-04 14:19:29 +01:00
Jacek Sieka 48b2b08cfb
Update docs (#480)
* new mdbook version with built-in Nim highlighting support
* describe examples in a dedicated page
* fixes
2023-12-01 12:33:28 +01:00
Eugene Kabanov 28a100b135
Fix processing callback missing asyncraises. (#479) 2023-11-28 18:57:13 +02:00
Eugene Kabanov b18d471629
Asyncraises HTTP client/server. (#476)
* Fixes.

* Make httpcommon no-raises.

* Make httpbodyrw no-raises.

* Make multipart no-raises.

* Make httpdebug no-raises.

* Make httpagent no-raises.

* Make httpclient no-raises.

* Make httpserver/shttpserver no-raises.

* fix prepend/remove when E is noraises

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-11-21 11:01:44 +01:00
Jacek Sieka fa0bf405e6
varargs overloads (#477)
* varargs overloads

for convenience and compatibility

* no parameterless varargs calls with generic overloads
2023-11-20 12:04:28 +02:00
Jacek Sieka f03cdfcc40
futures: sinkify (#475)
This avoids copies here and there throughout the pipeline - ie
`copyString` and friends can often be avoided when moving things into
and out of futures

Annoyingly, one has to sprinkle the codebase liberally with `sink` and
`move` for the pipeline to work well - sink stuff _generally_ works
better in orc/arc

Looking at nim 1.6/refc, sink + local variable + move generates the best
code:

msg directly:
```nim
	T1_ = (*colonenv_).msg1; (*colonenv_).msg1 = copyStringRC1(msg);
```

local copy without move:
```nim
	T60_ = (*colonenv_).localCopy1; (*colonenv_).localCopy1 =
copyStringRC1(msg);
```

local copy with move:
```nim
	asgnRef((void**) (&(*colonenv_).localCopy1), msg);
```

Annoyingly, sink is also broken for refc+literals as it tries to
changes the refcount of the literal as part of the move (which shouldn't
be happening, but here we are), so we have to use a hack to find
literals and avoid moving them.
2023-11-19 18:29:09 +01:00
Eugene Kabanov 0b136b33c8
Asyncstreams asyncraises. (#472)
* Fix transports addresses functions should not return so many exceptions.

* Add raising `Defect` functions to AsyncQueue.

* Add raises/asyncraises into async streams.

* Remove `Safe` primitives.
Make AsyncStreamError to be ancestor of AsyncError.
Make AsyncStreamReader/Writer loops requirement to not raise any exceptions

* Remove `par` fields.

* Remove `par` fields from TLSStream.

* Attempt to lower memory usage.
2023-11-17 23:18:09 +01:00
Jacek Sieka 1306170255
dedicated exceptions for `Future.read` failures (#474)
Dedicated exceptions for `read` failures reduce the risk of mixing up
"user" exceptions with those of Future itself. The risk still exists, if
the user allows a chronos exception to bubble up explicitly.

Because `await` structurally guarantees that the Future is not `pending`
at the time of `read`, it does not raise this new exception.

* introduce `FuturePendingError` and `FutureCompletedError` when
`read`:ing a future of uncertain state
* fix `waitFor` / `read` to return `lent` values
* simplify code generation for `void`-returning async procs
* document `Raising` type helper
2023-11-17 13:45:17 +01:00
Jacek Sieka f5ff9e32ca
introduce asyncraises in transports/asyncsync (#470)
With these fixes, `transports`/`asyncsync` correctly propagate and document their raises information - generally, most transport functions (send etc) raise `TransportError` and `CancelledError` - `closeWait` is special in that it generally doesn't fail.

This PR introduces the syntax `Future[void].Raises([types])` to create the `InternalRaisesFuture` type with the correct encoding for the types - this allows it to be used in user code while retaining the possibility to change the internal representation down the line.

* introduce raising constraints on stream callbacks - these constraints now give a warning when called with a callback that can raise exceptions (raising callbacks would crash 
* fix fail and its tests, which wasn't always given a good generic match
* work around nim bugs related to macro expansion of generic types
* make sure transports raise only `TransportError`-derived exceptions (and `CancelledError`)
2023-11-15 09:38:48 +01:00
Jacek Sieka 24be151cf3
move docs to docs (#466)
* introduce user guide based on `mdbook`
* set up structure for adding simple `chronos` usage examples
* move most readme content to book
* ci deploys book and api guide automatically
* remove most of existing engine docs (obsolete)
2023-11-15 09:06:37 +01:00
Eugene Kabanov 9c93ab48de
Attempt to fix CI crash at Windows. (#465)
* Attempt to fix CI crash at Windows.
Remove all cast[string] and cast[seq[byte]] from the codebase.

* Address review comments.
2023-11-13 13:14:21 +02:00
Jacek Sieka 0d55475c29
`stew/results` -> `results` (#468) 2023-11-13 10:56:19 +01:00
Jacek Sieka f0eb7a0ae9
simplify tests (#469)
* simplify tests

`chronosPreviewV4` is obsolete

* oops
2023-11-13 10:54:37 +01:00
Eugene Kabanov 8156e2997a
Fix not enough memory on i386. (#467)
* Fix waitFor() should not exit earlier last callback will be scheduled.

* Tune tests to use less memory.

* Fix `testutils`. There is no more last poll() needed.

* Update chronos/internal/asyncfutures.nim

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-11-10 07:42:36 +01:00
Eugene Kabanov 9896316599
Remove deprecated AsyncEventBus. (#461)
* Remove deprecated AsyncEventBus.
Change number of tests for ThreadSignal.

* Recover 1000 tests count.
2023-11-09 18:01:43 +02:00
Jacek Sieka 9759f01016
doc generation fixes (#464)
* doc generation fixes

* fix
2023-11-08 21:20:24 +01:00
Jacek Sieka c252ce68d8
verbose test output on actions rerun (#462) 2023-11-08 16:15:11 +01:00
Jacek Sieka 53690f4717
run tests outside of nim compilation (#463)
else we need memory for both compiler and test
2023-11-08 16:14:33 +01:00
Jacek Sieka 5ebd771d35
per-function `Exception` handling (#457)
This PR replaces the global strict exception mode with an option to
handle `Exception` per function while at the same time enabling strict
exception checking globally by default as has been planned for v4.

`handleException` mode raises `AsyncExceptionError` to distinguish it
from `ValueError` which may originate from user code.

* remove obsolete 1.2 config options
2023-11-08 15:12:32 +01:00
Jacek Sieka cd6369c048
`asyncraises` -> `async: (raises: ..., raw: ...)` (#455)
Per discussion in
https://github.com/status-im/nim-chronos/pull/251#issuecomment-1559233139,
`async: (parameters..)` is introduced as a way to customize the async
transformation instead of relying on separate keywords (like
asyncraises).

Two parameters are available as of now:

`raises`: controls the exception effect tracking
`raw`: disables body transformation

Parameters are added to `async` as a tuple allowing more params to be
added easily in the future:
```nim:
proc f() {.async: (name: value, ...).}`
```
2023-11-07 12:12:59 +02:00
Eugene Kabanov be2edab3ac
Consider ERROR_NETNAME_DELETED as ConnectionAbortedError. (#460) 2023-10-31 03:43:58 +02:00
Eugene Kabanov a70b145964
IPv4/IPv6 dualstack (#456)
* Initial commit.

* Fix tests.

* Fix linux compilation issue.

* Add getDomain() implementation.
Add getDomain() tests.
Add datagram tests.

* Fix style errors.

* Deprecate NetFlag.
Deprecate new flags in ServerFlags.
Add isAvailable().
Fix setDualstack() to ignore errors on `Auto`.
Updatetests.

* Deprecate some old procedures.
Improve datagram transport a bit.

* Address review comments, and fix tests.

* Fix setDescriptorBlocking() issue.
Recover connect() dualstack behavior.
Add test for connect() IPv6-[IPv4 mapped] addresses.

* Fix alignment code issue.
Fix TcpNoDelay was not available on Windows.

* Add dualstack support to HTTP/HTTPS client/server.
2023-10-30 15:27:50 +02:00
Eugene Kabanov 8375770fe5
Fix unreachable code places. (#459)
* Fix unreachable code.

* Use implicit returns instead.
2023-10-30 15:27:25 +02:00
Tanguy 12dc36cfee
Update README regarding cancellation (#450)
* Update README regarding cancellation

* Apply suggestions from code review

Co-authored-by: Eugene Kabanov <eugene.kabanov@status.im>

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: Eugene Kabanov <eugene.kabanov@status.im>
2023-10-25 15:16:10 +02:00
Jacek Sieka f56d286687
introduce `asyncraises` to core future utilities (#454)
* introduce `asyncraises` to core future utilities

Similar to the introduction of `raises` into a codebase, `asyncraises`
needs to be introduced gradually across all functionality before
deriving benefit.

This is a first introduction along with utilities to manage raises lists
and transform them at compile time.

Several scenarios ensue:

* for trivial cases, adding `asyncraises` is enough and the framework
deduces the rest
* some functions "add" new asyncraises (similar to what `raise` does in
"normal" code) - for example `wait` may raise all exceptions of the
future passed to it and additionally a few of its own - this requires
extending the raises list
* som functions "remove" raises (similar to what `try/except` does) such
as `nocancel` with blocks cancellations and therefore reduce the raising
set

Both of the above cases are currently handled by a macro, but depending
on the situation lead to code organisation issues around return types
and pragma limitations - in particular, to keep `asyncraises`
backwards-compatibility, some code needs to exist in two versions which
somewhat complicates the implementation.

* add `asyncraises` versions for several `asyncfutures` utilities
* when assigning exceptions to a `Future` via `fail`, check at compile
time if possible and at runtime if not that the exception matches
constraints
* fix `waitFor` comments
* move async raises to separate module, implement `or`
2023-10-24 16:21:07 +02:00
Jacek Sieka e3c5a86a14
Introduce chronos/internals, move some code (#453)
* Introduce chronos/internals, move some code

This PR breaks the include dependencies between `asyncfutures2` and
`asyncmacros2` by moving the dispatcher and some other code to a new
module.

This step makes it easier to implement `asyncraises` support for future
utilities like `allFutures` etc avoiding the need to play tricks with
include order etc.

Future PR:s may further articulate the difference between "internal"
stuff subject to API breakage and regular public API intended for end
users (rather than advanced integrators).

* names

* windows fix
2023-10-17 20:25:25 +02:00
Jacek Sieka be9eef7a09
move test data to c file (#448)
* move test data to c file

allows compiling with nlvm

* more nlvm compat
2023-10-17 14:19:20 +02:00
Tanguy a759c11ce4
Raise tracking (#251)
* Exception tracking v2

* some fixes

* Nim 1.2 compat

* simpler things

* Fixes for libp2p

* Fixes for strictException

* better await exception check

* Fix for template async proc

* make async work with procTy

* FuturEx is now a ref object type

* add tests

* update test

* update readme

* Switch to asyncraises pragma

* Address tests review comments

* Rename FuturEx to RaiseTrackingFuture

* Fix typo

* Split asyncraises into async, asyncraises

* Add -d:chronosWarnMissingRaises

* Add comment to RaiseTrackingFuture

* Allow standalone asyncraises

* CheckedFuture.fail type checking

* First cleanup

* Remove useless line

* Review comments

* nimble: Remove #head from unittest2

* Remove implict raises: CancelledError

* Move checkFutureExceptions to asyncfutures2

* Small refacto

* small cleanup

* Complete in closure finally

* cleanup tests, add comment

* bump

* chronos is not compatible with nim 1.2 anymore

* re-add readme modifications

* fix special exception handlers

* also propagate excetion type in `read`

* `RaiseTrackingFuture` -> `InternalRaisesFuture`

Use internal naming scheme for RTF (this type should only be accessed
via asyncraises)

* use `internalError` for error reading

* oops

* 2.0 workarounds

* again

* remove try/finally for non-raising functions

* Revert "remove try/finally for non-raising functions"

This reverts commit 86bfeb5c97.

`finally` is needed if code returns early :/

* fixes

* avoid exposing `newInternalRaisesFuture` in manual macro code
* avoid unnecessary codegen for `Future[void]`
* avoid reduntant block around async proc body
* simplify body generation for forward declarations with comment but no
body
* avoid duplicate `gcsafe` annotiations
* line info for return at end of async proc

* expand tests

* fix comments, add defer test

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-10-17 14:18:14 +02:00
Tanguy 253bc3cfc0
Complete futures in closure finally (fix #415) (#449)
* Complete in closure finally

* cleanup tests, add comment

* handle defects

* don't complete future on defect

* complete future in test to avoid failure

* fix with strict exceptions

* fix regressions

* fix nim 1.6
2023-10-16 10:38:11 +02:00
Eugene Kabanov 2e8551b0d9
Cancellation fixes and tests. (#445)
* Add callTick and stream cancellation tests.

* Fix stepsAsync() test.

* Cancellation changes.

* Update and add more cancellation tests.

* Fix Posix shutdown call to handle ENOTCONN error.

* With new changes to to cancellation its now possible.

* Refactor testsoon.nim to not produce artifacts after tests are finished.

* Debugging MacOS issue.

* Adjust flaky test times.

* Fix issue.

* Add test for issue #334 which was also addressed in this PR.
Avoid `break` in problematic test.

* Add noCancelWait() call which prohibits cancellation.
Fix closeWait() calls to use noCancelWait() predicate.
Adding sleep to flaky MacOS test.

* Remove all debugging echoes.

* Fix cancelAndWait() which now could perform multiple attempts to cancel target Future (mustCancel behavior).

* Fix issues revealed by switch to different cancelAndWait().

* Address review comments.

* Fix testutils compilation warning.

* Rename callTick() to internalCallTick().

* Add some documentation comments.

* Disable flaky ratelimit test.

* Rename noCancelWait() to noCancel().
Address review comments.
2023-09-15 19:38:39 +03:00