Commit Graph

233 Commits

Author SHA1 Message Date
Jacek Sieka 0fc82049ac enable --styleCheck:usages 2021-12-27 17:33:25 +02:00
Jacek Sieka c25fa1f6cd
posix: fast path for write ()
When `write` is called on a `StreamTransport`, the current sequence of
operations is:

* copy data to queue
* register for "write" event notification
* return unfinished future to `write` caller
* wait for "write" notification (in `poll`)
* perform one `send`
* wait for notification again if there's more data to write
* complete the future

In this PR, we introduce a fast path for writing:

* If the queue is empty, try to send as much data as possible
  * If all data is sent, return completed future without `poll` round
  * If there's more data to send than can be sent in one go, add the
rest to queue
* If the queue is not empty, enqueue as above
* When notified that write is possible, keep writing until OS buffer is
full before waiting for event again

The fast path provides significant performance benefits when there are
many small writes, such as when sending gossip to many peers, by
avoiding the poll loop and data copy on each send.

Also fixes an issue where the socket would not be removed from the
writer set if there were pending writes on close.
2021-12-08 11:35:27 +01:00
cheatfate 661eae5732
Fix identifiers to be `uint` instead of `uint64`. 2021-10-21 17:46:06 +03:00
Eugene Kabanov 3bc0bc36f7
Change Future identifier type from `int` to `uint`. ()
* Change Future identifier type from `int` to `uint64`.

* Fix compilation error and tests.

* Add integer overflow test, to avoid confusion with next Nim versions, we expect that unsigned integers do not raise any exceptions if overflow happens.

* Switch from `uint64` to `uint` type.

* Add `uint` overflow tests.
2021-10-21 17:22:11 +03:00
Eugene Kabanov 80102a3b6a
Eventbus implementation ()
* Initial commit.

* Move AsyncEventBus implementation to asyncsync.nim
Add subscriptAll(), waitAllEvents() primitives.
Add emitWait() primitive.
Add emitter source location implementation.
Add tests.
2021-09-15 16:55:15 +03:00
Eugene Kabanov bbbcb55493
Server-side events implementation. ()
Fix keep-alive issue.
Refactor some prepareXXX routines.
2021-09-14 20:32:58 +03:00
Eugene Kabanov 5034f0a5a6
Fix issue ()
* Address issue  and add tests for it.
Some cosmetic refactoring.

* Fix *nix tests.
2021-09-05 00:53:27 +03:00
Eugene Kabanov ef2430d08d
Add `Accept` header handling to httpserver.nim. ()
* Add `Accept` header handling to httpserver.nim.
Add simple test suite.
Bump version to 3.0.6.

* Fix compilation error.
2021-07-28 17:08:38 +03:00
Eugene Kabanov 15137f71c3
Basic authorization implementation for HTTP client. ()
* Basic authorization implementation for HTTP client.
Add tests for basic authorization.

* Bump chronos version to 3.0.5.
2021-06-29 02:38:08 +03:00
Jacek Sieka 252e5d0d50
simplify future tests () 2021-06-04 12:28:04 +02:00
Eugene Kabanov 7ccb170f7a
Enable comma as array delimiter and adding tests. ()
* Enable comma as array delimiter and adding tests.

* Bump version to 3.0.4.
2021-05-17 22:39:24 +03:00
Eugene Kabanov be184a815c
Httpclient ()
* Initial commit.

* Some refactoring.

* Allow boundstream to accept uint64.
Fix httpserver and asyncstream tests to follow new uint64 requirement.

* send() and getBodyBytes() implementations.

* Add closeWait for response and request.
Refactor finish/close flow.

* Changes in state machine
Add first test.

* Missing test file.

* Fixed tests
Add http leaking trackers and tests.

* Some fixes in multipart.
Fix automatic Content-Length header for requests with body.
Fix getBodyBytes() assertions.
Merging tests to main suite.

* Post rebase fixes.

* Fix tests big message generation.

* Fix response state management and leaks for getBodyXXX() procedures.

* Add redirection support to client and server.
Add fetch(url) procedure with redirection support.
Add tests for redirection.
2021-05-10 10:26:36 +03:00
Eugene Kabanov d270dba8a3
Fix unused warnings, result, asyncCheck and 80 cpl ()
* Fix sources to follow 80 characters per line.
Fix unused compilation warnings.
Refactor (remove result) handles.nim.
Fix tests to use asyncSpawn instead of asyncCheck.

* Fix handles for Unix platforms.
2021-05-07 23:52:24 +03:00
Eugene Kabanov fed6b0ac92
Restore functionality of zero-sized bounded reader/writer streams. ()
* Restore functionality of zero-sized bounded reader/writer streams.
Adding tests for it.

* run build_nim.sh unconditionally

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-05-07 18:52:44 +03:00
Eugene Kabanov 39f4060e07
Refactor and optimization of BoundedStream. ()
Documentation for BoundedStream.
Fix HttpServer bounding.
2021-04-26 14:05:37 +03:00
Eugene Kabanov 833d968782
Fix upload() issue and adding tests. () 2021-04-24 20:32:21 +03:00
Eugene Kabanov e6ba00cb81
Fix reading could stuck on very big chunked headers and tests. () 2021-04-22 15:32:28 +03:00
Eugene Kabanov aab1e30a72
Refactor common.nim and add more resolve procedures. ()
* Refactor common.nim to remove `result` usage.
Fix comparison of TransportAddress issue.
Add resolveTAddress procedures for both IPv4 and IPv6 addresses.
Fix tests.

* Bump version to 3.0.2.
2021-04-10 00:39:54 +03:00
Jacek Sieka 4abd7a5645
exception tracking ()
* exception tracking

This PR adds minimal exception tracking to chronos, moving the goalpost
one step further.

In particular, it becomes invalid to raise exceptions from `callSoon`
callbacks: this is critical for writing correct error handling because
there's no reasonable way that a user of chronos can possibly _reason_
about exceptions coming out of there: the event loop will be in an
indeterminite state when the loop is executing an _random_ callback.

As expected, there are several issues in the error handling of chronos:
in particular, it will end up in an inconsistent internal state whenever
the selector loop operations fail, because the internal state update
functions are not written in an exception-safe way. This PR turns this
into a Defect, which probably is not the optimal way of handling things
- expect more work to be done here.

Some API have no way of reporting back errors to callers - for example,
when something fails in the accept loop, there's not much it can do, and
no way to report it back to the user of the API - this has been fixed
with the new accept flow - the old one should be deprecated.

Finally, there is information loss in the API: in composite operations
like `poll` and `waitFor` there's no way to differentiate internal
errors from user-level errors originating from callbacks.

* store `CatchableError` in future
* annotate proc's with correct raises information
* `selectors2` to avoid non-CatchableError IOSelectorsException
* `$` should never raise
* remove unnecessary gcsafe annotations
* fix exceptions leaking out of timer waits
* fix some imports
* functions must signal raising the union of all exceptions across all
platforms to enable cross-platform code
* switch to unittest2
* add `selectors2` which supercedes the std library version and fixes
several exception handling issues in there

* fixes

* docs, platform-independent eh specifiers for some functions

* add feature flag for strict exception mode

also bump version to 3.0.0 - _most_ existing code should be compatible
with this version of exception handling but some things might need
fixing - callbacks, existing raises specifications etc.

* fix AsyncCheck for non-void T
2021-03-24 10:08:33 +01:00
Eugene Kabanov c8eefb9382
Split HTTPS and HTTP servers. ()
* Split HTTPS and HTTP servers.

* Fix review commens
2021-03-17 15:40:40 +02:00
Eugene Kabanov f774644129
Fix integer decoding overflow issue. ()
Switch to stew.base10 procedures.
Adjust tests to follow new behavior.
Bump version to 2.6.1.
2021-03-06 00:22:32 +02:00
cheatfate d6a39e46e1
Fix tests. 2021-02-21 17:34:55 +02:00
Eugene Kabanov d49e0a9c47
Add drop() implementation (was missed) to httpserver and fix transport leak. ()
* Add drop() implementation (was missed).
Add tests for drop().
Fix transport leak because of drop().

* Fix future leak.
2021-02-19 14:07:20 +02:00
cheatfate 25688cd0aa Add hexValue tests. 2021-02-18 22:16:04 +02:00
cheatfate eb81018d02 Address review comments and fix issues found.
Adding more tests.
2021-02-18 22:16:04 +02:00
cheatfate fc0d1bcb43 Address review comments. 2021-02-18 22:16:04 +02:00
cheatfate 3d74c5cdd2 Add tests for getContentEncoding() and getTransferEncoding(). 2021-02-18 22:16:04 +02:00
cheatfate 4406ab7072 Add positive decimal integer parsing tests. 2021-02-18 22:16:04 +02:00
cheatfate 3495122867 Fix getMultipartBoundary() issues and add tests for it. 2021-02-18 22:16:04 +02:00
cheatfate 9b48496513 Fix compilation error with nimcrypto/utils import. 2021-02-18 22:16:04 +02:00
cheatfate 3569e4d553 testutils must be last executed test. 2021-02-18 22:16:04 +02:00
cheatfate 3e9ffae407 Properly fix case when request body size exceeds maximum allowed size. 2021-02-18 22:16:04 +02:00
cheatfate 970e5641d7 Add less strict rules for BoundStream reader/writer. 2021-02-18 22:16:04 +02:00
cheatfate d43a9cb92d HttpServer now supports TLS.
Some TLSStream fixes to properly support EOF.
Some HttpServer to properly support TLS handshake problems.
HttpServer test suite for HTTPS.
2021-02-18 22:16:04 +02:00
cheatfate 1a3e9162a4 Fix multipart end of message handling.
Add apps.nim.
Change copyrights dates.
Add httpserver tests to test suite.
2021-02-18 22:16:04 +02:00
cheatfate 0b396c34d8 Add newline. 2021-02-18 22:16:04 +02:00
cheatfate 2defc4b822 Add sequence of bytes as boundary to BoundStream and tests. 2021-02-18 22:16:04 +02:00
cheatfate 13eddf382d Simplification and fixes for TLSStream state machine. 2021-02-18 22:16:04 +02:00
cheatfate e8d2a3ca0a Attempt . 2021-02-18 22:16:04 +02:00
cheatfate 49fd70f504 Attempt . 2021-02-18 22:16:04 +02:00
cheatfate bb176ba574 Investigation of Linux freezes . 2021-02-18 22:16:04 +02:00
cheatfate ac8b11d6ca close() procedure should not raise, otherwise its impossible to cleanup. 2021-02-18 22:16:04 +02:00
cheatfate 0cb6840f03 Big refactoring of AsyncStreams.
1. Implement all read() primitives using readLoop() like it was done in streams.
2. Fix readLine() bug.
3. Add readMessage() primitive.
4. Fixing exception hierarchy, handling code and simplification of (break/continue + exception).
5. Fix TLSStream closure procedure.
6. Add BoundedStream stream and tests.
7. Remove `result` usage from the code.
2021-02-18 22:16:04 +02:00
cheatfate 46c0bf3c5a
Fix define to skip flaky test on MacOS. 2020-11-27 03:50:38 +02:00
Eugene Kabanov bca5559c6a
Race() call ()
* Add `race` procedure call which extends `one` with FutureBase.

* Fix race() and add test procedures.
2020-11-27 00:50:55 +02:00
Eugene Kabanov ac9b3e304f
Fix deadlock for pending write() calls on transport close. ()
Add tests for read() and write() deadlocks.
2020-11-18 11:30:33 +02:00
cheatfate 493cb1dbfd Fix behavior of wait() and withTimeout() calls to cancel and wait for result of cancelled Future[T].
Add tests.
2020-11-17 19:28:52 +02:00
Eugene Kabanov b5915ecd29
Small fixes ()
* Fix cancellation behavior.
Fix some compilation warnings.

* Fix cancelAndWait() and add proper documentation.

* Add completed(Future) and done(Future) calls to check if Future[T] completed without an error.

* Add stepsAsync() and tests.

* Fix comments.

* Fix new primitive comment, to avoid usage for task switches.
2020-11-13 14:22:58 +02:00
Eugene Kabanov d3018ae908
Fix TLSStream SSL errors while in handshake could stuck connection. ()
Fix expired SSL certificate in tests.
2020-10-13 02:12:52 +03:00
Eugene Kabanov 2134980744
Fix AsyncLock.locked flag to be consistent. ()
* Fix `locked` flag to be more consistent.
Refactor AsyncLock to not use `result`.
Add test for `locked` flag.

* Fixes.

* Fix imports.

* Fix multiple release() without scheduler.
Add more tests.

* Fix review comments.
2020-09-10 23:28:20 +03:00
Jacek Sieka 483054cda6
small fixes ()
* small fixes

* more efficient codegen for nil check (much less code)
* release futures earlier in AsyncEvent
* release finished future earlier in AsyncQueue
* avoid searches for futures (deque variant unused / broken)
* avoid catching defects

* Fix AsyncEvent test, because of optimization.

* delete fix

* avoid seq allocs

* Keep style consistent with other code.
Refactor AsyncEvent and AsyncQueue to not use `result` keyword.

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2020-09-10 11:39:10 +03:00
Eugene Kabanov 126ea4bc56
Fix newFuture[T] location source for generated async procedures. ()
Add tests to for locations.
2020-09-03 14:37:53 +03:00
Eugene Kabanov a5442edfc0
Add asyncSpawn() procedure and tests. ()
Deprecated asyncDiscard() procedure.
Bump version to 2.5.2.
2020-09-01 21:41:18 +03:00
Zahary Karadjov 03f4a26829 Revert "yieldAsync() ()"
This reverts commit 284d677815.

There is no rough consensus for how this should be implemented and
whether it's needed indeed - a use case that cannot be solved with
a queue or a lock/event should be identified before pursuing this
functionality.
2020-08-16 01:48:25 +03:00
Ștefan Talpalaru 284d677815
yieldAsync() () 2020-08-10 15:31:21 +02:00
cheatfate 3968f09ae1
Skip flaky test on MacOS. 2020-07-17 09:38:58 +03:00
Jacek Sieka f856c885fa
fix endian conversion issues ()
* fixes call to `bigEndian32` on a uint64 which breaks on big endian
platforms
* prefer endians2 for less and safer code
2020-07-12 18:22:47 +02:00
Eugene Kabanov ce6e7d17b1
Make Future tracking and stack traces optional ()
* Make Future tracking optional via -d:chronosDutureTracking compilation flag.
* Stack traces is now optional, use -d:chronosStackTraces.
* Fix mistypes and add test for chronosStackTrace option.
2020-07-08 19:48:01 +03:00
Eugene Kabanov 5629b3c41f
[WIP] Zero-cost unattended Future[T] tracking mechanism. ()
* Zero-cost unattended Future[T] tracking mechanism with tests and tracking of test suite.
2020-07-06 09:33:13 +03:00
Eugene Kabanov 16ed169f25
Fix cancellation race when low level futures are already completed, while cancellation process is pending. ()
Added test.
2020-07-03 15:03:59 +03:00
Eugene Kabanov 319e2bfc09
Fix Nim's issue using and add tests. () 2020-06-24 13:03:36 +03:00
Eugene Kabanov 02b8da986b
Add accept() call ()
* Add accept() call and tests.
* Fix rare fd leaks on Windows.
* Fix compilation warnings.
* Add fd leak test.
* Bump version to 2.4.0.
2020-06-24 11:21:52 +03:00
Eugene Kabanov 2ecc5500c2
Undeprecate `or` operation. ()
* Undeprecate `or` operation.
Fix `or` for already finished futures.
Add tests.

* Bump version to 2.3.9.
2020-04-21 07:07:49 +03:00
Eugene Kabanov 3d745a4b0c
Fix Nim's issue https://github.com/nim-lang/Nim/issues/13889. () 2020-04-06 15:49:09 +03:00
Eugene Kabanov 4e2810cfe0
Fix issue with allFinished(), allFutures(), one() behavior when Futures passed are already finished. ()
Added test.
2020-04-06 13:56:24 +03:00
Araq f3827a13d1 prepare nim-chronos for Nim version 1.2 2020-03-23 20:17:29 +02:00
Jacek Sieka d8f8e3d9fc
readMsg branch by @arnetheduck with some changes. ()
Co-authored-by: Eugene Kabanov <ka@hardcore.kiev.ua>
2020-03-05 10:59:10 +01:00
cheatfate 64583b4269
Fix Windows datagram's AnyAddress issue.
Add test for datagram's AnyAddress.
Bump version to 2.3.6.
2020-02-25 23:50:39 +02:00
cheatfate 80351cb928
Fix . 2020-02-12 22:54:05 +02:00
cheatfate e34857364e
Attempt to fix . 2020-01-28 12:47:38 +02:00
cheatfate 74700fdcab Fix bug cancellation handlers not called in wait() and withTimeout().
Fix double completion bug because of callback race.
Fix deprecation warnings.
Rename some internal procedures.
Bump version to 2.3.5.
2020-01-27 22:32:08 +02:00
cheatfate 943a961201
Enable --threads:on testing. 2019-11-01 06:42:32 +02:00
cheatfate fb00b20cfa
Fix all compilation warnings for 1.0.2.
Fix all related problems.
2019-10-29 23:19:41 +02:00
cheatfate 5f1391f39f
Fix unused imports. 2019-10-24 16:21:31 +03:00
cheatfate 29d9274e03
Fix some compilation warnings. 2019-10-24 16:01:57 +03:00
Yuriy Glukhov a291f26c82
Merge pull request from status-im/tlsstream
TLS async stream.
2019-10-22 13:32:36 +03:00
cheatfate d008fa2087
Fix make serverName mandatory and check for empty serverName. 2019-10-16 09:07:46 +03:00
cheatfate a92ad6d2d2
Add TLS inbound stream.
Fix some review comments.
2019-10-16 09:01:52 +03:00
cheatfate 3b8874a9e8
Fix issue with Windows connect(0.0.0.0). 2019-10-09 15:12:19 +03:00
cheatfate c27624cfc0
Add TlsStream with client-only connections. 2019-10-08 18:46:27 +03:00
cheatfate c4a99447bd
Fix AsyncLock race and refactor asyncsync.nim to properly support cancellation.
Fix async macro to not transform nested procedures.
2019-07-17 16:12:31 +03:00
cheatfate ec7f2a14a8
Fix compilation warnings in osnet.nim
Fix non stable stream tests.
2019-07-12 10:40:18 +03:00
Eugene Kabanov cd2571e80a
Merge pull request from status-im/cancellation
[WIP] Initial cancellation proposal.
2019-07-06 15:44:21 +03:00
cheatfate 8ba4fc9876
Add `awaitne` command which will have `yield` behavior in async cancellation world.
Add tests for both `await` and `awaitne`.
2019-07-06 11:16:31 +03:00
cheatfate 9c15a09904
Bump version to 2.2.7.
Attempt to fix stability of MacOS test.
2019-07-04 15:30:07 +03:00
cheatfate 43ae07f20c
Fix deprecation warning. 2019-07-04 15:13:39 +03:00
cheatfate fcfb87d2a0
Deprecate oneIndex(), oneValue(), all(), or(), and().
Add allFutures() and one().
Fix deprecation in chunkedstream.
Add tests.
2019-07-04 15:04:59 +03:00
cheatfate 43eeceb8e5
Attempt to fix macos timers. 2019-06-27 13:12:52 +03:00
cheatfate 992cc57377
Fix timeout value. 2019-06-26 15:50:40 +03:00
cheatfate 059433ccad
Add cancellation tests. 2019-06-26 15:36:01 +03:00
Eugene Kabanov 9160dcdbba
Merge pull request from status-im/asyncstream
Asynchronous streams.
2019-06-06 15:05:08 +03:00
cheatfate c27c564d47
Add oneIndex(), oneValue() which are varargs `or` operation.
Fix some deprecation warnings for Nim devel.
Add tests for oneIndex(), oneValue().
2019-06-04 19:51:35 +03:00
cheatfate 317ce3c797
Ignore ECONNRESET and EPIPE errors on sending.
Add test for ECONNRESET error.
2019-05-28 09:29:00 +03:00
cheatfate 6e7a753793
Fix for Windows server close procedure. 2019-05-09 20:59:15 +03:00
cheatfate 454571f943
Fix tests to consume all the bytes from the nested stream.
Add error messages for nested exceptions.
2019-05-08 12:44:00 +03:00
cheatfate 296b8c23a5
Fix tests compilation. 2019-05-07 23:20:06 +03:00
cheatfate 1763c9dcff
Add AsyncStreams.
Add Chunked-Encoding AsyncStream reader/writer.
Add tests.
2019-05-07 23:11:40 +03:00
cheatfate 2f6b36e96d
Cleanup debug echos. 2019-04-15 11:32:58 +03:00
cheatfate bc4fa46566
Add more test vectors from `nim-libp2p`. 2019-04-15 11:30:52 +03:00
cheatfate 2c2e2f7fad
Add getInterfaces() and getBestRoute().
Add IpNet and IpMask.
Add TTL setting for UDP transports with {Broadcast}.
Fix comments.
Add tests.
Bump version to 2.2.5.
2019-04-15 04:27:12 +03:00