Commit Graph

309 Commits

Author SHA1 Message Date
Derek 呆 1233f8fb4b
fix tiny typo in comment (#274) 2022-05-19 17:37:51 +02:00
zah 875d7d8e6e
Tentative fix for the HTTP client connection state assertion failures (#272)
* Tentative fix for the HTTP client connection state assertion failures

I've traced the problem to a HTTP connection being closed while there
are outstanding requests that still go through the motions (the crash
occurs when a requests reaches its `finish` processing step, but it
was already put in a Closed state by the connection that owns it).

* Use distinct error exception instead of cancellation error.

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2022-05-14 11:08:13 +03:00
Etan Kissling dedd7cb1d0
Guarantee exclusive use of `HttpClientConnection`. (#273)
When calling the HTTP `send` / `open` functions, `acquireConnection` is
called to obtain a connection in state `Ready`. In the next code block,
the connection state is advanced to `RequestHeadersSending`. However,
returning from chronos `async` procs yields control, similar to `await`.
This means that a connection may be added to the pool in `Ready` state,
and then a different `acquireConnection` call may obtain a second ref.
Introducing a new `Acquired` state ensures consistency in this case.

No tests added due to this being scheduler dependent; ran manual tests
by adding a `doAssert item.state != HttpClientConnectionState.Acquired`
between `if not(isNil(conn)):` and `return conn`. Eventually, the assert
got hit after several hours of repeated tests, confirming the edge case
to be solved by applying this fix.

Not sure if it is by design that returning from an `async` proc yields.
Even if it's not, this should solve current HTTP issues in nimbus-eth2.
2022-05-13 02:54:34 +03:00
Tanguy ae19d5b6f0
Handle multiple childs for opensymchoice (#266)
And more tests. Follow up of #261
2022-05-05 11:05:04 +02:00
Jacek Sieka bb4c3298f5
fix crash after reading from fd > 1024 (#267)
The socket selector holds a `seq` of per-descriptor data. When a reader
is registered, a pointer to a seq item is stored - when the `seq` grows,
this pointer becomes dangling and causes crashes like
https://github.com/status-im/nimbus-eth2/issues/3521.

It turns out that there already exist two mechanisms for passing user
data around - this PR simply removes one of them, saving on memory usage
and removing the need to store pointers to the `seq` data that become
dangling on resize.
2022-04-11 11:56:30 +02:00
cheatfate ae2a87778f
Fix missing connection flag. 2022-04-10 19:35:04 +03:00
Tanguy 64386b3112
Fix nimdoc for async procs (#260) 2022-04-05 12:16:00 +02:00
Tanguy fc65ff8c5b
Cleanup return type open syms (#261) 2022-03-30 16:13:58 +03:00
Tanguy 8719723077
Fix TLS certificate allocation (#259)
* Fix TLS certificate allocation

* change style + comment
2022-01-28 13:46:15 +02:00
Thomas Jost b47b2a96ce
Fix documentation build (#258)
`nim doc --project --index:on chronos.nim` was failing with the following error:

    /path/to/nim-chronos/chronos/transports/common.nim(519, 21) Error: '*' expected

...which in turned caused errors in all files that import this one.

Now the entire docs can be built successfully, with only a few warnings.

Fixes #186.
2022-01-26 20:56:22 +01:00
Emil 1f37dac810 Refactor `preferredContentType` proc & add tests
Fix of the following bug:
In case of multiple accept headers with same preference
`preferredContentType` used to select the first match within content types
provided by the application. For example, if user specifies accept headers
`application/octet-stream, application/json` and application provides
`application/json, application/octet-stream`, `application/octet-stream`
will be returned, and that is a bug.

Now the procedure returns the most suitable match according
to the application preferences.
2022-01-25 16:05:36 +02:00
Eugene Kabanov 519ca463df
Fix transport.write() unable to send data through OS pipes. (#256)
* Fix transport.write() unable to send data through pipe.
Add test for pipes.

* Fix flaky test.

* Add workaround for Nim's issue #19425.
2022-01-20 18:38:41 +02:00
Ștefan Talpalaru 17fed89c99
style fixes (#253)
* style fixes

and disable the cron CI job
2022-01-04 23:14:30 +01:00
Tanguy d926415f42
Fixes for nimdoc (#252) 2022-01-04 18:00:17 +01:00
Jacek Sieka 0fc82049ac enable --styleCheck:usages 2021-12-27 17:33:25 +02:00
Tanguy 7dc58d42b6
Improve ram usage (#243)
Remove cyclic references of {.async.} Futures, allowing them to be picked up by the regular refc instead of Mark and Sweep
2021-12-10 11:19:14 +01:00
Zahary Karadjov 7ca85ddadc
Fix compilation errors with --styleCheck:usages --styleCheck:error 2021-12-08 16:58:24 +02:00
Tanguy cedc603b81
Remove future var (#247) 2021-12-08 12:30:12 +01:00
Jacek Sieka c25fa1f6cd
posix: fast path for write (#244)
When `write` is called on a `StreamTransport`, the current sequence of
operations is:

* copy data to queue
* register for "write" event notification
* return unfinished future to `write` caller
* wait for "write" notification (in `poll`)
* perform one `send`
* wait for notification again if there's more data to write
* complete the future

In this PR, we introduce a fast path for writing:

* If the queue is empty, try to send as much data as possible
  * If all data is sent, return completed future without `poll` round
  * If there's more data to send than can be sent in one go, add the
rest to queue
* If the queue is not empty, enqueue as above
* When notified that write is possible, keep writing until OS buffer is
full before waiting for event again

The fast path provides significant performance benefits when there are
many small writes, such as when sending gossip to many peers, by
avoiding the poll loop and data copy on each send.

Also fixes an issue where the socket would not be removed from the
writer set if there were pending writes on close.
2021-12-08 11:35:27 +01:00
Eugene Kabanov 36e5f6fc89
Attempt to fix connection state. (#248)
* Attempt to fix connection state.

* Add actual values dump on asserts.

* Total rework of connection's states.
2021-12-03 13:11:39 +02:00
Eugene Kabanov 1c17d4c094
Fix connections management in http client. (#246)
* Proper connection lifetime handling mechanism.
HttpClientResponse do not close connection anymore.

* Add unique HTTP client connection identifier.
2021-12-01 12:23:39 +02:00
Andreas Rumpf 37c62af579
Added an explanation for the additional `gcsafe` pragmas (#241)
* prepare nim-chronos for Nim version 1.2

* enable --gc:orc testing

* ensure asyncmacro2 works with the upcoming 1.6

* added a remark
2021-11-22 14:48:59 +02:00
Ștefan Talpalaru 4aaaa3379a
Nim-1.6 fixes (#239) 2021-11-20 01:30:23 +01:00
Eugene Kabanov f51158da65
Add NewConnectionAlways option to HttpClientFlags, which allows to ignore connection management in client. (#238)
Optimize server behavior when HeadersTimeout is set to Infinity.
2021-11-19 18:13:36 +02:00
Michael Bradley ad0029e3ee
add raises pragma to registerEvent in ioselectors_epoll (#237)
Include `IOSelectorsException` in addition to `Defect` in `raises: []`.
2021-11-19 17:47:58 +02:00
Ștefan Talpalaru 56e007d165
comments for recent gcsafe pragmas 2021-11-19 01:07:46 +01:00
Andreas Rumpf cc22234dc6
Prepare chronos for Nim version 1.6 (#226) 2021-11-19 01:05:47 +01:00
Eugene Kabanov 0c7a0bd0a5
Handle accept() errors properly. (#234)
* Add more accept() call error handlers.
Fix issue when HTTP server silently stops accepting new connections.
Remove unneeded MacOS syscall to disable SIG_PIPE on socket, we already mask this signal in process.

* User `if` instead `case` because constants are actually variables.

* Fix mistypes.

* Do not use case for posix constants.

* Fix Linux compilation error.
2021-11-12 18:13:56 +02:00
cheatfate 661eae5732
Fix identifiers to be `uint` instead of `uint64`. 2021-10-21 17:46:06 +03:00
Eugene Kabanov 3bc0bc36f7
Change Future identifier type from `int` to `uint`. (#228)
* Change Future identifier type from `int` to `uint64`.

* Fix compilation error and tests.

* Add integer overflow test, to avoid confusion with next Nim versions, we expect that unsigned integers do not raise any exceptions if overflow happens.

* Switch from `uint64` to `uint` type.

* Add `uint` overflow tests.
2021-10-21 17:22:11 +03:00
cheatfate 2e57ddb455
Fix preferredContentType() function declaration to avoid confusion with `preferredContentType(varargs)`. 2021-10-21 14:01:46 +03:00
Eugene Kabanov 59b91bf0ca
Add more helpers to handle Accept header. (#224)
* Add API working with strings instead of `HttpRequestRef` object.

* Fix comment.
2021-09-28 01:41:29 +03:00
Eugene Kabanov 80102a3b6a
Eventbus implementation (#214)
* Initial commit.

* Move AsyncEventBus implementation to asyncsync.nim
Add subscriptAll(), waitAllEvents() primitives.
Add emitWait() primitive.
Add emitter source location implementation.
Add tests.
2021-09-15 16:55:15 +03:00
Eugene Kabanov bbbcb55493
Server-side events implementation. (#222)
Fix keep-alive issue.
Refactor some prepareXXX routines.
2021-09-14 20:32:58 +03:00
Eugene Kabanov 5034f0a5a6
Fix issue #219 (#220)
* Address issue #219 and add tests for it.
Some cosmetic refactoring.

* Fix *nix tests.
2021-09-05 00:53:27 +03:00
Eugene Kabanov 05c91418be
Export async-related imports to allow partial imports. (#218) 2021-08-26 14:22:29 +03:00
Eugene Kabanov 9d80c7068f
Server must not send `Host` header. (#215) 2021-08-15 04:29:48 +03:00
Eugene Kabanov b14f66c295
No more Defect on stream close. (#213)
Fix async streams issue with replacing state.
Add `closing` states to HTTP's server connection, request and bodyrw.
Fix Http server cancellation leaks.
2021-08-06 13:13:55 +03:00
Eugene Kabanov ef2430d08d
Add `Accept` header handling to httpserver.nim. (#211)
* Add `Accept` header handling to httpserver.nim.
Add simple test suite.
Bump version to 3.0.6.

* Fix compilation error.
2021-07-28 17:08:38 +03:00
Johann Bauer 3a9cc6bfc9
Update `toIPv4` and `toIPv6` to keep the port (#209)
Currently, converting a `TransportAddress` from IPv4 to IPv6 or vice-versa will set the port to 0.
2021-07-27 14:39:48 +03:00
Eugene Kabanov 14ebf269e9
Fix accept defect (#206)
* Fix accept() FutureDefect on cancellation.

* Do not close pipe twice.

* Check for retFuture state before changing it state.
2021-06-30 18:22:37 +03:00
Eugene Kabanov 15137f71c3
Basic authorization implementation for HTTP client. (#204)
* Basic authorization implementation for HTTP client.
Add tests for basic authorization.

* Bump chronos version to 3.0.5.
2021-06-29 02:38:08 +03:00
Eugene Kabanov f7dd6b76c2
Safely close connection's streams. (#199) 2021-06-14 15:16:33 +03:00
Eugene Kabanov e6fd38fd49
Add one more state `Closing` to help avoid race condition while in `closeWait` of request, response, connection. (#198) 2021-06-10 13:58:41 +03:00
Eugene Kabanov 7ccb170f7a
Enable comma as array delimiter and adding tests. (#191)
* Enable comma as array delimiter and adding tests.

* Bump version to 3.0.4.
2021-05-17 22:39:24 +03:00
cheatfate 67f0f1224f
Bump nimble version.
Make some useful fields public.
2021-05-10 10:40:56 +03:00
Eugene Kabanov be184a815c
Httpclient (#182)
* Initial commit.

* Some refactoring.

* Allow boundstream to accept uint64.
Fix httpserver and asyncstream tests to follow new uint64 requirement.

* send() and getBodyBytes() implementations.

* Add closeWait for response and request.
Refactor finish/close flow.

* Changes in state machine
Add first test.

* Missing test file.

* Fixed tests
Add http leaking trackers and tests.

* Some fixes in multipart.
Fix automatic Content-Length header for requests with body.
Fix getBodyBytes() assertions.
Merging tests to main suite.

* Post rebase fixes.

* Fix tests big message generation.

* Fix response state management and leaks for getBodyXXX() procedures.

* Add redirection support to client and server.
Add fetch(url) procedure with redirection support.
Add tests for redirection.
2021-05-10 10:26:36 +03:00
Eugene Kabanov d270dba8a3
Fix unused warnings, result, asyncCheck and 80 cpl (#185)
* Fix sources to follow 80 characters per line.
Fix unused compilation warnings.
Refactor (remove result) handles.nim.
Fix tests to use asyncSpawn instead of asyncCheck.

* Fix handles for Unix platforms.
2021-05-07 23:52:24 +03:00
Eugene Kabanov fed6b0ac92
Restore functionality of zero-sized bounded reader/writer streams. (#184)
* Restore functionality of zero-sized bounded reader/writer streams.
Adding tests for it.

* run build_nim.sh unconditionally

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-05-07 18:52:44 +03:00
Ștefan Talpalaru c15c985c1f
support nimStackTraceOverride (#181) 2021-05-06 09:49:55 +02:00
Jacek Sieka 43b8aada20
Add message to task failure message (#183) 2021-05-06 09:46:58 +02:00
cheatfate 645d62db78
Remove fastpath execution and finally fix issue. 2021-04-27 00:07:49 +03:00
Eugene Kabanov 39f4060e07
Refactor and optimization of BoundedStream. (#180)
Documentation for BoundedStream.
Fix HttpServer bounding.
2021-04-26 14:05:37 +03:00
Eugene Kabanov 833d968782
Fix upload() issue and adding tests. (#179) 2021-04-24 20:32:21 +03:00
Eugene Kabanov e6ba00cb81
Fix reading could stuck on very big chunked headers and tests. (#178) 2021-04-22 15:32:28 +03:00
Eugene Kabanov aab1e30a72
Refactor common.nim and add more resolve procedures. (#177)
* Refactor common.nim to remove `result` usage.
Fix comparison of TransportAddress issue.
Add resolveTAddress procedures for both IPv4 and IPv6 addresses.
Fix tests.

* Bump version to 3.0.2.
2021-04-10 00:39:54 +03:00
Jacek Sieka 895fc53193
add explicit gcsafe annotations (fixes #174) (#175)
Seems like a variation of https://github.com/nim-lang/Nim/issues/17369
2021-04-03 10:05:08 +02:00
markspanbroek e5d3ecdd01
Ensures that asyncmacro2 does not raise exceptions (#172)
* Ensures that asyncmacro2 does not raise exceptions

* Remove unnecessary include
2021-03-30 16:48:20 +02:00
Jacek Sieka 075b7abf4c
fix Defect check in non-strict mode (#170) 2021-03-25 15:53:33 +01:00
Kim De Mey c206d2bc19
Add raises annotation to withTimeout callbacks (#168) 2021-03-25 14:20:26 +01:00
Jacek Sieka 4abd7a5645
exception tracking (#166)
* exception tracking

This PR adds minimal exception tracking to chronos, moving the goalpost
one step further.

In particular, it becomes invalid to raise exceptions from `callSoon`
callbacks: this is critical for writing correct error handling because
there's no reasonable way that a user of chronos can possibly _reason_
about exceptions coming out of there: the event loop will be in an
indeterminite state when the loop is executing an _random_ callback.

As expected, there are several issues in the error handling of chronos:
in particular, it will end up in an inconsistent internal state whenever
the selector loop operations fail, because the internal state update
functions are not written in an exception-safe way. This PR turns this
into a Defect, which probably is not the optimal way of handling things
- expect more work to be done here.

Some API have no way of reporting back errors to callers - for example,
when something fails in the accept loop, there's not much it can do, and
no way to report it back to the user of the API - this has been fixed
with the new accept flow - the old one should be deprecated.

Finally, there is information loss in the API: in composite operations
like `poll` and `waitFor` there's no way to differentiate internal
errors from user-level errors originating from callbacks.

* store `CatchableError` in future
* annotate proc's with correct raises information
* `selectors2` to avoid non-CatchableError IOSelectorsException
* `$` should never raise
* remove unnecessary gcsafe annotations
* fix exceptions leaking out of timer waits
* fix some imports
* functions must signal raising the union of all exceptions across all
platforms to enable cross-platform code
* switch to unittest2
* add `selectors2` which supercedes the std library version and fixes
several exception handling issues in there

* fixes

* docs, platform-independent eh specifiers for some functions

* add feature flag for strict exception mode

also bump version to 3.0.0 - _most_ existing code should be compatible
with this version of exception handling but some things might need
fixing - callbacks, existing raises specifications etc.

* fix AsyncCheck for non-void T
2021-03-24 10:08:33 +01:00
Eugene Kabanov c8eefb9382
Split HTTPS and HTTP servers. (#165)
* Split HTTPS and HTTP servers.

* Fix review commens
2021-03-17 15:40:40 +02:00
Eugene Kabanov 0b78606e41
Add `server` and `host` headers handling. (#164)
Fix response headers generation to avoid unnecessary computations.
2021-03-11 13:41:13 +02:00
Eugene Kabanov f774644129
Fix integer decoding overflow issue. (#163)
Switch to stew.base10 procedures.
Adjust tests to follow new behavior.
Bump version to 2.6.1.
2021-03-06 00:22:32 +02:00
Andreas Rumpf 1324b2f787
Enable --gc:orc testing for nim-chronos (#161)
* enable --gc:orc testing on nim devel (1.5.x)
2021-03-03 20:04:09 +02:00
Eugene Kabanov c31e4537fe
Fix contentType handling code. (#162)
Add more respond() aliases.
Add responded() to check if response is already sent.
2021-03-02 15:26:07 +02:00
cheatfate 72151b99cc
Fix `HttpTable.contains` argument requirements. 2021-02-26 11:31:46 +02:00
Eugene Kabanov eb6e32605f
Fix #156 and refactor osnet.nim (#157)
* Refactor osnet.nim to not use `result`.
Fix #156.

* Fix linux compilation error.
2021-02-25 22:04:56 +01:00
cheatfate d6a39e46e1
Fix tests. 2021-02-21 17:34:55 +02:00
cheatfate d595e0dbc6
Send `Content-Length: 0` on error.
Handle `DisconnectError`.
2021-02-21 04:22:59 +02:00
cheatfate 60a53eea99
Fix import/export issue of httputils.
Add bound address to HttpServer object.
2021-02-20 19:41:22 +02:00
Eugene Kabanov d49e0a9c47
Add drop() implementation (was missed) to httpserver and fix transport leak. (#158)
* Add drop() implementation (was missed).
Add tests for drop().
Fix transport leak because of drop().

* Fix future leak.
2021-02-19 14:07:20 +02:00
cheatfate 25688cd0aa Add hexValue tests. 2021-02-18 22:16:04 +02:00
cheatfate 4e9a857b90 Link to original procedure for hexValue(). 2021-02-18 22:16:04 +02:00
cheatfate 1d49ae7a47 Fix x86 int conversion. 2021-02-18 22:16:04 +02:00
cheatfate eb81018d02 Address review comments and fix issues found.
Adding more tests.
2021-02-18 22:16:04 +02:00
cheatfate fc0d1bcb43 Address review comments. 2021-02-18 22:16:04 +02:00
cheatfate 6f8d06f12d Add HttpDisconnectError to avoid client processor notification call. 2021-02-18 22:16:04 +02:00
cheatfate 1789328748 Set TLSStream default to use TLS1.2 only. 2021-02-18 22:16:04 +02:00
cheatfate b47fcb3e86 Annotate with `raises:[Defect]`. 2021-02-18 22:16:04 +02:00
cheatfate 45cb009be2 Remove `bodyTimeout` because downloading body is not under our control. 2021-02-18 22:16:04 +02:00
cheatfate 3495122867 Fix getMultipartBoundary() issues and add tests for it. 2021-02-18 22:16:04 +02:00
cheatfate 3e9ffae407 Properly fix case when request body size exceeds maximum allowed size. 2021-02-18 22:16:04 +02:00
cheatfate 970e5641d7 Add less strict rules for BoundStream reader/writer. 2021-02-18 22:16:04 +02:00
cheatfate d43a9cb92d HttpServer now supports TLS.
Some TLSStream fixes to properly support EOF.
Some HttpServer to properly support TLS handshake problems.
HttpServer test suite for HTTPS.
2021-02-18 22:16:04 +02:00
cheatfate 1a3e9162a4 Fix multipart end of message handling.
Add apps.nim.
Change copyrights dates.
Add httpserver tests to test suite.
2021-02-18 22:16:04 +02:00
cheatfate fb11d9f8ab Add httptable changes. 2021-02-18 22:16:04 +02:00
cheatfate 8c20b369b7 Fix queryParams() to not produce empty values.
Fix part() cancellation.
Add requestInfo() procedure.
Fix request.scheme.
Add MultiPart.isEmpty()
Fix MultiPart counter.
Add isEmpty() for HttpTable.
Add some documentation in HttpTable.
2021-02-18 22:16:04 +02:00
cheatfate a310a5620a General fixes. 2021-02-18 22:16:04 +02:00
cheatfate 0b03f8ec50 Expect header handling and response preparations. 2021-02-18 22:16:04 +02:00
cheatfate 74b0f85fc7 Add async post() procedure. 2021-02-18 22:16:04 +02:00
cheatfate 475c03ff7b Fix MultiPart handling mechanisms. 2021-02-18 22:16:04 +02:00
cheatfate c0472bd349 Fix streams to check only for Closed state. 2021-02-18 22:16:04 +02:00
cheatfate 534df03a0b Fixes and improvements 2021-02-18 22:16:04 +02:00
cheatfate 0e5ea5b737 Prepare for HttpResponse. 2021-02-18 22:16:04 +02:00
cheatfate 60e5396a9e Remove executable. 2021-02-18 22:16:04 +02:00
cheatfate 8381a40868 Initial commit. 2021-02-18 22:16:04 +02:00
cheatfate 2defc4b822 Add sequence of bytes as boundary to BoundStream and tests. 2021-02-18 22:16:04 +02:00
cheatfate fb137112be Remove debugging changes. 2021-02-18 22:16:04 +02:00
cheatfate 13eddf382d Simplification and fixes for TLSStream state machine. 2021-02-18 22:16:04 +02:00