status-im
/
dagger-research
mirror of https://github.com/status-im/dagger-research.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

[slot queue] Accomodate existing requests before availabilities added (#168) 

Update slot queue design to accomodate storage requested before availabilities were added.
This commit is contained in:
Eric 2023-07-25 10:46:04 +10:00 committed by GitHub
parent 7015f21320
commit 7ace179f2f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 17 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
design/sales.md
View File

@ -155,6 +155,10 @@ shuffled, as there will be many hosts in the network that could potentially pick
up the request and will process the first slot in the queue at the same time. up the request and will process the first slot in the queue at the same time.
This should avoid some clashes in slot indices chosen by competing hosts. This should avoid some clashes in slot indices chosen by competing hosts.
Before slots can be added to the queue, availabilities must be checked to ensure
a matching availability exists. This filtering prevents all slots in the network
from entering the queue.
### Removing slots from the queue ### Removing slots from the queue
Hosts will also recieve contract events for when any contract is started, Hosts will also recieve contract events for when any contract is started,
@ -186,21 +190,11 @@ in the future once profitability can be calculated.
Queue processing will be started only once, when the sales module starts and Queue processing will be started only once, when the sales module starts and
will process slots continuously, in order, until the queue is empty. If the will process slots continuously, in order, until the queue is empty. If the
queue is empty, processing of the queue will resume once items have been added queue is empty, processing of the queue will resume once items have been added
to the queue. to the queue. If the queue is not empty, but there are no availabilities, queue
processing will resume once availabilites have been added.
As soon as items are available in the queue, and there are workers available for As soon as items are available in the queue, and there are workers available for
processing, an item is popped from the queue. processing, an item is popped from the queue and processed.
> NOTE: the previous design suggested, "if the queue is not empty, but there are
no availabilities, queue processing will resume once availabilites have been
added". However, this is no longer recommended. Resumuption of queue processing
once availabilites have been added would require the host to keep track of all
new storage requests on the network while there are no host availabilities. If
availabilities are eventually added, the backlog of network requests would need
to be compared to availabilities to potentially be added to the slot queue (if
there are matches). Maintaining this backlog of requests could potentially have
a large memory impact, and may not even be necessary if the host never adds (or
does not add for a long duration) availabilities.
When a slot is processed, it is first checked to ensure there is a matching When a slot is processed, it is first checked to ensure there is a matching
availabilty, as these availabilities will have changed over time. Then, the availabilty, as these availabilities will have changed over time. Then, the
@ -225,6 +219,16 @@ from the queue and processed. Each time an item is popped and processed, a
worker is removed from the available workers. If there are no available workers, worker is removed from the available workers. If there are no available workers,
queue processing will resume once there are workers available. queue processing will resume once there are workers available.
#### Adding availabilities
When a host adds an availability, a signal is triggered in the slot queue with
information about the availability. This triggers a lookup of past request for
storage events, capped at a certain number of past events or blocks. The slots
of the requests in each of these events are added to the queue, where slots
without matching availabilities are filtered out (see [Adding slots
to the queue](#adding-slots-to-the-queue) above). Additionally, when slots of
these requests are processed in the queue, they will be checked to ensure that
the slots are not filled (see [Queue processing](#queue-processing) above).
### Implementation tips ### Implementation tips
Request queue implementations should keep in mind that requests will likely need Request queue implementations should keep in mind that requests will likely need