nim-codex/codex
Eric e6a387e8e8
feat[marketplace]: add slot queue pausing (#752)
* add seen flag

* Add MockSlotQueueItem and better prioritisation tests

* Update seen priority, and include in SlotQueueItem.init

* Re-add processed slots to queue

Re-add processed slots to queue if the sale was ignored or errored

* add pausing of queue

- when processing slots in queue, pause queue if item was marked seen
- if availability size is increased, trigger onAvailabilityAdded callback
- in sales, on availability added, clear 'seen' flags, then unpause the queue
- when items pushed to the queue, unpause the queue

* remove unused NoMatchingAvailabilityError from slotqueue

The slot queue should also have nothing to do with availabilities

* when all availabilities are empty, pause the queue

An empty availability is defined as size < DefaultBlockSize as this means even the smallest possible request could not be served. However, this is up for discussion.

* remove availability from onAvailabilitiesEmptied callback

* refactor onAvailabilityAdded and onAvailabilitiesEmptied

onAvailabilityAdded and onAvailabilitiesEmptied are now only called from reservations.update (and eventually reservations.delete once implemented).

- Add empty routine for Availability and Reservation
- Add allEmpty routine for Availability and Reservation, which returns true when all all Availability or Reservation objects in the datastore are empty.

* SlotQueue test support updates

* Sales module test support updates

* Reservations module tests for queue pausing

* Sales module tests for queue pausing

Includes tests for sales states cancelled, errored, ignored to ensure onCleanUp is called with correct parameters

* SlotQueue module tests for queue pausing

* fix existing sales test

* PR feedback

- indent `self.unpause`
- update comment for `clearSeenFlags`

* reprocessSlot in SaleErrored only when coming from downloading

* remove pausing of queue when availabilities are "emptied"

Queue pausing when all availiabilies are "emptied" is not necessary, given that the node would not be able to service slots once all its availabilities' freeSize are too small for the slots in the queue, and would then be paused anyway.

Add test that asserts the queue is paused once the freeSpace of availabilities drops too low to fill slots in the queue.

* Update clearing of seen flags

The asyncheapqueue update overload would need to check index bounds and ultimately a different solution was found using the mitems iterator.

* fix test

request.id was different before updating request.ask.slots, and that id was used to set the state in mockmarket.

* Change filled/cleanup future to nil, so no await is needed

* add wait to allow items to be added to queue

* do not unpause queue when seen items are pushed

* re-add seen item back to queue once paused

Previously, when a seen item was processed, it was first popped off the queue, then the queue was paused waiting to process that item once the queue was unpaused. Now, when a seen item is processed, it is popped off the queue, the queue is paused, then the item is re-added to the queue and the queue will wait until unpaused before it will continue popping items off the queue. If the item was not re-added to the queue, it would have been processed immediately once unpaused, however there may have been other items with higher priority pushed to the queue in the meantime. The queue would not be unpaused if those added items were already seen. In particular, this may happen when ignored items due to lack of availability are re-added to a paused queue. Those ignored items will likely have a higher priority than the item that was just seen (due to it having been processed first), causing the queue to the be paused.

* address PR comments
2024-05-26 00:38:38 +00:00
..
blockexchange Cleanup integration tests (#757) 2024-05-23 15:29:30 +00:00
contracts Cleanup integration tests (#757) 2024-05-23 15:29:30 +00:00
erasure Scheduling erasure coding on another thread (#716) 2024-03-23 09:56:35 +00:00
manifest Bug: Large manifests encoding (#800) 2024-04-30 15:27:17 +00:00
merkletree Rework builder & sampler (#697) 2024-02-08 02:27:11 +00:00
purchasing feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
rest Update logging for download (#799) 2024-05-16 10:06:12 -07:00
sales feat[marketplace]: add slot queue pausing (#752) 2024-05-26 00:38:38 +00:00
slots refactor: use serde for json de/serialization instead of utils/json (#704) 2024-03-19 03:25:13 +00:00
stores Cleanup integration tests (#757) 2024-05-23 15:29:30 +00:00
streams Update logging for download (#799) 2024-05-16 10:06:12 -07:00
utils feat(api): availabilities patch endpoint (#692) 2024-03-21 10:53:45 +00:00
blockexchange.nim Change every dagger to codex (#102) 2022-05-19 13:56:03 -06:00
blocktype.nim Block and manifest as objects (#708) 2024-02-19 02:10:16 +00:00
chunker.nim Cleanup integration tests (#757) 2024-05-23 15:29:30 +00:00
clock.nim Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707) 2024-02-16 22:12:16 +00:00
codex.nim Adds two traces and bumps nim-ethers (#761) 2024-03-29 11:51:17 +00:00
codextypes.nim wire in circom backend (#698) 2024-02-09 21:40:30 +00:00
conf.nim Remove codex_use_hardhat compilation flag (#741) 2024-03-19 11:08:54 +00:00
contracts.nim feat: contract address management (#405) 2023-05-03 09:24:25 +02:00
discovery.nim Update logging for download (#799) 2024-05-16 10:06:12 -07:00
erasure.nim Improving proc/func formatting consistency (#454) 2023-06-22 08:11:18 -07:00
errors.nim Build slots (#668) 2024-01-11 08:45:23 -08:00
indexingstrategy.nim rework indexing strategy to use an enum (#694) 2024-02-07 11:13:27 -08:00
logutils.nim refactor: use serde for json de/serialization instead of utils/json (#704) 2024-03-19 03:25:13 +00:00
manifest.nim Update multicodecs (#665) 2023-12-22 13:04:01 +01:00
market.nim feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
merkletree.nim Build slots (#668) 2024-01-11 08:45:23 -08:00
namespaces.nim Blockexchange uses merkle root and index to fetch blocks (#566) 2023-11-14 13:02:17 +01:00
node.nim Cleanup integration tests (#757) 2024-05-23 15:29:30 +00:00
periods.nim Validator (#387) 2023-04-19 15:06:00 +02:00
purchasing.nim feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
rng.nim track latest nim-libp2p's master branch (#248) 2022-11-02 11:40:28 -06:00
sales.nim feat[marketplace]: add slot queue pausing (#752) 2024-05-26 00:38:38 +00:00
slots.nim wire in circom backend (#698) 2024-02-09 21:40:30 +00:00
stores.nim Rework merkle tree (#654) 2023-12-21 06:41:43 +00:00
streams.nim adding AsyncStreamWrapper (#110) 2022-06-14 09:19:35 -06:00
systemclock.nim Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707) 2024-02-16 22:12:16 +00:00
units.nim feat: create logging proxy (#663) 2024-01-22 23:35:03 -08:00
utils.nim Fixes active cancellation for pending want requests (#714) 2024-02-22 14:54:45 +00:00
validation.nim inlines copying of hashset (#809) 2024-05-17 00:57:30 +00:00