roadmap/content/codex/updates/2024-02-19.md

title	tags	date	lastmod	draft	description
2024-02-19 Codex Weekly	codex-updates	2024-02-19	2024-02-19	false	Weekly update of Codex

Codex Update Feb 13th - Feb 19th

The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the Codex Testnet Starter documentation.

Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work. Development is currently broken into three distinct teams:

1. Client, Testing, and Infrastructure
2. Marketplace
3. Research

The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics.

Client

Epic: Nim Improvements

Completed:

• Initial PR up for Apatheia async-threading Ongoing:
• Started implementing an async-threading API

Epic: Wiring the Proving System

Completed: Ongoing:

• Code review of the circom circuit
• Progress on prover integration
• Determine why different amounts of data were being downloaded for different slot indices (and failing to release reservations due to be larger than the slotSize)
• Getting integration tests PR #662 into shape for merging
  • Integration tests were generally not using the right ec params (num nodes, dataset size, tolerance)
  • After EC params were corrected, some of the tests were not working due to the recent async clock.now update
    • Created a revert PR for the async clock.now PR

Epic: Improve Client Stability

Completed: Ongoing:

• Started work on delegating expensive computations to a separate thread
• Reviewing changes made to Apatheia
• Started integrating apaethia into nim-codex
• Rebasing two large PRs against latest master:
  • Safe block deletion (with ref count)
  • Expiry per dataset

Testing

Completed: Ongoing:

• Continue work to track down the timeout issues in Codex
  • Did a bunch of updates to logtools, which now understand what deploy IDs and test runs are (4 min video here);
• Worked on the block flow tracker in tandem with the logtools
  • Not cancelling pending WantHave's and under the right conditions this makes the peer implode
• Made some changes and opened PRs for test framework (one normal, and one tiny PR);
• Made changes to the build improvements proposal, so that the default version for the compiler is now represented as special version repo_version which can be accessed locally and in CI
  • For now this is hidden in this commit; started treating those PRs as low priority
• Isolated the Codex issue to a working hypothesis of failed cancel messages
  • Smaller-scale test validation of above issue
  • Learned how to actually slow down traffic on a given port
    • In the end that was not the issue, the issue being instead that peers need to create libp2p connections so they pop into each other's PeerCtxStore
• Put together failing test for the cancellation issue
  • Goes into testblockexc.nim
  • Need to provide a fix

Infrastructure

Completed:

• Adjust workflows for changelog generation #5
• Fix Docker entrypoint NAT helper variables #706

Ongoing:

• Create Codex Helm chart #58
• Deploy Codex Storage nodes in Testnet cluster #115

Marketplace

Epic: End-to-end Testing

Completed:

• Updated nim-ethers pull request to integrate serde
• Rebased integration test refactor
• Nim-serde changelog workflow in CI
• Finished moving nim-codex json util to serde

Ongoing:

• See Epic: Wiring the Proving System

Research

2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)

Completed:

• First working version of the Haskell Groth16 prover
  • Work towards an independent Groth16 prover (so that we can debug the Nim one) - an initial version of that seems to work now
• Reviewed improvements to the circuit

Ongoing:

• More work on the algebra backend (pairings, APIs)
• Started writing down some musings about aggregation (WIP)
• Found the bug in the Circom zkey usage preventing valid proof verification