mirror of
https://github.com/logos-co/roadmap.git
synced 2025-01-11 00:45:47 +00:00
add and modify latest weekly Codex updates
This commit is contained in:
jessiebroke
parent
26429c628f
commit
485c91a8e6
@ -26,7 +26,7 @@ The different teams have actively moving on various fronts. The following are th
|
||||
### Epic: Nim Improvements
|
||||
Completed:
|
||||
- [Add Nim-matrix workflow to run on merge queue](<https://github.com/codex-storage/nim-codex/pull/693>)
|
||||
- [changes to build improvements proposal so default version for the compiler is now represented as special version `repo_version` which can be accessed locally and in CI](https://github.com/codex-storage/nim-codex/pull/673/commits/f92d069d82741bb6ae5f51574f7610d79566cc78)
|
||||
- [Changes to build improvements proposal so default version for the compiler is now represented as special version `repo_version` which can be accessed locally and in CI](https://github.com/codex-storage/nim-codex/pull/673/commits/f92d069d82741bb6ae5f51574f7610d79566cc78)
|
||||
|
||||
Ongoing:
|
||||
- [Nim memory profiler](https://github.com/codex-storage/nim-chroprof)
|
||||
@ -47,6 +47,7 @@ Ongoing:
|
||||
- Run Continuous-Test outside of the cluster to reproduce 'file size issue'
|
||||
|
||||
### Epic: Infra
|
||||
Completed:
|
||||
Ongoing:
|
||||
- [Create Codex Helm chart](<https://github.com/codex-storage/infra-codex/issues/58>)
|
||||
- [Deploy Codex Storage nodes in Testnet cluster #115](<https://github.com/codex-storage/infra-codex/issues/115>)
|
||||
@ -55,9 +56,7 @@ Ongoing:
|
||||
- Now understand what deploy IDs and test runs are ([4 min video here](https://vimeo.com/911297714/5afc6e0056));
|
||||
- [Worked on the block flow tracker](https://github.com/gmega/blockexc-parser)
|
||||
|
||||
|
||||
## Marketplace
|
||||
|
||||
### Epic: End-to-end Testing
|
||||
Completed:
|
||||
- [Fixed decoding issue in nim-ethers](https://github.com/codex-storage/nim-ethers/pull/63)
|
||||
@ -92,8 +91,6 @@ Ongoing:
|
||||
6. Dynamic data (appendable data)
|
||||
```
|
||||
Completed:
|
||||
- N/A
|
||||
|
||||
Ongoing:
|
||||
- more work on the algebra backend (pairings, APIs)
|
||||
- work towards an independent Groth16 prover (so that we can debug the Nim one) - an initial version of that seems to work now
|
||||
- More work on the algebra backend (pairings, APIs)
|
||||
- Work towards an independent Groth16 prover (so that we can debug the Nim one) - an initial version of that seems to work now
|
||||
109
content/codex/updates/2024-02-19.md
Normal file
109
content/codex/updates/2024-02-19.md
Normal file
@ -0,0 +1,109 @@
|
||||
---
|
||||
title: 2024-02-19 Codex Weekly
|
||||
tags:
|
||||
- codex-updates
|
||||
date: 2024-02-19
|
||||
lastmod: 2024-02-19
|
||||
draft: false
|
||||
description: Weekly update of Codex
|
||||
---
|
||||
|
||||
# Codex Update Feb 13th - Feb 19th
|
||||
|
||||
> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
|
||||
>
|
||||
> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work.
|
||||
Development is currently broken into three distinct teams:
|
||||
|
||||
1. Client, Testing, and Infrastructure
|
||||
2. Marketplace
|
||||
3. Research
|
||||
|
||||
The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics.
|
||||
|
||||
## Client
|
||||
### Epic: Nim Improvements
|
||||
Completed:
|
||||
- [Initial PR up for Apatheia async-threading](https://github.com/elcritch/apatheia/pull/1)
|
||||
Ongoing:
|
||||
- [Started implementing an async-threading API](https://github.com/elcritch/apatheia)
|
||||
|
||||
### Epic: Wiring the Proving System
|
||||
Completed:
|
||||
Ongoing:
|
||||
- [Code review of the circom circuit](https://github.com/codex-storage/codex-storage-proofs-circuits/pull/5)
|
||||
- [Progress on prover integration](https://github.com/codex-storage/nim-codex/pull/702)
|
||||
- Determine why different amounts of data were being downloaded for different slot indices (and failing to release reservations due to be larger than the slotSize)
|
||||
- Getting integration tests PR #662 into shape for merging
|
||||
- Integration tests were generally not using the right ec params (num nodes, dataset size, tolerance)
|
||||
- After EC params were corrected, some of the tests were not working due to the recent [async](http://async.now) [clock.now](http://clock.now) update
|
||||
- [Created a revert PR for the async clock.now PR](https://github.com/codex-storage/nim-codex/pull/707)
|
||||
|
||||
### Epic: Improve Client Stability
|
||||
Completed:
|
||||
Ongoing:
|
||||
- Started work on delegating expensive computations to a separate thread
|
||||
- [Reviewing changes made to Apatheia](https://github.com/elcritch/apatheia/pull/1)
|
||||
- Started integrating `apaethia` into `nim-codex`
|
||||
- Rebasing two large PRs against latest master:
|
||||
- [Safe block deletion (with ref count)](https://github.com/codex-storage/nim-codex/pull/631)
|
||||
- [Expiry per dataset](https://github.com/codex-storage/nim-codex/pull/678)
|
||||
|
||||
## Testing
|
||||
Completed:
|
||||
Ongoing:
|
||||
- Continue work to track down the timeout issues in Codex
|
||||
- Did a bunch of updates to logtools, which now understand what deploy IDs and test runs are (4 min video here);
|
||||
- Worked on the block flow tracker in tandem with the logtools
|
||||
- Not cancelling pending WantHave's and under the right conditions this makes the peer implode
|
||||
- Made some changes and opened PRs for test framework (one normal, and one tiny PR);
|
||||
- Made changes to the build improvements proposal, so that the default version for the compiler is now represented as special version repo_version which can be accessed locally and in CI
|
||||
- For now this is hidden in this commit; started treating those PRs as low priority
|
||||
- Isolated the Codex issue to a working hypothesis of failed cancel messages
|
||||
- [Smaller-scale test validation of above issue](https://github.com/gmega/alternate-test-runner/blob/main/bin/local-codex-net.sh)
|
||||
- [Learned how to actually slow down traffic on a given port](https://github.com/gmega/alternate-test-runner/blob/main/bin/setup-delays.sh)
|
||||
- In the end that was not the issue, the issue being instead that peers need to create libp2p connections so they pop into each other's `PeerCtxStore`
|
||||
- [Put together failing test for the cancellation issue](https://gist.github.com/gmega/35aef177a37d21cb0524723d8dfbd3b4)
|
||||
- [Goes into testblockexc.nim](https://github.com/codex-storage/nim-codex/blob/master/tests/codex/blockexchange/engine/testblockexc.nim)
|
||||
- Need to provide a fix
|
||||
|
||||
## Infrastructure
|
||||
Completed:
|
||||
- [Adjust workflows for changelog generation #5](<https://github.com/codex-storage/nim-serde/pull/5>)
|
||||
- [Fix Docker entrypoint NAT helper variables #706](<https://github.com/codex-storage/nim-codex/pull/706>)
|
||||
|
||||
Ongoing:
|
||||
- [Create Codex Helm chart #58](<https://github.com/codex-storage/infra-codex/issues/58>)
|
||||
- [Deploy Codex Storage nodes in Testnet cluster #115](<https://github.com/codex-storage/infra-codex/issues/115>)
|
||||
|
||||
## Marketplace
|
||||
|
||||
### Epic: End-to-end Testing
|
||||
Completed:
|
||||
- [Updated nim-ethers pull request to integrate serde](https://github.com/codex-storage/nim-ethers/pull/64)
|
||||
- Rebased integration test refactor
|
||||
- Nim-serde changelog workflow in CI
|
||||
- [Finished moving nim-codex json util to serde](https://github.com/codex-storage/nim-codex/pull/704)
|
||||
|
||||
Ongoing:
|
||||
- See Epic: Wiring the Proving System
|
||||
|
||||
## Research
|
||||
```
|
||||
2024 R&D Goals
|
||||
1. Proving system and aggregation improvements (folding or lookups)
|
||||
2. Aggregator/validator design
|
||||
3. DHT improvements
|
||||
4. Tokenomics and incentive design
|
||||
5. Bandwidth incentives
|
||||
6. Dynamic data (appendable data)
|
||||
```
|
||||
Completed:
|
||||
- First working version of the Haskell Groth16 prover
|
||||
- Work towards an independent Groth16 prover (so that we can debug the Nim one) - an initial version of that seems to work now
|
||||
- Reviewed improvements to the circuit
|
||||
|
||||
Ongoing:
|
||||
- More work on the algebra backend (pairings, APIs)
|
||||
- Started writing down some musings about aggregation (WIP)
|
||||
- Found the bug in the Circom zkey usage preventing valid proof verification
|
||||
71
content/codex/updates/2024-02-26.md
Normal file
71
content/codex/updates/2024-02-26.md
Normal file
@ -0,0 +1,71 @@
|
||||
---
|
||||
title: 2024-02-26 Codex Weekly
|
||||
tags:
|
||||
- codex-updates
|
||||
date: 2024-02-26
|
||||
lastmod: 2024-02-26
|
||||
draft: false
|
||||
description: Weekly update of Codex
|
||||
---
|
||||
|
||||
# Codex Update Feb 20th - Feb 26th
|
||||
|
||||
> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
|
||||
>
|
||||
> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work.
|
||||
Development is currently broken into three distinct teams:
|
||||
|
||||
1. Client, Testing, and Infrastructure
|
||||
2. Marketplace
|
||||
3. Research
|
||||
|
||||
The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics.
|
||||
|
||||
## Client, Testing and Infrastructure
|
||||
Completed:
|
||||
Ongoing:
|
||||
- Codex team members heading to attend ETH Denver
|
||||
|
||||
### Epic: Nim Improvements
|
||||
Completed:
|
||||
Ongoing:
|
||||
|
||||
### Epic: Wiring the Proving System
|
||||
Completed:
|
||||
Ongoing:
|
||||
|
||||
### Epic: Improve Client Stability
|
||||
Completed:
|
||||
Ongoing:
|
||||
|
||||
## Marketplace
|
||||
|
||||
### Epic: End-to-end Testing
|
||||
Completed:
|
||||
- [Finished PR for availabilities update endpoints](https://github.com/codex-storage/nim-codex/pull/692)
|
||||
- Ran into very weird bug in one of these:
|
||||
- Chronos server implementation, Presto or Nim's HTTPClient
|
||||
- Could not debug and decided to rather move from HTTP 204 response to 200 one
|
||||
- [Created reproduction repo](https://github.com/AuHau/nim-http204-error)
|
||||
- [Logged the issue in Presto repo](https://github.com/status-im/nim-presto/issues/77)
|
||||
- [Debugged non-working Dist. testing contracts](https://github.com/codex-storage/codex-contracts-eth/pull/89)
|
||||
|
||||
Ongoing:
|
||||
|
||||
## Research
|
||||
```
|
||||
2024 R&D Goals
|
||||
1. Proving system and aggregation improvements (folding or lookups)
|
||||
2. Aggregator/validator design
|
||||
3. DHT improvements
|
||||
4. Tokenomics and incentive design
|
||||
5. Bandwidth incentives
|
||||
6. Dynamic data (appendable data)
|
||||
```
|
||||
Completed:
|
||||
- helped with figuring out the Ethereum elliptic curve conventions
|
||||
- gave an elliptic curve talk at the local university; [slides](https://raw.githubusercontent.com/bkomuves/slides/master/mathematics/elliptic_curves_2024.pdf) for the interested
|
||||
- found the second bug in the Nim Groth16 prover (in Constantine again)
|
||||
|
||||
Ongoing:
|
||||
- started thinking about the small / dynamic data problem
|
||||
86
content/codex/updates/2024-03-04.md
Normal file
86
content/codex/updates/2024-03-04.md
Normal file
@ -0,0 +1,86 @@
|
||||
---
|
||||
title: 2024-03-04 Codex Weekly
|
||||
tags:
|
||||
- codex-updates
|
||||
date: 2024-03-04
|
||||
lastmod: 2024-03-04
|
||||
draft: false
|
||||
description: Weekly update of Codex
|
||||
---
|
||||
|
||||
# Codex Update Feb 27th - Mar 4th
|
||||
|
||||
> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
|
||||
>
|
||||
> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work.
|
||||
Development is currently broken into three distinct teams:
|
||||
|
||||
1. Client, Testing, and Infrastructure
|
||||
2. Marketplace
|
||||
3. Research
|
||||
|
||||
The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics.
|
||||
|
||||
## Client, Testing and Infrastructure
|
||||
Completed:
|
||||
- Codex team members attended ETH Denver
|
||||
Ongoing:
|
||||
|
||||
### Epic: Nim Improvements
|
||||
Completed:
|
||||
Ongoing:
|
||||
- Started a PR to fix Atlas after SAT integration
|
||||
- [Finishing basic fixes PR in Atlas](https://github.com/nim-lang/atlas/pull/120)
|
||||
- [extracted the profiler into its own library](https://github.com/codex-storage/nim-chroprof)
|
||||
- retained only the instrumentation modifications in our [Chronos V4 branch](https://github.com/codex-storage/nim-chronos/pull/7/files)
|
||||
- after we merge V4 into Codex ([blocked by a bunch of issues](https://github.com/codex-storage/nim-codex/pull/673#issuecomment-1904108922)), start a discussion with the Chronos folks to try to get the instrumentation part in
|
||||
|
||||
### Epic: Wiring the Proving System
|
||||
Completed:
|
||||
Ongoing:
|
||||
- Ceremony file setup
|
||||
|
||||
### Epic: Improve Client Stability
|
||||
Completed:
|
||||
Ongoing:
|
||||
- [tracked a Codex bug during a V4 merge which turned out to be a compiler bug](https://github.com/nim-lang/Nim/issues/23354)
|
||||
- with [consequences to Chronos](https://github.com/status-im/nim-chronos/issues/511)
|
||||
- bug resolution handled by Nim compiler team after being flagged as "showstopper"
|
||||
|
||||
## Marketplace
|
||||
|
||||
### Epic: End-to-end Testing
|
||||
Completed:
|
||||
- Add IsSyncing to nim-ethers
|
||||
- Add sync check to codex startup
|
||||
- In-flight flag for outgoing blocks
|
||||
- Discord role rewards based on on-chain state and events
|
||||
- Partially tested, requires fixed marketplace-dist-test
|
||||
- Updated and merged: fix for block-retransmit issue
|
||||
- Debug isSync (nim-ethers update) and update to testnet-starter
|
||||
|
||||
Ongoing:
|
||||
|
||||
## Research
|
||||
```
|
||||
2024 R&D Goals
|
||||
1. Proving system and aggregation improvements (folding or lookups)
|
||||
2. Aggregator/validator design
|
||||
3. DHT improvements
|
||||
4. Tokenomics and incentive design
|
||||
5. Bandwidth incentives
|
||||
6. Dynamic data (appendable data)
|
||||
```
|
||||
Completed:
|
||||
- Nim Groth16 prover finally appears to work correctly (though with the current workaround it also became significantly slower)
|
||||
- Multi-threading support in the Nim Groth16 prover
|
||||
|
||||
Ongoing:
|
||||
- [Thinking about using FRI commitment for storage proofs](https://hackmd.io/8gtpagUYSz2F8e6J_JkCyg)
|
||||
- Hash based, includes RS, outsourceable, easy to prove large/r data sets
|
||||
- [Thinking about using KZG as updateable commitment](https://hackmd.io/YrGjYOJiSImgCqiaMZzyLw)
|
||||
- With efficient proofs for small/updateable data sets
|
||||
- Debugging the multithreading bug in the prover
|
||||
- [checking out new research](https://zkmesh.substack.com/p/zkmesh-feb-2024-recap)
|
||||
- Speed up the "binary circuit" proposal by a factor of ~2x
|
||||
- Thinking about a (supposedly fast) dedicated SHA256 proof protocol, and how to implement it
|
||||
73
content/codex/updates/2024-03-11.md
Normal file
73
content/codex/updates/2024-03-11.md
Normal file
@ -0,0 +1,73 @@
|
||||
---
|
||||
title: 2024-03-11 Codex Weekly
|
||||
tags:
|
||||
- codex-updates
|
||||
date: 2024-03-11
|
||||
lastmod: 2024-03-11
|
||||
draft: false
|
||||
description: Weekly update of Codex
|
||||
---
|
||||
|
||||
# Codex Update Mar 5th - Mar 11th
|
||||
|
||||
> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
|
||||
>
|
||||
> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work.
|
||||
Development is currently broken into three distinct teams:
|
||||
|
||||
1. Client, Testing, and Infrastructure
|
||||
2. Marketplace
|
||||
3. Research
|
||||
|
||||
The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics.
|
||||
|
||||
## Client, Testing and Infrastructure
|
||||
|
||||
### Epic: Multithreading
|
||||
Completed:
|
||||
- [Marked PR as ready - scheduling erasure encoding and decoding on a thread pool](https://github.com/codex-storage/nim-codex/pull/716)
|
||||
|
||||
Ongoing:
|
||||
- Started work on scheduling prover computation on a thread pool
|
||||
|
||||
### Epic: Wiring the Proving System
|
||||
Completed:
|
||||
Ongoing:
|
||||
- [Marked PR as ready - scheduling erasure encoding and decoding on a thread pool](https://github.com/codex-storage/nim-codex/pull/716)
|
||||
- Started work on scheduling prover computation on a thread pool
|
||||
|
||||
### Epic: Improve Client Stability
|
||||
Completed
|
||||
- [Reasoning on why the profiler works the way it does](https://hackmd.io/fA4RXNWsQZSSgYLNcIksEw)
|
||||
- [Closed bug: fixes double lookups when block does not exist](https://github.com/codex-storage/nim-codex/pull/739)
|
||||
|
||||
|
||||
Ongoing:
|
||||
- Some triaging of bugs in the Codex repo, still some way to go
|
||||
|
||||
## Marketplace
|
||||
|
||||
### Epic: End-to-end Testing
|
||||
Completed:
|
||||
Ongoing:
|
||||
|
||||
## Research
|
||||
```
|
||||
2024 R&D Goals
|
||||
1. Proving system and aggregation improvements (folding or lookups)
|
||||
2. Aggregator/validator design
|
||||
3. DHT improvements
|
||||
4. Tokenomics and incentive design
|
||||
5. Bandwidth incentives
|
||||
6. Dynamic data (appendable data)
|
||||
```
|
||||
Completed:
|
||||
- EthResearch post on sampling techniques (https://ethresear.ch/t/lossydas-lossy-incremental-and-diagonal-sampling-for-data-availability/18963)
|
||||
- Logos X space with Leo and Danny
|
||||
- Minor refactor the proof circuit [PR](https://github.com/codex-storage/codex-storage-proofs-circuits/pull/7)
|
||||
|
||||
Ongoing:
|
||||
- DAS simulator (https://github.com/codex-storage/das-research) paper prep (calls + work)
|
||||
- Shadow based P2P emulator for DAS (https://github.com/cskiraly/dst-gossipsub-test-node/tree/FullDAS)
|
||||
- Looked into how circom handles custom gates (tldr: it's easy to abuse for our own purposes, but snarkjs doesn't implement anything)
|
||||
- Started working on prototyping stuff for the new proof system
|
||||
Loading…
x
Reference in New Issue
Block a user