add Codex weekly reports for past 9 weeks

This commit is contained in:
jessiebroke 2024-07-10 01:03:21 -04:00
parent 3b7eb69de4
commit b245f2c693
No known key found for this signature in database
GPG Key ID: D901DC638A938F8C
9 changed files with 582 additions and 0 deletions

View File

@ -0,0 +1,71 @@
---
title: 2024-05-13 Codex Weekly
tags:
- codex-updates
date: 2024-05-13
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update May 7th - May 13th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Organized new Kanban board to sort through new and existing Epics and Issues associated with ongoing work to test & fix bugs in the core client towards stability - two large items of focus for the coming two months is:
- EthCC Workshop Client work
- First release of Codex for the testnets
- Fixed upload/addition of blocks but did not fix it in the advertising loop yet
- Finished evaluation of LevelDB vs SQLite
Ongoing
- Splitting two big PRs into a series of smaller ones
- [Expiry per dataset](https://github.com/codex-storage/nim-codex/pull/678)
- [Safe block deletion (with ref count)](https://github.com/codex-storage/nim-codex/pull/631)
## Marketplace
Completed
- New proposal for slot reservations and capturing the original bloem method idea; developed simplified version of slot reservations opting to allow for emergent behaviors before adding to complexity
- Completed PR Reviews and merged:
- [Feat: expiry specified with number of seconds](https://github.com/codex-storage/nim-codex/pull/793)
- [Feat: expiry specified as duration](https://github.com/codex-storage/codex-contracts-eth/pull/99)
- Follow up fix with adding confirmation: https://github.com/codex-storage/nim-codex/pull/802
- Updated Marketplace sections of Codex Whitepaper
Ongoing
- Tokenomics meeting to discuss existing questions as well as the new slot reservations proposal
- Discussion with Research team about fill reward curve
- Application Properties for Codex contract
- Learning about Properties and how to write them in Certora system
- New UI team sync
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- Wrote down current thinking about [tracking sets of proofs](https://hackmd.io/@bkomuves/SyPHG0PfR) (WIP)
- Notes on [Tracking proofs in Codex](https://hackmd.io/@bkomuves/SyPHG0PfR)
Ongoing
- Learning about different FFT / NNT algorithms
- Continued work on [Plonk notes](https://github.com/codex-storage/zk-research-artifacts/blob/master/notes/plonk/plonk-notes.pdf)
- Current focus is mainly on recursive proofs in the elliptic curve setting

View File

@ -0,0 +1,67 @@
---
title: 2024-05-20 Codex Weekly
tags:
- codex-updates
date: 2024-05-20
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update May 14th - May 20th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Packaged self-contained LevelDB Nim project so no dependencies on system packages lifted from existing wrapper and then wrapped it in a datastore implementation
- Preliminary tests shows increased performance is as expected
- Resolved issue with dependency bump and broken dependency chain and now stuck with serialization issue - bug was related to resolving to wrong deserializer and work on nim-serde and nim-RPC
Ongoing
- Started work on LevelDB migration; Discord bot, reward system and chain inspection broken
- Resolving failing integration tests re: expiry of the request in storage
- Working on issue related to inability to see amount of tokens sent in block explorer
- Deploying faucet web app for testnet tokens (ETH & TST)
- Working on Codex release workflow
- Trying to use DST's k8s cluster and have not been able to use it yet
- Migrate dist test cluster from Digital Ocean to Hetzner and deploy different mointoring tools (e.g., Victoria Metrics) - parametrics logs replacing elastic search with similar API and CLI scrollingto be added to UI later on
## Marketplace
Completed
- Helped with some work related to LevelDB integration
- Added custom errors PR in nim-ethers that required attention
Ongoing
- Writing documentation around some of the Solidity code ahead of audit
- Coordinating audit with SC team (Vac) and looping in Security; still deciding on 2nd security audit company
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- Sync w/ Client team about `circom-compat`
- Studying bulletproofs with the goal to try to use them in proof recursion, on the "wrong" curve - [see writeup here](https://hackmd.io/@bkomuves/BkBm9X0zC)
- Updated thinking about [proof recursion](https://hackmd.io/@bkomuves/rJ-bGisfC)
- Wrote a [high-level overview](https://github.com/codex-storage/codex-storage-proofs-circuits/blob/master/workflow/PROOFS.md) explaining all the moving parts of a ZK proofs (different files and steps and their meaning)
Ongoing
- Proposal for tracking proofs (more precisely, the multiset commitments in there) has a scaling problem, and started thinking about an alternative motivated by Plookup
- Looking more into Winograd-type FFT

View File

@ -0,0 +1,67 @@
---
title: 2024-05-27 Codex Weekly
tags:
- codex-updates
date: 2024-05-27
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update May 21th - May 27th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- N/A
Ongoing
- Continued work on LevelDB migration; [failing unit test uncovered bug](https://github.com/codex-storage/nim-codex/pull/816) on race condition in one of the state machines in the Marketplace module and switch to LevelDB reveleaed it and changing timing of DB reproduced it; reproduced it in SQLite by putting a sleep statement and nothing wrong in LevelDB
- Marketplace picked this bug up
- Docker build problems; Rust build failed and Circom devices no longer building - rolled back Codex to previous Docker image and it doesn't build anymore so not something changed on our end
- Working on debugging Rust FFI threading
## Marketplace
Completed
- [Contract deployment and security writeup](https://notes.status.im/s/D1AWG1FY-)
- [More adjustments to the slot queue PR, and merge](https://github.com/codex-storage/nim-codex/pull/752)
- [Re-review of ethers custom errors PR](https://github.com/codex-storage/nim-ethers/pull/69)
- [Merge fix for forcing scope of deserialization](https://github.com/codex-storage/nim-serde/pull/24)
- [Release new version of nim-serde to unblock chronos v4](https://github.com/codex-storage/nim-serde/pull/25)
- [Bump serde version and merge](https://github.com/codex-storage/nim-ethers/pull/73)
- Merged [Remove overloaded UInt256.fromJson](https://github.com/codex-storage/nim-ethers/pull/74)
- Cleanup [integration tests review](https://github.com/codex-storage/nim-codex/pull/757)
- Addressed [Slot Reservations](https://github.com/codex-storage/codex-research/pull/190) PR comments with some rewrites
Ongoing
- "Application Properties" writeup for Certora issues [tracked here](https://github.com/codex-storage/codex-contracts-eth/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3ACertora) and [ongoing discussion of issues here](https://github.com/codex-storage/codex-contracts-eth/issues/103)
- Reviewing [RFC spec for Marketplace](https://github.com/vacp2p/rfc-index/pull/36)
- Building frontend for the EthCC Demo that shows past and real time contract events, and downloads CIDs and displays as images
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- Initial [Dynamic data proposal WIP](https://hackmd.io/@bkomuves/ByvVCpYNR)
Ongoing
- Understanding aggregator node hybrid-system dynamics

View File

@ -0,0 +1,73 @@
---
title: 2024-06-03 Codex Weekly
tags:
- codex-updates
date: 2024-06-03
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update May 28th - June 3rd
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Added custom Docker file for release workflow
Ongoing
- EthCC planning meeting with Marketplace
- Continued bug hunting and fixing towards stability of client
- Planning around EthCC Codex demo logistics; Dockerized Codex in VMs, cloud VPN, and physical MikroTik router setup
- Naming conventions of testnet deployments
- NAT Traversal issues; AutoNAT issues not working as expected - formal writeup to come describing issues and potential solutions
- Challenges with DHT and peer discovery
- Fast-track reviews of 'rework async iter', 'safe block deletion' and 'expiry per dataset' PRs
- Explore scalability issues with LevelDB and large node tests
## Marketplace
Completed
- Marketplace discussion meeting ([release branches/versions](https://github.com/codex-storage/nim-codex/issues/747), [conditional deployments](https://discord.com/channels/895609329053474826/1244983945867694123/1244993023507435604), pointer calculation, contract gas estimates, [freezing/upgradability legal implications]((https://discord.com/channels/895609329053474826/1230457525854539819/1245555994465931294)))
- [Add freezing functionality for emergency use cases](https://github.com/codex-storage/codex-contracts-eth/issues/111)
- Call with Legal for POV of Contract's security and upgradability
- Call with SC team (Vac) about Legal POV for Contract's security and upgradability
- Marketplace team meeting with Legal about [contract freezing/upgradability](https://github.com/codex-storage/codex-contracts-eth/issues/111)
- Review of [Bug/sales race PR](https://github.com/codex-storage/nim-codex/pull/816/files)
- Sales module concurrency issue - reworking fix after feedback
Ongoing
- More progress on [EthCC Demo frontend](https://github.com/codex-storage/ethcc-demo)
- Gas reporting plugin attempt to run against Codex integration tests
- Investigate deploying unchanged contracts using hardhat-deploy and newer hardhat Ignition (created [issue for upgrade](https://github.com/codex-storage/codex-contracts-eth/issues/109))
- Frontend dev application reviews
- WIP in the Certora's Application Properties work
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- L2 Deployment
Ongoing
- Proxy re-encryption call with ACZ (Vac)
- Interviewing ZK engineer candidates
- [Encryption options WIP](https://hackmd.io/@bkomuves/rJyHMrU4A)
- [Set commitment proposal with details WIP](https://hackmd.io/@bkomuves/SJXrDIw40)

View File

@ -0,0 +1,64 @@
---
title: 2024-06-10 Codex Weekly
tags:
- codex-updates
date: 2024-06-10
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update June 4th - June 10th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Setup similar VPN from home to cloud VPN for router OS similar to MikroTik router and APs; purchased hardware to test - ordered physical SIM adapter for router
Ongoing
- Workshop preparation
- Ongoing scalability work; intially had to fix the Docker build and got updated build with LevelDB, odd stuff happening in cluster but 4.5GiB exchange for upload/download succeeded
- NAT Planning and roll out
## Marketplace
Completed
- TBD
Ongoing
- Workshop preparation
- Certora setup & integration
- WIP in the Certora's Application Properties work
- Persistent Availabilities
- Slot Reservations
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- Ongoing work on [details of the dynamic data proposal](https://hackmd.io/M6uwhb0eQh2bIh5_O6AX7Q) (the proof parts)
- Helped Nomos with a small FFT issue
Ongoing
- Looking at [new research](https://zkmesh.substack.com/p/zkmesh-may-2024-recap)
- Working on Winograd-type NTT
- Working on proof recursion
- Looking into more FFT/NTT algorithms (ECFFT, prime order FFT, etc; we need non-power-of-two sizes for foreign field arithmetic)
- [Reliability and Proof Frequency (WIP)](https://colab.research.google.com/gist/cskiraly/e6c87427166323af02a0b4d4442fc61f/codex-reliability-and-proof-frequency.ipynb)

View File

@ -0,0 +1,61 @@
---
title: 2024-06-17 Codex Weekly
tags:
- codex-updates
date: 2024-06-17
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update June 11th - June 17th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- [L2 assumptions, planning and deployment options](https://github.com/orgs/codex-storage/projects/3/views/11?pane=issue&itemId=66247672)
- [NAT Traversal in Codex](https://hackmd.io/UxYaQ3dqS1CUU8pxO6-PuQ)
Ongoing
- Workshop preparations
- Developing single binary pipeline for releases
- Some splitting of PRs further into datastore and rework
- [Need this Safe block deletion (with ref count) PR approved](https://github.com/search?q=org%3Acodex-storage+631&type=pullrequests)
## Marketplace
Completed
- Merged [fix: createReservation lock](https://github.com/codex-storage/nim-codex/pull/825)
- [L2 overview and Marketplace support/compatibility check](https://github.com/codex-storage/codex-research/pull/192/files)
Ongoing
- Workshop preparations
- Looking into more research about contract upgradability and industry practices
- [Reviews of Marketplace spec](https://github.com/vacp2p/rfc-index/pull/36)
- WIP in the Certora's Application Properties work
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- Figuring out usable NTT options on the Grumpkin curve; [WIP notes](https://hackmd.io/@bkomuves/BJe6hiZBC)
Ongoing
- Prototyping with intent of a full implementation soon

View File

@ -0,0 +1,62 @@
---
title: 2024-06-24 Codex Weekly
tags:
- codex-updates
date: 2024-06-24
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update June 18th - June 24th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Merged safe block deletion PR
Ongoing
- Workshop preparations
- Slides presentation and complete demo run-through
- Manifest initialization bug
- Crash fixing; second release failing because of CPU load - possible prover blocking I/O and need better threading (to use async profiler to resolve this)
- Work on scheduler
- Working on bug fix for EC issues
- Problem with verifiable manifest (was not enabled for Athens demo) & tree roots issue
- Crashes resulting in SIGSEGV error
## Marketplace
Completed
- Hired full-stack dev to work on Codex App and some Client/Marketplace tasks
Ongoing
- Workshop preparations
- Frontend session cookies issue - CORS requests problems; sticky sessions are problem for replicated services because reqeusts can be sent to different sessions, problem is how to persist RPC session w/ 3 RPC nodes and if you want to send all requests to same RPC node is problematic
- WIP in the Certora's Application Properties work
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- [Plausible deniability writeup from ACZ (Vac)](https://www.notion.so/Approaches-to-plausible-deniability-87c6fef92df946fcbc1327d51d936ce1)
Ongoing
- [Dynamic data proposal WIP](https://hackmd.io/M6uwhb0eQh2bIh5_O6AX7Q)

View File

@ -0,0 +1,60 @@
---
title: 2024-07-01 Codex Weekly
tags:
- codex-updates
date: 2024-07-01
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update June 25th - July 1st
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Release pipeline for Linux setup (x64 & ARM for Ubuntu and MacOS)
Ongoing
- Workshop preparations
- Slides presentation and complete demo run-through
- General bug hunting and fixing towards baseline stability and BitTorrent-like performance
## Marketplace
Completed
- Workshop preparations
Ongoing
- WIP in the Certora's Application Properties work
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- [Gathered all ZK related documents here](https://www.notion.so/Codex-ZK-stuff-ca2a503baa504e99ab489b345d412361)
- [Writeup on OTP-like encryption to prevent outsourcing attacks](https://hackmd.io/prAbwLpUReqR4OaDLfeeMg)
Ongoing
- Ongoing research into the details of folding
- Interviewing more ZK engineer candidates
- Creating take-home programming exercise
- Thinking about how to describe Plonkish circuits
- Looking at [new ZK research](https://zkmesh.substack.com/p/zkmesh-june-2024-recap)

View File

@ -0,0 +1,57 @@
---
title: 2024-07-08 Codex Weekly
tags:
- codex-updates
date: 2024-07-08
lastmod: 2024-07-09
draft: false
description: Weekly update of Codex
---
# Codex Update July 2nd - July 8th
> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC.
>
> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation.
Development is currently broken into three distinct teams:
1. Client, Testing, and Infrastructure
2. Marketplace
3. Research
The following are their team updates.
## Client, Testing and Infrastructure
Completed
- Workshop preparations
- Slides presentation and complete demo run-through
- Release pipeline for Windows (x64 & ARM) setup
Ongoing
- General bug hunting and fixing towards baseline stability and BitTorrent-like performance
- [Writeup of Merkle Trees and Proof Tree](https://hackmd.io/DaLh-BU4R7WVblRVkCRF8w)
- [EC in Codex](https://hackmd.io/XlsefH1ZTcWwaU6ngwBMcg)
## Marketplace
Completed
- Workshop preparations
Ongoing
- WIP in the Certora's Application Properties work
## Research
```
2024 R&D Goals
1. Proving system and aggregation improvements (folding or lookups)
2. Aggregator/validator design
3. DHT improvements
4. Tokenomics and incentive design
5. Bandwidth incentives
6. Dynamic data (appendable data)
7. Encryption
```
Completed
- N/A
Ongoing
- N/A