From adff7f4dce9bdbe9f55bff9af833c65e8dcf0934 Mon Sep 17 00:00:00 2001 From: jessiebroke <18238304+jessiebroke@users.noreply.github.com> Date: Sun, 12 May 2024 21:30:23 -0400 Subject: [PATCH] add Codex weekly reports for March 18 to May 6th --- content/codex/updates/2024-03-18.md | 88 ++++++++++++++++++++++++ content/codex/updates/2024-03-25.md | 99 ++++++++++++++++++++++++++ content/codex/updates/2024-04-01.md | 103 ++++++++++++++++++++++++++++ content/codex/updates/2024-04-08.md | 22 ++++++ content/codex/updates/2024-04-15.md | 22 ++++++ content/codex/updates/2024-04-22.md | 44 ++++++++++++ content/codex/updates/2024-04-29.md | 54 +++++++++++++++ content/codex/updates/2024-05-06.md | 65 ++++++++++++++++++ 8 files changed, 497 insertions(+) create mode 100644 content/codex/updates/2024-03-18.md create mode 100644 content/codex/updates/2024-03-25.md create mode 100644 content/codex/updates/2024-04-01.md create mode 100644 content/codex/updates/2024-04-08.md create mode 100644 content/codex/updates/2024-04-15.md create mode 100644 content/codex/updates/2024-04-22.md create mode 100644 content/codex/updates/2024-04-29.md create mode 100644 content/codex/updates/2024-05-06.md diff --git a/content/codex/updates/2024-03-18.md b/content/codex/updates/2024-03-18.md new file mode 100644 index 000000000..9e4a75318 --- /dev/null +++ b/content/codex/updates/2024-03-18.md @@ -0,0 +1,88 @@ +--- +title: 2024-03-18 Codex Weekly +tags: + - codex-updates +date: 2024-03-18 +lastmod: 2024-03-18 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Mar 12th - Mar 18th + +> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. +> +> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work. +Development is currently broken into three distinct teams: + +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research + +The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics. + +## Client, Testing and Infrastructure + +### Client +Completed +- Marked PR as ready - scheduling erasure encoding and decoding on a thread pool https://github.com/codex-storage/nim-codex/pull/716 +- Put together [reasoning on why the profiler works the way it does](https://hackmd.io/fA4RXNWsQZSSgYLNcIksEw); +- Closed [a bug re: double lookups when block does not exist](https://github.com/codex-storage/nim-codex/pull/739); + + +Ongoing +- Started a PR to fix Atlas after SAT integration +- Finishing basic fixes PR in Atlas https://github.com/nim-lang/atlas/pull/120 +- Started work on scheduling prover computation on a thread pool +- Fixing comments and rebasing large PRs: + - https://github.com/codex-storage/nim-codex/pull/631 + - https://github.com/codex-storage/nim-codex/pull/678 +- Work on Frontend + - Implemented dropdown list for expiry input + - Started working on caching node information for displayed information +- Some triaging of bugs in the Codex repo, still some way to go; + +### Infrastructure +Completed +- ✅ [Install VictoriaMetrics in Testnet cluster #130]() +- ✅ [Adjust number of nodes in Testnet cluster #136]() +- ✅ [Redeploy Testnet cluster components #138]() +- ✅ [Install ELK in Testnet cluster #135]() + +Ongoing +- �️ [Install Oauth2 proxy in Testnet cluster #141]() +- �️ [Deploy Testnet cluster components #57]() + +## Marketplace + +Completed +- Discussions about ceremony files retrival +- PR Reviews: + - https://github.com/codex-storage/codex-storage-proofs-circuits/pull/7 + - https://github.com/codex-storage/codex-contracts-eth/pull/83 + - https://github.com/codex-storage/nim-codex/pull/730 + +Ongoing +- WIP on merging https://github.com/codex-storage/nim-codex/pull/692 +- WIP on persistant availabilities + +## Research +``` +2024 R&D Goals +1. Proving system and aggregation improvements (folding or lookups) +2. Aggregator/validator design +3. DHT improvements +4. Tokenomics and incentive design +5. Bandwidth incentives +6. Dynamic data (appendable data) +7. Encryption +``` +Completed +- EthResearch post on sampling techniques (https://ethresear.ch/t/lossydas-lossy-incremental-and-diagonal-sampling-for-data-availability/18963) +- DAS simulator (https://github.com/codex-storage/das-research) paper prep (calls + work) +- Shadow based P2P emulator for DAS (https://github.com/cskiraly/dst-gossipsub-test-node/tree/FullDAS) +- Looked into how circom handles custom gates (tldr: it's easy to abuse for our own purposes, but snarkjs doesn't implement anything) +- Minor refactor the proof circuit [PR](https://github.com/codex-storage/codex-storage-proofs-circuits/pull/7) + +Ongoing +- Started working on prototyping for the new proof system \ No newline at end of file diff --git a/content/codex/updates/2024-03-25.md b/content/codex/updates/2024-03-25.md new file mode 100644 index 000000000..0dd549d98 --- /dev/null +++ b/content/codex/updates/2024-03-25.md @@ -0,0 +1,99 @@ +--- +title: 2024-03-25 Codex Weekly +tags: + - codex-updates +date: 2024-03-25 +lastmod: 2024-03-25 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Mar 19th - Mar 25th + +> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. +> +> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work. +Development is currently broken into three distinct teams: + +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research + +The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics. + +## Client, Testing and Infrastructure + +### Client +Completed +- Various fixes for prover-verifier integration + - https://github.com/codex-storage/nim-codex/pull/702 +- Split prover-verifier integration into several PRs and merged them into master + - https://github.com/codex-storage/nim-codex/pull/702#issuecomment-1986256598\ + +Ongoing +- Syncing up with testnet owners to help think through our use cases and whether or not we have gaps in our QA/testing plan; +- Put together a [small tutorial on deploying contracts on a local Geth network](https://hackmd.io/qdgaIroiSbmZc14g6kzBQQ) with [some code](https://github.com/gmega/geth-net-fun). + +### Testing +Completed +- Fixed proof generation is dist-test environment +- Fixed crash in dist-test env related to blockchain inspection +- Bring up-to-date continuous tests (enable transient-node and marketplace test scenarios) +Ongoing +- Why do proofs with EC params 1-0 fail? +- Node response to fill-slot call is unreliable in geth env +- Wrap up discord-reward bot + +### Infrastructure +Completed +- ✅ [Install Oauth2 proxy in Testnet cluster #141]() +- ✅ [Deploy Testnet cluster components #57]() +- ✅ [Install Vector in Testnet cluster #144]() +- ✅ [Configure monitoring in Testnet cluster #147]() +- ✅ [Send Codex and Geth Bootstrap nodes logs and metrics to the Testnet cluster #148]() + +Ongoing +- �️ [Configure Grafana dashboards for Testnet resources #154]() +- �️ [Add Release workflow for Codex #749]() + +## Marketplace +Completed +- nim-ethers + - Support Solidity's custom errors + - https://github.com/codex-storage/nim-ethers/pull/69 + - Update dependencies + - https://github.com/codex-storage/nim-ethers/pull/70 + - Fix flaky test + - https://github.com/codex-storage/nim-ethers/pull/71 + - Merged fix for Solidity getters + - https://github.com/codex-storage/nim-ethers/pull/63 +- codex-contracts-eth: + - Finished verifier refactor: + - https://github.com/codex-storage/codex-contracts-eth/pull/83 +- codex-storage-proofs-circuits: + - Merged code review PR: + - https://github.com/codex-storage/codex-storage-proofs-circuits/pull/5 +- questionable: + - Worked out a way to handle generic Results in `without` statements: + - https://github.com/codex-storage/questionable/pull/59 + +Ongoing +- See previous week + +## Research +``` +2024 R&D Goals +1. Proving system and aggregation improvements (folding or lookups) +2. Aggregator/validator design +3. DHT improvements +4. Tokenomics and incentive design +5. Bandwidth incentives +6. Dynamic data (appendable data) +7. Encryption +``` +Completed +- wrote an initial draft notes about Plonk; [github (WIP)](https://github.com/codex-storage/zk-research-artifacts/blob/master/notes/plonk/plonk-notes.pdf) + +Ongoing +- working towards implementing a Plonkish prover with custom accelerator gadgets, so that we can test proof system ideas +- realized that proof system idea has a botleneck; may still speed up Poseidon hashing, but the new botleneck is much worse for binary circuits like SHA256. To make the latter practical needs new ideas. \ No newline at end of file diff --git a/content/codex/updates/2024-04-01.md b/content/codex/updates/2024-04-01.md new file mode 100644 index 000000000..095099fa7 --- /dev/null +++ b/content/codex/updates/2024-04-01.md @@ -0,0 +1,103 @@ +--- +title: 2024-04-01 Codex Weekly +tags: + - codex-updates +date: 2024-04-01 +lastmod: 2024-04-01 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Mar 26th - Apr 1st + +> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q1/Q2 public testnet release. An internal testnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. +> +> Ongoing and new lines of research and development will soon begin in preparation for the next version of Codex that will be used for the mainnet release.. Here are the updates from different team members and their ongoing work. +Development is currently broken into three distinct teams: + +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research + +The different teams have actively moving on various fronts. The following are their team updates to various ongoing Epics. + +## Client, Testing and Infrastructure + +### Client +Completed +- Created task for NAT traversal and gathered various info for it https://github.com/codex-storage/nim-codex/issues/753 +- Finished work on scheduling erasure coding on a thread pool https://github.com/codex-storage/nim-codex/pull/716 + +Ongoing +- Working on scheduling prover computation on a thread pool, draft PR in progress +- Addressing comments for large PRs + - https://github.com/codex-storage/nim-codex/pull/631 + - https://github.com/codex-storage/nim-codex/pull/678 + +### Testing +Completed +- Documented issue for running Codex integration tests on certain macOS configurations https://github.com/codex-storage/codex-contracts-eth/issues/95 + +### Infrastructure +Completed +- None + +Ongoing +- � [Install Fluent Bit to get Testnet Kubernetes events]() +- � [Adjust Vector config]() +- � [Deploy ethereum web wallet for Testnet]() +- � [Run Public RPC node on Testnet]() +- � [Add more Codex Storage nodes]() +- � [Use multi-stage builds for Rewarder #101]() +- � [codex-testnet-starter updates]() +- �️ [Configure Grafana dashboards for Testnet resources #154]() +- �️ [Add Release workflow for Codex #749]() + +### Nim Tooling +Completed +- Add recursion limit for SAT (packaging) https://github.com/nim-lang/sat/pull/3 +- Disable stacktrace on SAT solver to avoid stackoverflows in debug builds https://github.com/nim-lang/sat/pull/5#event-12262613353 +- Wrote up draft for Nimble Caching RFC https://hackmd.io/@elcritch/rJ4VrExJR + +## Marketplace +Completed +- Revisit design of https://github.com/codex-storage/nim-codex/issues/467 +- Review https://github.com/codex-storage/nim-codex/pull/692 +- Review https://github.com/codex-storage/nim-codex/pull/678, many reviews and ongoing discussion +- Review https://github.com/codex-storage/nim-codex/pull/730 +- Chore to remove no longer used compilation flag: https://github.com/codex-storage/nim-codex/pull/741 +- Tried to understand why an integration was indeterminately failing in CI and locally, thought was very difficult to reproduce +- Merged https://github.com/codex-storage/nim-codex/pull/704, which had the integration test failure. Re-running the failed test in CI was enough to get it to merge, but indicates there is an issue with this test +- Rebased and merged MarketError PR: https://github.com/codex-storage/nim-codex/pull/670 +- Updated pausing slot queue design PR: https://github.com/codex-storage/codex-research/pull/188/ +- Re-reviewed [feat[marketplace]: add slot queue pausing PR #752](https://github.com/codex-storage/nim-codex/pull/752) in Codex and started review of [Expiry per dataset PR #678](https://github.com/codex-storage/nim-codex/pull/678/) +- Added protection branch rule for `nim-ethers` in Codex +- Reviewed PRs [#757](https://github.com/codex-storage/nim-codex/pull/757), [#752](https://github.com/codex-storage/nim-codex/pull/752), [#760](https://github.com/codex-storage/nim-codex/pull/760), [#69](https://github.com/codex-storage/nim-ethers/pull/69/), and [#70](https://github.com/codex-storage/nim-ethers/pull/70) in Codex +- Reviewed comments for the Availability Patch PR and [Pausing Queue PR](https://github.com/codex-storage/nim-codex/pull/752/files) in Codex + +Ongoing +- Discussed with team plans to improve integration tests: what they should cover, parallelism, and whether or not some could be replaced with unit tests. Also discussed some open issues and PR comments. +- Continued work on implementation of slot queue pausing. Most implementation in, need to add tests: https://github.com/codex-storage/nim-codex/tree/feat/marketplace/slot-queue-improvements +- Worked on updating a macro for Optionalize to add the serialize pragma annotation on the type. Tried to get the macro to read existing serialize annotations on the type and fields first but was taking too long, so moved on +- Discussions about release branches, proof circuit assets, and versioning +- Discussed `Optionalize()` macro and `serialize` pragma +- Discussed the parallelism of integration tests +- Progress on Persistent Availabilities implementation + +## Research +``` +2024 R&D Goals +1. Proving system and aggregation improvements (folding or lookups) +2. Aggregator/validator design +3. DHT improvements +4. Tokenomics and incentive design +5. Bandwidth incentives +6. Dynamic data (appendable data) +7. Encryption +``` +Completed +- None + +Ongoing +- Trying to understand what can be salvaged from my binary circuit idea (by implementing and measuring stuff) +- Working on the Plonk notes \ No newline at end of file diff --git a/content/codex/updates/2024-04-08.md b/content/codex/updates/2024-04-08.md new file mode 100644 index 000000000..15eca55d0 --- /dev/null +++ b/content/codex/updates/2024-04-08.md @@ -0,0 +1,22 @@ +--- +title: 2024-04-08 Codex Weekly +tags: + - codex-updates +date: 2024-04-08 +lastmod: 2024-04-08 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Apr 2nd - Apr 8th + +> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q2 public testnet release. An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. +> +> The Codex team is currently attending the week long Logos all hands off-site, attending ZK Summit and having their team specific off-site the following week. + +## Logos and Codex Off-site Links (2024/04/01 - 2024/04/15) +The following are links containing artifacts from both off-sites: +- [Codex Athens Presentation](https://docs.google.com/presentation/d/1c3iNOHxwbxeRpwHFTZPBBdXaa6nwLd4EbwJ5txlR0qU/edit?usp=drive_link) +- [Codex Athens Presentation Video Recording](https://drive.google.com/file/d/1XOzTYAl29KQCUQovojpHpBjW4Kelv2sr/view?usp=drive_link) +- [Codex Workshop Presentation](https://docs.google.com/presentation/d/1farcX6P7cJsK_p6KVc63fmT-3dR3cPXyW2Nj7tJ2xXQ/edit?usp=drive_link) +- [Codex Athens Off-site Artifacts](https://drive.google.com/drive/folders/1nsJCirfJX0B8p7Tr8eEaaa9tOTHme912?usp=drive_link) \ No newline at end of file diff --git a/content/codex/updates/2024-04-15.md b/content/codex/updates/2024-04-15.md new file mode 100644 index 000000000..815409f7f --- /dev/null +++ b/content/codex/updates/2024-04-15.md @@ -0,0 +1,22 @@ +--- +title: 2024-04-15 Codex Weekly +tags: + - codex-updates +date: 2024-04-15 +lastmod: 2024-04-15 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Apr 9th - Apr 16th + +> The Codex team continues to make progress with various initiatives to wrap up the demo for the Q2 public testnet release. An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. +> +> The Codex team is currently attending the week long Logos all hands off-site, attending ZK Summit and having their team specific off-site the following week. + +## Logos and Codex Off-site Links (2024/04/01 - 2024/04/15) +The following are links containing artifacts from both off-sites: +- [Codex Athens Presentation](https://docs.google.com/presentation/d/1c3iNOHxwbxeRpwHFTZPBBdXaa6nwLd4EbwJ5txlR0qU/edit?usp=drive_link) +- [Codex Athens Presentation Video Recording](https://drive.google.com/file/d/1XOzTYAl29KQCUQovojpHpBjW4Kelv2sr/view?usp=drive_link) +- [Codex Workshop Presentation](https://docs.google.com/presentation/d/1farcX6P7cJsK_p6KVc63fmT-3dR3cPXyW2Nj7tJ2xXQ/edit?usp=drive_link) +- [Codex Athens Off-site Artifacts](https://drive.google.com/drive/folders/1nsJCirfJX0B8p7Tr8eEaaa9tOTHme912?usp=drive_link) \ No newline at end of file diff --git a/content/codex/updates/2024-04-22.md b/content/codex/updates/2024-04-22.md new file mode 100644 index 000000000..e25bcc73a --- /dev/null +++ b/content/codex/updates/2024-04-22.md @@ -0,0 +1,44 @@ +--- +title: 2024-04-22 Codex Weekly +tags: + - codex-updates +date: 2024-04-22 +lastmod: 2024-04-22 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Mar 16th - Mar 22nd +> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC. +> +> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. + +Development is currently broken into three distinct teams: + +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research + +The following are their team updates. + +## Client, Testing and Infrastructure +- most members OOO + +## Marketplace +- most members OOO + +## Research +``` +2024 R&D Goals +1. Proving system and aggregation improvements (folding or lookups) +2. Aggregator/validator design +3. DHT improvements +4. Tokenomics and incentive design +5. Bandwidth incentives +6. Dynamic data (appendable data) +7. Encryption +``` + +Completed +- started looking into the Goldilocks field and Plonky2 (unfortunately, there appears to be NO documentation _at all_...) +- [wrote a draft document about our proof research needs](https://hackmd.io/@bkomuves/SJGyefxZR) \ No newline at end of file diff --git a/content/codex/updates/2024-04-29.md b/content/codex/updates/2024-04-29.md new file mode 100644 index 000000000..7ee81a5bf --- /dev/null +++ b/content/codex/updates/2024-04-29.md @@ -0,0 +1,54 @@ +--- +title: 2024-04-29 Codex Weekly +tags: + - codex-updates +date: 2024-04-29 +lastmod: 2024-04-29 +draft: false +description: Weekly update of Codex +--- + +# Codex Update Mar 22nd - Mar 29th +> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC. +> +> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. + +Development is currently broken into three distinct teams: + +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research + +The following are their team updates. + +## Client, Testing and Infrastructure +- Members OOO + +## Marketplace +Completed +- Work on duration for expiry + - nim-codex PR https://github.com/codex-storage/nim-codex/pull/793 + - contracts PR https://github.com/codex-storage/codex-contracts-eth/pull/99 + +Ongoing +- whitepaper review +- Tokenomics discussions + +## Research +``` +2024 R&D Goals +1. Proving system and aggregation improvements (folding or lookups) +2. Aggregator/validator design +3. DHT improvements +4. Tokenomics and incentive design +5. Bandwidth incentives +6. Dynamic data (appendable data) +7. Encryption +``` +Completed +- None + +Ongoing +- Whitepaper review (90% done) +- Coordinating Vac cooperation +- Tried to look a little bit into set accumulator schemes \ No newline at end of file diff --git a/content/codex/updates/2024-05-06.md b/content/codex/updates/2024-05-06.md new file mode 100644 index 000000000..bbe9fe07f --- /dev/null +++ b/content/codex/updates/2024-05-06.md @@ -0,0 +1,65 @@ +--- +title: 2024-05-06 Codex Weekly +tags: + - codex-updates +date: 2024-05-06 +lastmod: 2024-05-06 +draft: false +description: Weekly update of Codex +--- + +# Codex Update April 30th - May 6th +> The Codex team wrapped up a successful demo at the Logos off-site and now aims to prepare the demo for the Q2 public testnet release potentially coinciding with EthCC. +> +> An internal devnet has been running for the past few weeks and has been used to test the latest version of Codex and can be accessed using the [Codex Testnet Starter](https://github.com/codex-storage/codex-testnet-starter) documentation. + +Development is currently broken into three distinct teams: + +1. Client, Testing, and Infrastructure +2. Marketplace +3. Research + +The following are their team updates. + +## Client, Testing and Infrastructure +Completed +- None + +Ongoing +- Splitting two big PRs into a series of smaller ones + - [Expiry per dataset](https://github.com/codex-storage/nim-codex/pull/678) + - [Safe block deletion (with ref count)](https://github.com/codex-storage/nim-codex/pull/631) +- Reviewing various PRs in the client + +## Marketplace +Completed +- Tokenomics meeting to discuss existing questions, as well as the new slot reservations proposal +- Discussion about the slot reservations proposal, in which we came up with a simplified version, opting to allow for emergent behaviors before adding to complexity +- Discussion about fill reward curve +- Discussion about the new slot reservations proposal +- Reviews : + - https://github.com/codex-storage/nim-codex/pull/793 + - https://github.com/codex-storage/codex-contracts-eth/pull/99 + +Ongoing +- Writing out new proposal for slot reservations as well as capturing the original bloem method idea +- Tokenomics & Marketplace proposal discussions + +## Research +``` +2024 R&D Goals +1. Proving system and aggregation improvements (folding or lookups) +2. Aggregator/validator design +3. DHT improvements +4. Tokenomics and incentive design +5. Bandwidth incentives +6. Dynamic data (appendable data) +7. Encryption +``` +Completed +- Discussion with marketplace team about slot fill reward and address window expansion curves + +Ongoing +- Studying various FFT/NTT algorithms +- Looking at [new research](https://zkmesh.substack.com/p/zkmesh-apr-2024-recap) +- Looking a bit into the recent "Foreign Arithmetic" (SigmaBus) paper. Unfortunately it seems to be much less useful for us than originally thought. \ No newline at end of file