updates to monthlies

content/codex/monthly-reports/2023-sept.md

@@ -7,17 +7,61 @@ lastmod: 2023-09-18
 ## Executive Summary
 
 ## Key Updates
-
+### Personnel
-### Personel
+A new Business Development job description [was posted](https://jobs.status.im/?gh_jid=5329400) and candidates are currently being interviewed. This role is expected to help facilitate strategy around the much needed partnerships for Codex and laisse with the other BD related resources we have within Logos to ensure efficient communications. 
-- 
 
 ### Milestones
 
+The Codex team is broken up into 5 sections, and the weekly reports give details on how each of them have performed. Currently the Milestone definitions are not in line with this reporting process and will be worked on in the subsequent month. The teams are broken up into the following sections:
+- `Client`
+- `Infra`
+- `Marketplace`
+- `Research`
+- `DAS`
+
+Below is summary of key updates to these sections over the month of September 2023
+
+#### Client
+The client continues to push towards Milestone 1.3: Codex v1.0 - Katana, which is slotted to be completed by the end of the year. Significant work has been done on merkelization which is required in order to integrate the proving system, and can be followed in this [working branch](https://github.com/codex-storage/nim-codex/compare/master...sending-blocks-with-proofs-over-the-network). 
+
+The Block Exchange protocol got some attention and refinement. Notes on the associated thoughts can be found in these two writeups:
+- [Block Discovery Problem](https://rpubs.com/giuliano_mega/1067876)
+- [Codex Swarm Overlays](https://rpubs.com/giuliano_mega/1082104)
+
+Progress has stalled on the Block Exchange protocol. 
+
+Progress was made on the ability for Codex to manage asynchronous and threaded disk IO. In the process of doing this work, a bug within Nim's `SharedPtr` was [discovered](https://github.com/nim-lang/threading/issues/45) and [fixed](https://github.com/nim-lang/threading/pull/46).
+
+#### Infra
+A [Grafana](https://grafana.dist-tests.codex.storage/) and [Kibana](https://kibana.dist-tests.codex.storage/) instance were deployed in order to facilitate the various testing being done. 
+
+#### Marketplace
+In order to alleviate a concurrency issue with Data Availabilities in the contract, a [Reservation System](https://github.com/codex-storage/nim-codex/pull/535) has been proposed and worked on. This removes the previous constraint that current downloads was limited by the number of Availabilities.  
+
+#### Research
+The [Codex Whitepaper v0.1](https://docs.google.com/document/d/1LCy23m90IHf32aUVhRT4r4772w1BfVcSLaJ0z9VTw9A/edit#heading=h.qs3bayckj5u4) was drafted and scheduled for release in October 2023. It is currently under review and improving based on feedback. 
+
+#### DAS
+Work continues on the DAS research in coordination with the Ethereum Foundation (EF). As a result of SBC, a blog post was written by the EF that discussed a forward thinking proposal for[ _PeerDAS - a simpler DAS approach using battle-tested p2p components_](https://ethresear.ch/t/peerdas-a-simpler-das-approach-using-battle-tested-p2p-components/16541) which the team has contributed to (referenced inside). Conversations of relevancy continue. 
+
+A Codex Blog post [was published](https://blog.codex.storage/big-blocks-on-mainnet/) discussing two by-products of the DAS research: the characterization of big block diffusion latency on the existing Ethereum Mainnet and the existence of big organic blocks on Ethereum Mainnet and its implications. The conclusion is quoted below:
+
+>[!QUOTE] 
+>We have discovered a large number of big blocks (>250 KB) that occur organically every day in the Ethereum Mainnet. We have measured the propagation time of those blocks in three different world regions and compared their latency based on geographical location as well as block size. We have analysed how these propagation differences are reflected in the five CL clients separately, as they have different ways of reporting blocks. The empirical results measured in Ethereum Mainnet and presented in this work give us the first clear idea of how block propagation times might look when EIP-4844 is deployed, and 1 MB blocks become the standard and not the exception.
+>
+>In the future, we plan to continue with these block propagation measurements and monitor the behaviour of big blocks in the Ethereum network. Additionally, we want to help different CL clients harmonise their event recording and publication systems in order to be able to compare CL clients between them.
+
+Discussions with Felix Lange began around some fixes for `Discv5`. 
+
+### Other
+A [Codex YouTube channel](https://www.youtube.com/@CodexStorage) has been setup and many tutorial videos and conference talks were uploaded. 
 ## Perceived Changes in Project Risk
 
+
 ## Future Plans
 
 ### Insight
+Because of the mismatch of weekly updates with Milestone definitions, it is difficult to assess the impact of any given update. Next month should have all milestone definitions within this site
 
 ### Project
 
@@ -25,4 +69,4 @@ lastmod: 2023-09-18
 
 Weekly Reports
 - [[2023-09-15]]
-- 
+- [[content/codex/updates/2023-09-29|2023-09-29]]

content/codex/updates/2023-09-15.md

@@ -6,7 +6,6 @@ date: 2023-09-15
 lastmod: 2023-09-18
 draft: false
 ---
-
 ## Client
 
 ### Milestone: Block Merkelization

content/codex/updates/2023-09-29.md

@@ -0,0 +1,53 @@
+---
+title: 2023-09-29 Codex weekly
+tags:
+  - codex-updates
+date: 2023-09-29
+lastmod: 2023-09-30
+draft: false
+---
+## Client
+
+### Milestone: Block Merkelization
+- Merkelization current working branch 
+  - https://github.com/codex-storage/nim-codex/tree/sending-blocks-with-proofs-over-the-network
+
+### Milestone: Block Exchange protocol refinements and simulations
+- Not much done on it this past couple of weeks, progress can be tracked here when it resumes
+  - https://github.com/codex-storage/swarmsim
+
+### Milestone: Async Disc Access & Threading support
+- Work on IO threads support
+  - All related PRs are here https://github.com/codex-storage/nim-datastore/pulls
+    - We now have a clear idea of how to implement and integrate it - https://github.com/codex-storage/nim-codex/pull/552
+  - found a leak in Nim's `SharedPtr` https://github.com/nim-lang/threading/issues/45 and fix  https://github.com/nim-lang/threading/pull/46
+
+### Milestone: Client stability and debugging
+- Major effort to stabilize the Codex client through continuous automated testing
+  - Most team members are working on this on and off, testing is ongoing
+
+## Infra
+ - Installed cert-manager in Dist-Tests cluster
+ - Implemented External OAUTH Authentication for Grafana/Kibana (need to adjust internal authentication)
+ - Exposed [Grafana](https://grafana.dist-tests.codex.storage/) and [Kibana](https://kibana.dist-tests.codex.storage/)
+ - [Losing the logs during Continuous Tests run](https://github.com/codex-storage/infra-codex/issues/48)
+
+## Research
+
+### Milestone: Publications
+- White paper ongoing - https://docs.google.com/document/d/1LCy23m90IHf32aUVhRT4r4772w1BfVcSLaJ0z9VTw9A/edit#heading=h.qs3bayckj5u4
+
+### Milestone: Sampling for storage proofs
+- Large discussion around erasure coding for sampling
+  - https://github.com/codex-storage/zk-research-artifacts/blob/master/sampling/sampling.pdf
+  - https://hackmd.io/DxJzAuTZROulBhPWqScmCg?view
+  - https://github.com/codex-storage/codex-research/pull/184
+  - https://hackmd.io/kxSF8wjPS3arDFcqFJrNDw
+
+## DAS
+### Milestone: DAS Design
+- Ongoing discussions around - https://ethresear.ch/t/peerdas-a-simpler-das-approach-using-battle-tested-p2p-components/16541
+- space-DAS (don't have link), a proposal to use spacefilling curves for sample placement
+
+### Milestone: Measurements and simulations
+- Work continues on simulating various aspects of the DHT in https://github.com/cortze/py-dht

content/nomos/monthly-reports/2023-sept.md

@@ -8,7 +8,7 @@ lastmod: 2023-09-27
 
 ## Key Updates
 
-### Personel
+### Personnel
 - Filip has joined as a Project Manager to assist in various activities. His position is to sit in between the Insights team and the Nomos project to facilitate development tracking and resource allocation. It is anticipated that his involvement will speed up development as current resources are then freed to focus on research and development. Until demands within Nomos require full-time engagement from him, he will also be assisting with Vac Program Management. 
 - After a lot of candidate interviews, Mehmet was offered a position and accepted to focus on the privacy and cryptography research needs within the project. His background in cryptography and security auditing of popular zero-knowledge platforms is expected to be very useful in aiding to architect Nomos. Mehmet's expecting starting date is ???
 	- Another candidate for this role that was considered, Ramses, has joined the Vac team and will initially be aiding in this work and growing the relationship with their [[vac/acz/index|Applied Crypto and Zero-Knowledge]] team. Ramses expecting starting data is ???

content/waku/monthly-reports/2023-sept.md

@@ -63,12 +63,17 @@ Much of the work this month was fleshing out the available REST APIs of `nwaku`.
 
 
 ## Perceived Changes in Project Risk
+- Richard doing most integration of waku into status-go
+	- C-bindings in Nim is a lot of effort and has large unknowns
+		- this effort doesn't progress on the rest of the roadmap
+- waku network mvp target being tracked well
+- status remaining items
+	- postgres integration (should have been finished)
+	- fleet setup (lack of infra resources, not entirely on them though)
+	- 10k node simulations by DST 
 
-## Future Plans
+## Future Improvement Plans
 
-### Insight
-
-### Project
 
 ## Sources and Useful Links
 Weekly Reports