improve requirements section

This commit is contained in:
munna0908 2025-09-01 16:52:21 +05:30
parent f77b547ae1
commit 3b893905bd
No known key found for this signature in database
GPG Key ID: 2FFCD637E937D3E6

View File

@ -57,66 +57,79 @@ The current implementation has several shortcomings:
## 4. Functional Requirements ## 4. Functional Requirements
**Atomic Block Operations** ### Available Today
- Store, retrieve, and delete operations must be atomic - **Atomic Block Operations**
- Support retrieval via: - Store, retrieve, and delete operations must be atomic.
- Support retrieval via:
- Direct CID - Direct CID
- Tree-based addressing (`treeCid` + index) - Tree-based addressing (`treeCid + index`)
- Unified block address - Unified block address
**Metadata Management** - **Metadata Management**
- Store protocol-level metadata (e.g., storage proofs, quota usage).
- Store block-level metadata (e.g., reference counts, total block count).
- Store protocol-level metadata (e.g., sales data, storage proofs, quota usage) - **Multi-Datastore Support**
- Store block-level metadata (e.g., reference counts, total block count) - Pluggable datastore interface supporting various backends.
- Typed datastore operations for metadata type safety.
**Multi-Datastore Support** - **Lifecycle & Maintenance**
- BlockMaintainer service for removing expired data.
- Configurable maintenance intervals (default: 10 min).
- Batch processing (default: 1000 blocks/cycle).
- Pluggable datastore interface supporting various backends ### Future Requirements
- Typed datastore operations for metadata type safety
- Uniform API across implementations
**Lifecycle & Maintenance** - **Transaction Rollback & Error Recovery**
- Rollback support for failed multi-step operations.
- Consistent state restoration after failures.
- **BlockMaintainer** service for removing expired data - **Dataset-Level Operations**
- Configurable maintenance intervals (default: 10 min) - Handle Dataset level meta data.
- Batch processing (default: 1000 blocks/cycle) - Batch operations for dataset block groups.
- Cooperative scheduling to avoid blocking
- State tracking for large datasets
**Transaction Rollback & Error Recovery** - **Concurrency Control**
- Consistent locking and coordination mechanisms to prevent inconsistencies during crashes or long-running operations.
- Rollback for failed multi-step operations - **Lifecycle & Maintenance**
- Consistent state restoration after failures - Cooperative scheduling to avoid blocking.
- State tracking for large datasets.
--- ---
## 5. Non-Functional Requirements ## 5. Non-Functional Requirements
**Performance** ### Available Today
- Batch metadata updates - **Security**
- Efficient key lookups with configurable prefix lengths - Verify block content integrity upon retrieval.
- Lazy iteration and streaming for large datasets - Enforce quotas to prevent disk exhaustion.
- Support for both fast and slower storage tiers - Safe orphaned data cleanup.
**Security** - **Scalability**
- Configurable storage quotas (default: 20 GiB).
- Pagination for metadata queries.
- Reference countingbased garbage collection.
- Verify block content integrity upon retrieval - **Reliability**
- Enforce quotas to prevent disk exhaustion - Metrics collection (`codex_repostore_*`).
- Safe orphaned data cleanup - Graceful shutdown with resource cleanup.
**Scalability** ### Future Requirements
- Configurable storage quotas (default: 20 GiB) - **Performance**
- Pagination for metadata queries - Batch metadata updates.
- Reference countingbased garbage collection - Efficient key lookups with configurable prefix lengths.
- Support for both fast and slower storage tiers.
- Streaming APIs optimized for extremely large datasets.
**Reliability** - **Security**
- Finer-grained quota enforcement across tenants/namespaces.
- Metrics collection (`codex_repostore_*`) - **Reliability**
- Auto-recovery from inconsistent states - Stronger rollback semantics for multi-node consistency.
- Graceful shutdown with resource cleanup - Auto-recovery from inconsistent states.
--- ---
@ -222,7 +235,7 @@ This store maintains **two separate LRU caches**:
- Acts as the primary cache for block content - Acts as the primary cache for block content
2. **CID/Proof Cache**`LruCache[(Cid, Natural), (Cid, CodexProof)]` 2. **CID/Proof Cache**`LruCache[(Cid, Natural), (Cid, CodexProof)]`
- Maps `(treeCid, index)` to `(blockCid, proof)` - Maps `(treeCid, index)` to `(blockCid, proof)`
- Enables efficient tree-based block access without re-fetching proofs - Supports direct access to block proofs keyed by `treeCid` and index
**Characteristics**: **Characteristics**: