logos-storage-docs-obsidian/10 Notes/libp2p MIX Architecture and API.md

The copy of this document can be found in https://github.com/logos-storage/libp2p-storage-mix-transport/blob/master/docs/mix.md. The content of this document should contain the most recent version.

This document belong to the series of Mix Implementation notes.

This document describes the MIX implementation vendored with this [project] under nimbledeps/pkgs2/libp2p-*/libp2p/protocols/mix. It focuses on how the code is structured, what API surface is intended for callers, and what constraints matter when using MIX as a transport-like wrapper for other libp2p protocols.

Overview

MIX adds an anonymous message routing layer on top of libp2p streams. A sender wraps an application protocol message in a fixed-size Sphinx packet, sends it to a randomly selected mix path, and lets each hop peel one routing layer before forwarding the packet. The final mix hop acts as an exit node and forwards the inner message to the real destination protocol. Replies are optional and are returned through Single Use Reply Blocks (SURBs), so the destination can respond without learning the sender.

The libp2p protocol ID is:

const MixProtocolID* = "/mix/1.0.0"

At a high level:

sender protocol
    |
    | writes to MixEntryConnection
    v
entry MixProtocol
    |
    | Sphinx packet over /mix/1.0.0
    v
mix hop 1 -> mix hop 2 -> ... -> exit mix hop
                                      |
                                      | original codec
                                      v
                                destination protocol
                                      |
                                      | optional response via SURBs
                                      v
sender MixEntryConnection

Main Modules

The public import is libp2p/protocols/mix. It re-exports the main types and helpers from the implementation modules:

• mix_protocol.nim: main MixProtocol implementation, path selection, Sphinx packet handling, SURB creation, forwarding, and reply processing.
• mix_node.nim: private/public mix node identity records.
• pool.nim: MixNodePool, backed by the libp2p PeerStore.
• entry_connection.nim: MixEntryConnection, a Connection facade used by local protocols to write through MIX and optionally read replies.
• exit_layer.nim: destination forwarding and reply dispatch from the exit node.
• reply_connection.nim: MixReplyConnection, a write-only connection facade used by the exit layer to send replies over SURBs.
• serialization.nim, sphinx.nim, fragmentation.nim, mix_message.nim: fixed packet formats, Sphinx processing, padding, and application message encoding.
• multiaddr.nim: compact encoding for next-hop and destination addresses.
• delay_strategy.nim: pluggable per-hop delay strategy.
• spam_protection.nim: optional per-hop proof interface.
• tag_manager.nim: replay detection state for Sphinx tags.

Core Types

MixProtocol

MixProtocol is an LPProtocol mounted on a libp2p Switch. Each running mix node owns one instance.

Important fields and collaborators:

• mixNodeInfo: this node's libp2p identity and Curve25519 MIX keypair.
• switch: the underlying libp2p switch used for dialing peers and destinations.
• nodePool: public information for candidate mix nodes.
• tagManager: replay detection for processed Sphinx packets.
• exitLayer: forwards exit messages to destination protocols and sends replies.
• connCreds: temporary SURB credentials for expected replies.
• destReadBehavior: per-codec callbacks that tell the exit node how to read a destination response.
• spamProtection: optional proof generator/verifier.
• delayStrategy: pluggable encoded/actual delay policy.

Create and mount it with:

let proto = MixProtocol.new(nodeInfo, switch)
await proto.start()
proto.nodePool.add(otherMixNodes)
switch.mount(proto)

MixProtocol.new defaults to NoSamplingDelayStrategy and no spam protection.

MixNodeInfo and MixPubInfo

MixNodeInfo contains private node data and is used by the local mix node:

type MixNodeInfo* = object
  peerId*: PeerId
  multiAddr*: MultiAddress
  mixPubKey*: FieldElement
  mixPrivKey*: FieldElement
  libp2pPubKey*: SkPublicKey
  libp2pPrivKey*: SkPrivateKey

MixPubInfo is the shareable subset stored in MixNodePool:

type MixPubInfo* = object
  peerId*: PeerId
  multiAddr*: MultiAddress
  mixPubKey*: FieldElement
  libp2pPubKey*: SkPublicKey

Convenience helpers:

• MixNodeInfo.generateRandom(port)
• MixNodeInfo.generateRandomMany(count, basePort = 4242)
• initMixNodeInfo(...)
• toMixPubInfo(info)
• includeAllExcept(allNodes, exceptNode)

In local examples, switches are started first so wildcard listen addresses are resolved, then initMixNodeInfo rebuilds each node record with a dialable address from switch.peerInfo.addrs.

MixNodePool

MixNodePool manages public mix node information through the switch peer store. It writes to the peer store's mix public key book, address book, and key book.

Public operations:

let pool = MixNodePool.new(switch.peerStore)
pool.add(info)
pool.add(infos)
discard pool.remove(peerId)
let infoOpt = pool.get(peerId)
let ids = pool.peerIds()
let n = pool.len

Supported addresses are IPv4 TCP and IPv4 QUIC-v1, including supported circuit-relay addresses over those base transports. IPv6, DNS, WebSocket, and related address forms are not supported by the compact address encoder in this implementation.

The implementation requires Secp256k1 libp2p peer IDs when serializing hop addresses.

MixDestination

MixDestination identifies the final target of a mixed message.

Normal usage forwards from the exit node to a destination address:

let destination = MixDestination.init(destPeerId, destAddr)

This is equivalent to MixDestination.forwardToAddr(destPeerId, destAddr).

There is also compile-time experimental support for exit == destination behind -d:libp2p_mix_experimental_exit_is_dest. Without that define, the destination must be a separate forwarded address.

MixParameters

MixParameters controls entry connection behavior:

type MixParameters* = object
  expectReply*: Opt[bool]
  numSurbs*: Opt[uint8]

If expectReply is true, toConnection creates an incoming queue and the sender embeds SURBs in the outgoing payload. If numSurbs is omitted while a reply is expected, MixEntryConnection uses a default of 4 SURBs. If no reply is expected, the number of SURBs is forced to 0.

DestReadBehavior

When a request expects a reply, the exit node needs to know how to read a response from the destination protocol connection. Register that per application codec on every mix node that may be selected as an exit:

proto.registerDestReadBehavior(PingCodec, readExactly(32))

The public helpers are:

readExactly(nBytes: int): DestReadBehavior
readLp(maxSize: int): DestReadBehavior

Use readExactly for fixed-size response protocols. Use readLp when the destination protocol returns length-prefixed messages using libp2p's varint LP framing. For readLp, the exit layer restores the length prefix before sending the reply through MIX, allowing the original caller to call readLp() on the returned MixEntryConnection.

Sending Through MIX

What "writes to MixEntryConnection" means

In libp2p Nim, application protocols usually talk to a generic Connection. For example, the Ping client does not know whether the connection is a direct TCP stream, QUIC stream, relay stream, or MIX-backed connection:

proc ping*(p: Ping, conn: Connection): Future[Duration] =
  await conn.write(randomBytes)
  await conn.readExactly(addr resultBuf[0], 32)

MixEntryConnection is a custom Connection implementation. It has the same write and readOnce/readExactly behavior that application protocols expect, but its write method does not write directly to the destination. Instead, it calls into MixProtocol.anonymizeLocalProtocolSend, which wraps that write in a Sphinx packet and sends it through the mixnet.

So "writes to MixEntryConnection" normally happens indirectly:

1. Application code calls an existing protocol helper, such as pingProto.ping(conn).
2. That helper calls conn.write(...).
3. Because conn is actually a MixEntryConnection, the write becomes a MIX send.
4. If the helper later calls conn.readExactly(...), MixEntryConnection waits for a SURB reply and serves the reply bytes from its local incoming queue.

This is why mix_ping_tcp.nim can reuse the normal Ping client logic:

let pingProto = Ping.new()
destNode.mount(pingProto)

let conn = mixProtos[senderIndex]
  .toConnection(
    MixDestination.init(destNode.peerInfo.peerId, destNode.peerInfo.addrs[0]),
    PingCodec,
    MixParameters(expectReply: Opt.some(true), numSurbs: Opt.some(byte(1))),
  )
  .expect("could not build connection")

let response = await pingProto.ping(conn)

There are two different uses of the same Ping instance here:

• Mounted on destNode, Ping acts as the destination server handler. It reads 32 bytes and writes the same 32 bytes back.
• Called as pingProto.ping(conn), Ping acts as client-side helper code. It only needs a Connection, so it can use the MIX-backed connection returned by toConnection.

Preferred API: toConnection

The ergonomic API is toConnection, which returns a libp2p Connection implementation. Use this when you want an existing libp2p protocol helper to run over MIX:

let conn = mixProto
  .toConnection(
    MixDestination.init(destNode.peerInfo.peerId, destNode.peerInfo.addrs[0]),
    PingCodec,
    MixParameters(expectReply: Opt.some(true), numSurbs: Opt.some(byte(1))),
  )
  .expect("could not build connection")

let response = await pingProto.ping(conn)
await conn.close()

MixEntryConnection.write sends each write as one independent MIX message by calling MixProtocol.anonymizeLocalProtocolSend. readOnce waits for one reply if the connection was created with expectReply = true.

If expectReply = true and no destination read behavior has been registered on the sender's local MixProtocol, toConnection returns an error. The actual response read happens on the selected exit node, so practical deployments should register the same behavior on all candidate exit nodes. If no reply is expected, missing read behavior only logs a warning.

Lower-level API: anonymizeLocalProtocolSend

MixProtocol.anonymizeLocalProtocolSend is also exported from mix_protocol.nim itself. It sends one byte message through MIX without first constructing a Connection facade:

let incoming = newAsyncQueue[seq[byte]]()
let sendRes = await mixProto.anonymizeLocalProtocolSend(
  incoming,
  msg,
  codec,
  destination,
  numSurbs,
)

This is useful if you are building a custom adapter and want direct control over the incoming reply queue. It is less convenient for normal libp2p protocols because those protocols usually expect to read and write through a Connection.

In this implementation, the practical outbound interaction models are:

• Use toConnection and pass the returned connection to an existing protocol client helper. This is the normal path.
• Use toConnection directly and call conn.write, conn.readExactly, conn.readLp, etc. yourself.
• Call anonymizeLocalProtocolSend from mix_protocol.nim for a lower-level one-message send.

Inbound mixnet traffic is not handled by application code directly. Mix nodes receive /mix/1.0.0 streams through the mounted MixProtocol; the exit layer then dials the destination using the original application codec.

Internal Message Flow

Entry

anonymizeLocalProtocolSend performs the sender-side work:

1. Check that the node pool has at least PathLength available public mix nodes.
2. Randomly select a path from nodePool, excluding the final destination if it is also present in the pool.
3. Use the last selected mix node as the exit node.
4. Encode each hop as compact bytes containing IPv4 address, transport, port, optional relay peer ID, and peer ID.
5. Generate per-hop delay values using delayStrategy.generateForEntry.
6. Optionally create SURBs for replies and prepend them to the application message.
7. Encode the application message as MixMessage(message, codec).
8. Pad it into one fixed-size MessageChunk.
9. Wrap it into a Sphinx packet using the selected public keys, delays, hops, and final destination.
10. Optionally append a spam protection proof.
11. Send the packet to the first hop over /mix/1.0.0.

Intermediate Hop

Each mix node handles inbound /mix/1.0.0 streams in handleMixNodeConnection. For every length-prefixed packet read from the stream, it spawns handleMixMessages.

For intermediate packets:

1. Extract the optional spam proof.
2. Deserialize the Sphinx packet.
3. Check replay tags with the node's MIX private key and TagManager.
4. Verify spam protection after the replay check.
5. Process the Sphinx packet to reveal the next hop and transformed packet.
6. Compute the actual delay using delayStrategy.generateForIntermediate.
7. Generate a fresh per-hop spam proof for the outgoing packet if enabled.
8. Sleep for the computed delay.
9. Forward to the next hop over /mix/1.0.0.

Connections to next hops are cached by peer ID in connPool. If a cached stream is closed or reset, the implementation dials a fresh stream and retries the write.

Exit

When Sphinx processing returns Exit, the node:

1. Deserializes and unpads the MessageChunk.
2. Deserializes the MixMessage to recover codec and message.
3. Extracts any embedded SURBs.
4. Passes the request to ExitLayer.onMessage.

For normal forwarded destinations, ExitLayer dials the destination peer using the recovered codec, writes the raw inner message, and optionally reads a response using the registered DestReadBehavior.

If SURBs were provided, the exit layer sends the response back through every SURB using a MixReplyConnection.

Reply

Replies do not contain an application codec; they are associated with a SURB identifier generated by the original sender.

The original sender stores each generated SURB identifier in connCreds with the SURB key, secret, identifier group, and incoming queue. When a Reply packet arrives:

1. The sender looks up the SURB identifier.
2. It processes the reply using the stored SURB key and secret.
3. It deletes all credentials in the same identifier group.
4. It deserializes, unpads, and decodes the reply message.
5. It places the reply bytes on the entry connection's incoming queue.

Because the credentials are single-use, one response consumes the associated SURB group.

Packet and Payload Limits

Important constants from serialization.nim and fragmentation.nim:

PacketSize = 4608
MessageSize = PacketSize - HeaderSize - k
DataSize = MessageSize - PaddingLengthSize - SeqNoSize
PathLength = 3

Current code builds a single padded MessageChunk for each write; it does not automatically split large application messages in anonymizeLocalProtocolSend. MixEntryConnection.write rejects messages larger than DataSize, and buildMessage rejects serialized MixMessage values larger than DataSize.

Because MIX adds codec and SURB overhead inside the fixed payload, callers should use:

let maxPayload = getMaxMessageSizeForCodec(codec, numberOfSurbs)

This returns the maximum application message size available for the given codec and number of embedded SURBs.

Delay Strategies

Delay behavior is pluggable through DelayStrategy.

Built-in strategies:

• NoSamplingDelayStrategy: default. The entry node encodes a random delay in the range 0-2 ms, and intermediate nodes use that value directly.
• ExponentialDelayStrategy: the entry node encodes a mean delay, and intermediate nodes sample from an exponential distribution with that mean. The default mean is DefaultMeanDelayMs = 100.

Example:

let rng = newRng()
let delay = ExponentialDelayStrategy.new(meanDelayMs = 100, rng = rng)
let proto = MixProtocol.new(nodeInfo, switch, rng = rng, delayStrategy = Opt.some(delay))

When spam protection is enabled, proof generation runs in parallel with the intermediate delay. If proof generation takes longer than the configured delay, the effective delay becomes the proof generation time.

Spam Protection

Spam protection is optional and is represented by an abstract SpamProtection object:

type SpamProtection* = ref object of RootObj
  proofSize*: int

method generateProof(self: SpamProtection, bindingData: seq[byte]): Result[seq[byte], string]
method verifyProof(self: SpamProtection, encodedProofData: seq[byte], bindingData: seq[byte]): Result[bool, string]

If provided to MixProtocol.new, the wire packet becomes:

[Sphinx packet: 4608 bytes][proof: proofSize bytes]

Each hop extracts and verifies the incoming proof, processes the Sphinx packet, then generates a fresh proof for the next hop. If no spam protection instance is provided, packets are just the fixed-size Sphinx packet.

Address Constraints

Hop and destination addresses are encoded into a fixed AddrSize field. The implementation currently supports:

• /ip4/.../tcp/...
• /ip4/.../udp/.../quic-v1
• circuit-relay variants over those base transports

Current limitations:

• No IPv6 support.
• No DNS or DNS4 support.
• No WebSocket, WebSocket Secure, or SNI support.
• Peer IDs must serialize to the expected Secp256k1 multihash length.

Invalid or unsupported mix node addresses are removed from consideration during path construction.

Minimal Setup Pattern

The local TCP and QUIC examples follow this shape:

let mixNodeInfos = MixNodeInfo.generateRandomMany(NumMixNodes)

for nodeInfo in mixNodeInfos:
  let switch = createSwitch(nodeInfo.multiAddr, Opt.some(nodeInfo.libp2pPrivKey))
  await switch.start()
  switches.add(switch)

let resolvedInfos = collect:
  for i, nodeInfo in mixNodeInfos:
    initMixNodeInfo(
      nodeInfo.peerId,
      switches[i].peerInfo.addrs[0],
      nodeInfo.mixPubKey,
      nodeInfo.mixPrivKey,
      nodeInfo.libp2pPubKey,
      nodeInfo.libp2pPrivKey,
    )

for i, nodeInfo in resolvedInfos:
  let proto = MixProtocol.new(nodeInfo, switches[i])
  await proto.start()
  proto.nodePool.add(resolvedInfos.includeAllExcept(nodeInfo))
  proto.registerDestReadBehavior(PingCodec, readExactly(32))
  switches[i].mount(proto)

For QUIC, build switches with TransportType.QUIC and use /ip4/0.0.0.0/udp/0/quic-v1 listen addresses.

Operational Notes

• Every mix participant must mount MixProtocol on its switch.
• The sender's node pool must contain enough usable mix nodes for PathLength.
• Destination protocols do not need to know about MIX when using normal exit-forwarding mode; the exit node dials them by their original codec.
• Replies require a registered DestReadBehavior for the application codec. Register it on all candidate exit nodes; the sender also checks for a local registration before creating a reply-capable connection.
• MIX messages are independent and stateless at the routing layer, apart from replay protection tags, cached outbound streams, and temporary SURB credentials.
• MixProtocol.stop marks the protocol stopped and stops the tag manager, but it does not perform broad connection-pool or credential cleanup.
• Benchmark-only metadata framing is compiled in with -d:enable_mix_benchmarks.