Add project files.
This commit is contained in:
micro
+264
@@ -0,0 +1,264 @@
|
||||
# Groups in Wingnal — status + full Groups v2 build plan
|
||||
|
||||
Last updated 2026-06-16. This is the **detailed, executable plan** for taking Wingnal from "group
|
||||
crypto primitive only" to real Signal group chats (Groups v2 / "GV2"). It is written to be resumed
|
||||
across many sessions; each phase has concrete files, pinned facts, and a **test gate** that must be
|
||||
green before the next phase starts. Reference source is libsignal `rust/` tag **v0.96.1** (same tag the
|
||||
Sender Key + SPQR ports were pinned to).
|
||||
|
||||
---
|
||||
|
||||
## Reality check (decided 2026-06-16, before writing this plan)
|
||||
|
||||
Two facts were verified against the installed toolchain because they decide the whole effort:
|
||||
|
||||
1. **BouncyCastle 2.5.1 has NO Ristretto255.** Its XML surface exposes only `Math.EC.Rfc8032.Ed25519`
|
||||
and `Math.EC.Rfc7748.X25519Field` (the latter is already wrapped in
|
||||
`Wingnal.Protocol/Curve/Ed25519Ct.cs`). There is no public Ristretto group element, no scalar-mod-ℓ
|
||||
type, no multiscalar mul. **Therefore zkgroup must be hand-ported** (Ristretto255 group + `Scalar`
|
||||
field + `poksho` Schnorr proofs), reusing BC's `X25519Field` for the base field GF(2²⁵⁵−19) and the
|
||||
`ScReduce`/`ScMulAdd` already in `Ed25519Ct` for scalar reduction. This is a Kyber/SPQR-scale port and
|
||||
is **the long pole** of the whole feature.
|
||||
2. **BouncyCastle 2.5.1 HAS AES-256-GCM-SIV** (`Org.BouncyCastle.Crypto.Modes.GcmSivBlockCipher`). zkgroup
|
||||
encrypts the group's title/avatar/description blobs with AES-256-GCM-SIV, so that part binds to BC
|
||||
instead of being hand-ported. Validate our wrapper against an RFC 8452 test vector before use.
|
||||
|
||||
**Hard rule for the whole feature:** every crypto phase is validated against libsignal's own test vectors
|
||||
**before** any network call. zkgroup has deterministic vectors (`rust/zkgroup/src/lib.rs` `tests` +
|
||||
`rust/zkgroup/tests/`) that fix the server's random params, so client/server flows reproduce byte-exactly
|
||||
offline. Wrong ZK math fails silently (server returns 403 with no detail), so KATs are non-negotiable.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — DONE: the crypto building blocks already in the tree
|
||||
|
||||
These are complete and tested; GV2 builds directly on them. Do not re-port them.
|
||||
|
||||
- **Sender Key messaging primitive** (`Wingnal.Protocol/Groups/`, byte-exact vs libsignal v0.96.1). One
|
||||
ciphertext that every member decrypts: `SenderKeyMessage`/`SenderKeyDistributionMessage`,
|
||||
`SenderChainKey`/`SenderMessageKey` (HKDF "WhisperGroup" 48B → iv‖cipherKey), `SenderKeyState`
|
||||
(chainId, signing key, bounded skipped-key FIFO `MAX_MESSAGE_KEYS=2000`), `SenderKeyRecord`
|
||||
(`MAX_SENDER_KEY_STATES=5`), `GroupSessionBuilder` (Create→SKDM / Process), `GroupSessionCipher`
|
||||
(Encrypt/Decrypt, AES-256-CBC, `MAX_FORWARD_JUMPS=25000`), `InMemorySenderKeyStore`. Pinned facts:
|
||||
version byte `0x33`; `distribution_uuid` serialized RFC-4122/big-endian (`SenderKeyWire.DistributionBytes`);
|
||||
XEdDSA sig over `version‖protobuf`. Tests: `Wingnal.Tests/Groups/GroupCipherTests.cs`.
|
||||
- **Sealed sender (UD) for 1:1** — `SealedSenderDecryptor` (receive v1 + `EncryptWithCertificate`),
|
||||
`SenderCertificateValidator` (cert chain vs production trust roots), `MessageSender.TrySealedAsync`
|
||||
(opportunistic sealed send w/ auth fallback), `ProfileKeyStore`, `UnidentifiedAccess.DeriveAccessKey`.
|
||||
GV2 fan-out reuses `EncryptWithCertificate` per member device. (Sealed Sender **v2** multi-recipient is
|
||||
NOT done — see Phase G; v1-per-device is sufficient to ship.)
|
||||
|
||||
**What's still missing for real groups:** durable Sender Key storage, the entire zkgroup credential
|
||||
system, the group storage-service client, membership/change application, and the group DataMessage
|
||||
wiring + UI. That is Phases A–H below.
|
||||
|
||||
---
|
||||
|
||||
## Phase A — Sender Key persistence (SQLite store + state serialization)
|
||||
|
||||
**Goal:** the Sender Key primitive survives app restart, mirroring how `SqliteSignalProtocolStore`
|
||||
persists the Double Ratchet today. No new crypto.
|
||||
|
||||
- `Wingnal.Protocol/Groups/SenderKeyRecordSerialization.cs` — custom binary (or reuse the
|
||||
existing `SessionRecord` serialization style) for `SenderKeyRecord` → bytes: per state the chainId,
|
||||
iteration, chainKey, signing key pair, and the skipped-key FIFO cache. (libsignal uses
|
||||
`storage.proto SenderKeyRecordStructure`; we already chose custom binary for sessions, stay consistent.)
|
||||
- `Wingnal.Service/Account/SqliteSenderKeyStore.cs` — `ISenderKeyStore` over SQLite, blobs
|
||||
DPAPI-wrapped via `LocalCipher` (same pattern as `SqliteSignalProtocolStore`). Key = (senderAddress,
|
||||
distributionId).
|
||||
- **Test gate:** `SenderKeyPersistenceTests` — build a 3-member synthetic group, exchange a few messages,
|
||||
serialize→restore the store, confirm decryption (incl. an out-of-order/skipped message) continues
|
||||
across the restart. Offline. Must keep the existing 121 green.
|
||||
|
||||
Independently useful and zero-risk; do this first to lock the storage shape.
|
||||
|
||||
---
|
||||
|
||||
## Phase B — zkgroup math foundation (`Wingnal.Protocol/ZkGroup/Curve/`)
|
||||
|
||||
**Goal:** a constant-time-ish Ristretto255 group + scalar field, validated against curve25519-dalek
|
||||
vectors. This is the bottom of the long pole.
|
||||
|
||||
- `Scalar25519.cs` — integers mod ℓ = 2²⁵² + 27742317777372353535851937790883648493. Ops: add, sub, mul,
|
||||
negate, invert (Fermat or `ScMulAdd`-based), `FromBytesModOrder` (32B LE), `FromBytesModOrderWide`
|
||||
(64B LE → reduce; reuse `Ed25519Ct.ScReduce`), `ToBytes`. Start on `BigInteger` for clarity; the
|
||||
reduction helpers in `Ed25519Ct` (ref10 `sc_reduce`/`sc_muladd`) are the constant-time upgrade path.
|
||||
- `RistrettoPoint.cs` — edwards25519 extended (X:Y:Z:T) coordinates over BC `X25519Field`, with the
|
||||
**Ristretto** layer on top: canonical 32-byte encode/decode (the Ristretto equality + sign rules),
|
||||
`FromUniformBytes` (Elligator2, 64B → point, for hash-to-group), add, negate, scalar-mul
|
||||
(`mul(Scalar)`), and the fixed basepoint. Reuse the field add/sub/mul/sqr/invert/cmov already proven in
|
||||
`Ed25519Ct` (extract them into a shared `Field25519` helper rather than duplicating).
|
||||
- `RistrettoGenerators.cs` — `RISTRETTO_BASEPOINT` + deterministic generator derivation via the SHO
|
||||
(Phase C) — but the basepoint constant can land here.
|
||||
- **Test gate:** `RistrettoVectorTests` — port the curve25519-dalek canonical vectors: the 16
|
||||
multiples-of-basepoint encodings, the Elligator/`from_uniform_bytes` test, decode-rejects-non-canonical,
|
||||
and scalar arithmetic identities. Byte-exact. (This is the gate that proves the group is correct before
|
||||
any credential logic depends on it — treat like the Ed25519Ct cross-check.)
|
||||
|
||||
RISK: Ristretto encode/decode sign conventions are subtle (the `IS_NEGATIVE`/`abs` rules). Budget real
|
||||
time here and lean on the dalek vectors. ~2 dense files + heavy testing.
|
||||
|
||||
---
|
||||
|
||||
## Phase C — poksho (SHO + Schnorr proof system) (`Wingnal.Protocol/ZkGroup/Poksho/`)
|
||||
|
||||
**Goal:** the Fiat-Shamir transcript + generic Σ-protocol that every zkgroup credential is expressed in.
|
||||
Must be byte-exact or proofs verify nowhere.
|
||||
|
||||
- `Sho.cs` — the "stateful hash object": `ShoHmacSha256` (and/or `ShoSha256`) with libsignal's exact
|
||||
absorb/ratchet/squeeze construction. Used to (a) derive `SystemParams` generators, (b) hash proof
|
||||
transcripts, (c) `get_point`/`get_scalar` (hash-to-group / hash-to-scalar). Byte-exactness is verified
|
||||
against poksho's own test vectors (`rust/poksho/tests`).
|
||||
- `Statement.cs` + `Proof.cs` — the generic linear-relation proof: a statement is a set of equations
|
||||
"point = Σ scalarᵢ · generatorᵢ"; the prover knows the scalars, the verifier checks. This is the engine
|
||||
behind AuthCredential/ProfileKeyCredential presentations.
|
||||
- **Test gate:** `PokshoVectorTests` — SHO output vectors + a known Schnorr proof prove/verify round-trip
|
||||
+ a tampered-proof rejection, matching poksho vectors.
|
||||
|
||||
RISK: medium. The SHO byte layout and the proof challenge computation must match exactly. The proof
|
||||
framework itself is mechanical once SHO is right.
|
||||
|
||||
---
|
||||
|
||||
## Phase D — zkgroup credentials, ciphertexts, params (`Wingnal.Protocol/ZkGroup/`)
|
||||
|
||||
**Goal:** everything the group server speaks, expressed in zkgroup. Pure crypto; validated offline.
|
||||
|
||||
- `GroupSecretParams.cs` / `GroupPublicParams.cs` — derive `GroupSecretParams` from the 32-byte
|
||||
**group master key** via SHO; derive `GroupPublicParams` + the 32-byte **group identifier** from it.
|
||||
(The Sender Key **distribution id** for a group derives from the group id + sender, per libsignal
|
||||
`sender_key_name` — wire this into Phase G.)
|
||||
- `SystemParams.cs` — the fixed credential-system generators, derived deterministically via SHO. Must
|
||||
reproduce libsignal's constants exactly (assert against the hard-coded `SystemParams::get_hardcoded()`
|
||||
bytes in zkgroup).
|
||||
- `UuidCiphertext.cs` / `ProfileKeyCiphertext.cs` — encrypt a `ServiceId`/profile key under
|
||||
`GroupSecretParams` (deterministic, so the server can match without learning the ACI). Reuse the
|
||||
`ServiceId` binary form from `Wingnal.Service/ServiceIds.cs`.
|
||||
- `AuthCredentialWithPni.cs` + `AuthCredentialWithPniPresentation.cs` — receive+verify the server's
|
||||
`AuthCredentialWithPniResponse`, store the credential, and build the **presentation** (the ZK proof) put
|
||||
in the `Authorization` header for group-server calls.
|
||||
- `ProfileKeyCredential` / `ExpiringProfileKeyCredential` + presentation — needed to add members by proving
|
||||
knowledge of their profile key (used in `GroupChange` for adds).
|
||||
- `GroupAttributeBlob` encryption — `ClientZkGroupCipher.EncryptBlob/DecryptBlob` using **AES-256-GCM-SIV**
|
||||
(BC `GcmSivBlockCipher`) under a key derived from `GroupSecretParams`. Wraps title/avatar/description.
|
||||
- **Test gate:** `ZkGroupVectorTests` — port libsignal's zkgroup integration vectors: fixed
|
||||
`ServerSecretParams` seed → issue an AuthCredentialWithPni → client verifies → builds presentation →
|
||||
server-side verify accepts; UuidCiphertext/ProfileKeyCiphertext round-trip; blob enc vs vector. All
|
||||
byte-exact, offline. **This is the gate that proves we can talk to the real group server before we try.**
|
||||
|
||||
This is the largest single phase. Do not start Phase E until these vectors pass.
|
||||
|
||||
**STATUS (2026-06-16):** split in two after discovering the dependency surface.
|
||||
- **D1 DONE** — `Wingnal.Protocol/ZkGroup/GroupSecretParams.cs`: master-key → group_id + blob_key
|
||||
derivation and AES-256-GCM-SIV blob encrypt/decrypt-with-padding (BC `GcmSivBlockCipher`). Gated by
|
||||
zkgroup's `test_encrypt_with_padding` vectors (`GroupSecretParamsTests`). Enough to derive a group id
|
||||
(route inbound group messages — Phase G1) and decrypt group title/avatar blobs.
|
||||
- **D2 REMAINING (large)** — the member-hiding credential/ciphertext layer pulls in TWO more ports that
|
||||
weren't visible from the top-level plan: (1) the **zkcredential** crate (the attribute/credential ZK
|
||||
system that `uid_encryption` / `profile_key_encryption` / AuthCredential all route through), and
|
||||
(2) **Lizard** encoding — `RistrettoPoint::lizard_encode::<Sha256>(16 bytes)` in the
|
||||
*curve25519-dalek-signal fork* (NOT RFC 9496, NOT upstream dalek), plus `lizard_decode` for decryption,
|
||||
plus `get_point_single_elligator` (single-Elligator map; only the double `FromUniformBytes` is ported).
|
||||
Then `UuidCiphertext` / `ProfileKeyCiphertext` / `AuthCredentialWithPni` + presentations. Fetch
|
||||
`rust/zkcredential/src` + the lizard impl from the fork; validate against zkgroup integration vectors.
|
||||
|
||||
---
|
||||
|
||||
## Phase E — group storage-service client (`Wingnal.Service/Groups/`)
|
||||
|
||||
**Goal:** fetch and decrypt the real group state for a group the user is already in.
|
||||
|
||||
- Pinning: `storage.signal.org` is a **separate host** from chat — confirm whether it chains to the same
|
||||
bundled Signal CA (`SignalTrust`) or needs its own pin; add to `SignalServiceConfig`.
|
||||
- `Protos/Group.proto` (vendored) — `Group`, `GroupChange`, `GroupChanges`, `Member`, `PendingMember`,
|
||||
`RequestingMember`, `AccessControl`. All member identity fields are zkgroup ciphertexts.
|
||||
- `GroupsApiClient.cs` — `GET /v1/groups` (auth = `AuthCredentialWithPniPresentation` in the header),
|
||||
`GET /v1/groups/logs/{fromRevision}` (incremental `GroupChanges`), `PATCH /v1/groups` (apply a change).
|
||||
- `GroupStateCodec.cs` — decrypt a fetched `Group` into a plaintext local model (member ACIs via
|
||||
`UuidCiphertext` decrypt, title via blob decrypt, roles, revision).
|
||||
- **Test gate:** `GroupFetchLiveTests [Category=Live]` — using a real linked account that's in a group,
|
||||
fetch + decrypt the group and assert the member list/title match what the phone shows. Offline unit test
|
||||
for `GroupStateCodec` against a captured (decrypted-locally) `Group` blob.
|
||||
|
||||
RISK: depends entirely on Phase D being byte-exact (a wrong presentation → opaque 403).
|
||||
|
||||
---
|
||||
|
||||
## Phase F — membership model + GroupChange application
|
||||
|
||||
**Goal:** maintain local group state and apply server changes safely.
|
||||
|
||||
- `Wingnal.Service/Groups/GroupStore.cs` (SQLite, encrypted via `LocalCipher`) — per-group: masterKey,
|
||||
revision, decrypted roster (ACI, role), title/avatar, access control.
|
||||
- `GroupChangeApplier.cs` — apply add/remove/promote/invite/requesting-member diffs, **verify the server's
|
||||
signature on each `GroupChange`**, reconcile `GET /v1/groups/logs` incrementally to the latest revision.
|
||||
- **Test gate:** `GroupChangeTests` — apply a sequence of captured changes to a base state, assert the
|
||||
resulting roster/revision; reject a change with a bad server signature.
|
||||
|
||||
---
|
||||
|
||||
## Phase G — group DataMessage wiring (the part the user sees working)
|
||||
|
||||
**Goal:** send and receive actual group messages. Two milestones — receive first (cheaper), then send.
|
||||
|
||||
- **G1 (receive-only, can land right after Phase A — no zkgroup needed to *decrypt*):** detect
|
||||
`DataMessage.GroupV2 { masterKey, revision }` on inbound (sealed) messages; derive the group id (needs
|
||||
the Phase B/D `GroupSecretParams` for the id, but NOT the full credential system); process any inbound
|
||||
`SenderKeyDistributionMessage` (already on `Content`) via `GroupSessionBuilder.Process`; decrypt the
|
||||
`SenderKeyMessage` with `GroupSessionCipher`; route into a group thread keyed by group id. Roster/names
|
||||
show as raw ACIs until Phase E fills them in. **This gives early visible value** — group messages start
|
||||
appearing — before the storage-service work is finished.
|
||||
- **G2 (send):** attach `GroupContextV2 { masterKey, revision }` to the `DataMessage`; encrypt the body
|
||||
once with `GroupSessionCipher`; for each member device (roster from Phase E/F), wrap the
|
||||
`SenderKeyMessage` as **sealed sender** via the existing `EncryptWithCertificate` and fan out; first
|
||||
distribute our `SenderKeyDistributionMessage` (also sealed 1:1) to members who don't have our current
|
||||
sender key. Handle 409/410 device-set changes per member (reuse the Sesame retry logic in
|
||||
`MessageSender`). Optionally adopt **Sealed Sender v2** (multi-recipient) later to cut fan-out bandwidth.
|
||||
- **Test gate:** `GroupMessagingTests` — offline 3-member loopback (build group state in-proc, A sends,
|
||||
B and C both decrypt, incl. a late-joiner who needs the SKDM); a live send to a real group gated
|
||||
`[Category=Live]`.
|
||||
|
||||
---
|
||||
|
||||
## Phase H — group UI
|
||||
|
||||
**Goal:** groups are first-class in the app.
|
||||
|
||||
- Group conversations in the conversation list (group avatar/title from decrypted state), a member roster
|
||||
view, "new group"/add-member/leave flows (each a `PATCH /v1/groups` `GroupChange`), group typing/receipt
|
||||
fan-out (reuse Phase 0 + the receipts work), and the "X added/removed Y" change banners in-thread.
|
||||
- Reuses the existing `ChatPage` two-pane shell; a group thread is just a conversation keyed by group id.
|
||||
|
||||
---
|
||||
|
||||
## Dependency graph (what unblocks what)
|
||||
|
||||
```
|
||||
Phase 0 (done) ─┬─> A (persistence) ─────────────────────────────> G1 (receive groups) [early value]
|
||||
│
|
||||
└─> B (Ristretto+Scalar) ─> C (poksho) ─> D (credentials/ciphertexts)
|
||||
│
|
||||
└─> E (storage API) ─> F (membership) ─> G2 (send) ─> H (UI)
|
||||
```
|
||||
|
||||
So the cheapest path to *visible* groups is **A → (B + D-just-enough-for-group-id) → G1**. Full
|
||||
send/management requires the whole B→C→D→E→F→G2 spine. zkgroup (B+C+D) is ~60% of the total effort and
|
||||
must be vector-perfect before E.
|
||||
|
||||
## Risk register
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|------|----------|-----------|
|
||||
| Ristretto255 hand-port correctness | High | dalek canonical vectors (Phase B gate); extract proven field ops from `Ed25519Ct` |
|
||||
| poksho SHO byte-exactness | High | poksho vectors (Phase C gate) — proofs verify nowhere if off |
|
||||
| zkgroup presentation rejected by server (opaque 403) | High | full offline issue→present→verify vector (Phase D gate) before any network |
|
||||
| Scalar arithmetic not constant-time (BigInteger) | Medium | acceptable for v1 (client-side, ephemeral randomness); harden with ref10 `sc_*` later |
|
||||
| storage.signal.org cert pin differs from chat | Low | verify host CA during Phase E; reuse `SignalTrust` if same |
|
||||
| Sealed Sender v1 fan-out bandwidth on large groups | Low | ship v1-per-device; add SSv2 multi-recipient later |
|
||||
|
||||
## How to resume
|
||||
|
||||
Start at the lowest unfinished phase; do not skip a test gate. Keep the two baselines green every phase:
|
||||
`dotnet test Wingnal.Tests/Wingnal.Tests.csproj --filter "Category!=Live&Category!=Kat"` and
|
||||
`dotnet build Wingnal/Wingnal.csproj -p:Platform=x64`. Pin all ports to libsignal v0.96.1. Record each
|
||||
phase's completion + gotchas in `memory/project_build_plan.md` and `SHORTCUTS.md`, same as prior steps.
|
||||
@@ -0,0 +1,143 @@
|
||||
# SPQR port plan (Sparse Post-Quantum Ratchet)
|
||||
|
||||
Porting Signal's **SparsePostQuantumRatchet** to pure C# so Wingnal can decrypt/encrypt modern
|
||||
Signal messages. Required because new linked devices must declare the `spqr` capability, and the peer
|
||||
then mixes SPQR output into every message's key schedule — a classic Double Ratchet gets "bad MAC".
|
||||
|
||||
- **Source:** `github.com/signalapp/SparsePostQuantumRatchet`, **tag v1.5.1** (the version libsignal
|
||||
pins in its root `Cargo.toml`). Reference checked out under `_spqrref/` (scratch; re-fetch from the
|
||||
tag if missing). Rust is formally verified (hax_lib/F* annotations) — ignore those attributes.
|
||||
- **libsignal integration:** `rust/protocol/src/ratchet.rs` calls `spqr::initial_state(Params{
|
||||
direction, version:V1, min_version:V1, auth_key, chain_params})` at session init (A2B for initiator,
|
||||
B2A for recipient), and per message the SPQR `SecretOutput` (`None` / `Send(secret)` / `Recv(secret)`)
|
||||
is mixed into the sending/receiving chain before deriving message keys. `spqr_chain_params(self_connection)`
|
||||
builds ChainParams. The `pq_ratchet` bytes ride in `SignalMessage.pq_ratchet` (field 5) and prekey msgs.
|
||||
- **Target namespace:** `Wingnal.Protocol.Spqr`.
|
||||
|
||||
## Module dependency order (bottom-up) and status
|
||||
|
||||
1. **Gf16** — GF(2^16), poly 0x1100b. ✅ DONE (`Spqr/Gf16.cs`, Gf16Tests: field axioms + all-inverses).
|
||||
2. **encoding/polynomial.rs** — systematic fountain code over GF16. ✅ DONE (`Spqr/Polynomial.cs`:
|
||||
Poly(interpolate/evaluate), Encoder.ChunkAt, Decoder; PolynomialTests round-trips systematic +
|
||||
erasure at 1184/1088B). KEY: msg→16 polys round-robin by 2-byte symbol; chunk idx = all 16 polys at
|
||||
x=idx (32B); first ⌈M/16⌉ chunks are the message (systematic), rest are parity. NOTE: state
|
||||
serialization (into_pb/from_pb PolynomialEncoder) deferred to the proto layer.
|
||||
3. **encoding/round_robin.rs** — TEST-ONLY stub (#![cfg(test)]), NOT ported. encoding.rs API
|
||||
(Chunk{index:u16,data:[u8;32]}, Encoder/Decoder traits) folded into Polynomial.cs. ✅ N/A.
|
||||
4. **incremental_mlkem768.rs** — incremental/chunked ML-KEM-768 over libcrux's
|
||||
`mlkem768::incremental` API. PENDING — HARDEST/LINCHPIN. Splits keygen into pk1=hdr(64B) +
|
||||
pk2=ek(1152B), and encaps into encaps1(hdr)->ct1(960B)+state(2080B)+ss(32) and encaps2(ek,state)->
|
||||
ct2(128B); decaps(dk(2400B),ct1,ct2)->ss. Must be BYTE-EXACT with libcrux (incl. issue-1275
|
||||
endianness quirk in serialized state). Likely needs porting FIPS-203 ML-KEM-768 K-PKE from
|
||||
cryspen/libcrux. BC has ml_kem_768 but NOT the incremental split. RISK: feasibility of byte-exact match.
|
||||
**ANALYSIS (key for porting):** the "incremental" API is just standard FIPS-203 ML-KEM-768 with the
|
||||
ek and ciphertext SPLIT so they can be chunked:
|
||||
- keygen: pk1/header(64B) = rho(32) || H(ek)(32); pk2/ek(1152B) = ByteEncode12(t_hat). dk(2400B) =
|
||||
standard ML-KEM-768 dk = dkPke(1152)||ek(1184)||H(ek)(32)||z(32).
|
||||
- encaps1(hdr): m random; (K,r)=G(m||H(ek)) [H(ek) from hdr]; gen A from rho; sample r_hat,e1,e2;
|
||||
u = Compress_du=10(A^T r + e1) = ct1(960B = 3*320); ss = K (returned now); state = (r_hat, e2, m).
|
||||
- encaps2(ek, state): v = Compress_dv=4(t_hat·r + e2 + Decompress_1(m)) = ct2(128B). state is LOCAL
|
||||
(never transmitted) so we can use OUR OWN representation — the libcrux issue-1275 endianness quirk
|
||||
is irrelevant to interop.
|
||||
- decaps(dk,ct1,ct2): standard ML-KEM-768 decaps on ct=ct1||ct2.
|
||||
INTEROP-CRITICAL bytes (all standard FIPS-203, so cross-impl compatible): header=rho||H(ek),
|
||||
ek=ByteEncode12(t_hat), ct1=compress10(u), ct2=compress4(v), ss=32. CONFIRM hdr byte order
|
||||
(rho||H(ek)) against cryspen/libcrux incremental source before relying on it.
|
||||
REUSE: same ring q=3329 as our round-3 Kyber-1024 (Kyber1024.cs) — NTT/zetas/montgomery/barrett/
|
||||
basemul/cbd-eta2 are identical; differences are k=3, du=10/dv=4 compression, and FIPS-203 hashing
|
||||
(keygen G(d||k); encaps (K,r)=G(m||H(ek)); implicit reject J(z||c); NO final KDF unlike round-3).
|
||||
VALIDATE: FIPS-203 ML-KEM-768 NIST KAT (deterministic) + end-to-end vs captured messages.
|
||||
**SCOPE-REDUCER:** BouncyCastle 2.5.1 has MLKemParameters.ml_kem_768 (standard FIPS-203). For the
|
||||
RECEIVING side we can likely reuse BC: keygen via BC (then header=rho||SHA3-256(ek), ek=BCek[..1152]),
|
||||
and decaps by reassembling ct=ct1(960)||ct2(128)=1088 (standard ct size) and calling BC decaps with
|
||||
our dk. Only the SENDING side's encaps1/encaps2 SPLIT (compute u=ct1 from header before ek arrives,
|
||||
then v=ct2) needs a from-scratch K-PKE encaps — and only if the protocol requires emitting ct1 before
|
||||
the peer's ek is fully received. CHECK lib.rs send/recv flow to confirm which ML-KEM ops the RECEIVE
|
||||
path actually invokes (we are B2A/recipient for the captured messages) before deciding how much to port.
|
||||
5. **kdf.rs** + **authenticator.rs** — HKDF-SHA256 + HMAC. ✅ DONE (`Spqr/Authenticator.cs` via
|
||||
CryptoPrimitives; AuthenticatorTests). util.compare -> FixedTimeEquals. kdf -> CryptoPrimitives.Hkdf.
|
||||
6. **chain.rs** (706 lines) — keyed chain / ChainParams. ✅ DONE (`Spqr/Chain.cs`: Direction,
|
||||
ChainParams[maxJump=25000,maxOoo=2000], KeyHistory(OOO+gc/trim), ChainEpochDirection(HKDF hash
|
||||
chain), Chain[new/AddEpoch/SendKey/RecvKey]; ChainTests: A2B==B2A, out-of-order, add-epoch). Info
|
||||
strings exact incl. "Chain Start" (two spaces). Serialization (into_pb/from_pb) deferred to proto.
|
||||
7. **v1/chunked/** (states, send_ct, send_ek) — the chunked SCKA state machine. ✅ DONE
|
||||
(`Spqr/SckaUnchunked.cs` = 9 crypto states; `Spqr/SckaChunked.cs` = 11 chunked states + `SckaStates`
|
||||
Send/Recv machine + message/payload types). In-memory object form (serialize.rs deferred).
|
||||
8. **proto/pq_ratchet.rs** + **serialize.rs** — prost protobuf for STATE. ⏸ DEFERRED (not needed while
|
||||
sessions are in-memory; required only for durable session persistence). NOTE the WIRE message format
|
||||
(V1Msg / `pq_ratchet` bytes) is NOT protobuf — it is a custom compact format already implemented in
|
||||
`SpqrRatchet` (`[ver=1][varint epoch][varint index][type:1][varint chunkIdx‖32B]`), and IS done +
|
||||
interop-correct (it's MAC'd and a real message decrypted).
|
||||
9. **lib.rs** top-level API: ✅ DONE (`Spqr/SpqrRatchet.cs`: InitialState/Send/Recv, Params, Version,
|
||||
Direction, version-negotiation guard, custom wire (de)serialization).
|
||||
10. **Integration**: ✅ DONE & VALIDATED. `SessionState.Spqr` (in-memory `SpqrRatchet`);
|
||||
`RatchetingSession` inits A2B(Alice)/B2A(Bob) from the PQXDH `pqr_key`; `SignalMessage` carries
|
||||
`pq_ratchet` field 5 (already MAC'd via raw-bytes MAC); `ChainKey.DeriveMessageKeys(seed, pqrSalt)`
|
||||
mixes the SPQR key as the WhisperMessageKeys HKDF **salt** (NOT root/chain); `ReceiverChain` caches
|
||||
message-key SEEDS so out-of-order messages get their own salt. A real captured PreKeySignalMessage
|
||||
decrypts to "Test". (Captured-envelope decrypt harness: `CapturedEnvelopeDecryptTests` [Live] — some
|
||||
stale pre-re-link captures still fail with bad MAC; that's mismatched prekey material, not a bug.)
|
||||
|
||||
## Resume status (updated)
|
||||
|
||||
- **Full reference re-fetched:** `_spqrref/` now holds the COMPLETE v1.5.1 tree (the old partial checkout
|
||||
was missing `v1/chunked/*`, `v1/unchunked/*`, `proto/pq_ratchet.{proto,rs}`). v1/chunked is what lib.rs
|
||||
uses (`v1::chunked::states`). unchunked is the inner per-byte logic the chunked layer wraps.
|
||||
- **Module 4 (incremental ML-KEM-768): ✅ DONE & KAT-validated.** `Spqr/MlKem768.cs` — standard FIPS-203
|
||||
ML-KEM-768 (k=3, du=10/dv=4, G(d‖k=3) keygen, (K,r)=G(m‖H(ek)) encaps, implicit reject J(z‖c)=SHAKE256,
|
||||
NO final KDF) + the incremental split (Generate→hdr/ek(pk2)/dk; Encaps1(hdr,m)→ct1/es/ss; Encaps2(pk2,es)
|
||||
→ct2; Decaps(dk,ct1,ct2)→ss). Reuses round-3 Kyber ring arith; matrix A[i][j]=XOF(rho,j,i) keygen
|
||||
(transposed:false) / XOF(rho,i,j) encrypt (transposed:true) — SAME convention as round-3 (the "FIPS
|
||||
swapped the index" claim is a MYTH; pq-crystals `standard` branch gen_a uses (j,i), gen_at (i,j)).
|
||||
`es` (encaps state) is LOCAL → our own int16-LE format. Validated `MlKem768Tests` against C2SP/CCTV
|
||||
ML-KEM-768.txt vector: Encaps/Decaps/incremental-split byte-exact (SHA256(c) + K match). NOTE: that
|
||||
CCTV vector is FIPS-203 **IPD** (G(d) with no rank byte); the keygen rank byte (final FIPS-203, matches
|
||||
libcrux) is LOCAL-ONLY/interop-irrelevant (keypairs are generated locally, only ek/ct cross the wire),
|
||||
so keygen is validated by self-consistency + dk-structure, encaps/decaps by the vector (ek/dk loaded
|
||||
directly from `MlKem768IpdVector`). 50 offline tests green.
|
||||
|
||||
## INTEGRATION MODEL (confirmed from libsignal v0.96.1, pins SPQR v1.5.1) — read before phase 10
|
||||
|
||||
- **PQXDH HKDF bug FIXED:** `RatchetingSession.DeriveKeys` used info `"WhisperText"` + 64B for BOTH X3DH
|
||||
and PQXDH. PQXDH must use info **`"WhisperText_X25519_SHA-256_CRYSTALS-KYBER-1024"`** + **96B** =
|
||||
root[32]‖chain[32]‖**pqr_key[32]**. The pqr_key is the SPQR **auth_key**. This wrong label corrupted
|
||||
root+chain keys → the real cause of "bad MAC" (independent of SPQR). Now stored in
|
||||
`SessionState.SpqrAuthKey`; serialized SPQR state in `SessionState.PqRatchetState`. Secret-input order
|
||||
(0xFF*32 ‖ DH1 ‖ DH2 ‖ DH3 ‖ [DH4] ‖ KEM_ss) already matched libsignal pqxdh.rs.
|
||||
- **SecretOutput mixing:** the SPQR per-message `key` (Option<32B>) is used ONLY as the **HKDF salt** of
|
||||
the per-message `WhisperMessageKeys` derivation — NOT mixed into root/chain. i.e. `ChainKey.GetMessageKeys`
|
||||
must become `derive(IKM=HMAC(chainKey,0x01), salt=pqr_key, info="WhisperMessageKeys", 80)`. salt=null
|
||||
when SPQR key absent (classic). Root→chain (`WhisperRatchet`) is untouched.
|
||||
- **Wire:** `pq_ratchet` = SignalMessage **field 5** and IS covered by the MAC (libsignal MACs the raw
|
||||
serialized bytes incl. field 5; our `SignalMessage.VerifyMac` already MACs raw `_serialized[..-8]`, so
|
||||
field 5 is covered — just need to PARSE field 5 out and ROUND-TRIP it when we send). MAC = HMAC-SHA256
|
||||
over senderIK(33)‖recvIK(33)‖(verbyte‖proto), truncated 8B (we already do this).
|
||||
- **First message HAS a real salt:** SPQR `Chain` is seeded from auth_key at construction (epoch 0 exists);
|
||||
`send_key(0)`→index 1 + real key, so the first PreKeySignalMessage's salt is non-None. (The
|
||||
`msg_key_epoch==0 && index==0` empty-key case never fires for V1 since sent indices are ≥1.) So the full
|
||||
SPQR recv path (parse V1Msg → states.recv → chain.recv_key(epoch-1,index)) is needed even for msg 1.
|
||||
- **chain_params:** max_jump=25000, max_ooo=2000 (non-self session).
|
||||
- **A2B = initiator/Alice, B2A = recipient/Bob.** We are **B2A** for the captured phone messages (phone
|
||||
initiated). B2A init → `NoHeaderReceived` (send_ct role first epoch).
|
||||
- **OOO caveat:** SPQR salt is per-message, so skipped/out-of-order message keys must be cached as the
|
||||
message-key SEED (HMAC(chainKey,0x01)) + counter and salted lazily at arrival (libsignal
|
||||
MessageKeyGenerator), NOT pre-derived. Refactor `ReceiverChain` cache accordingly (stage after in-order
|
||||
works).
|
||||
- **State persistence:** libsignal stores serialized SPQR state in SessionStructure.pq_ratchet_state
|
||||
field 15; recv commits new SPQR state ONLY after MAC+decrypt succeed. For our captured-PreKeyMessage
|
||||
decryption, in-memory state per session is enough (each PreKeySignalMessage re-establishes).
|
||||
|
||||
## Validation strategy
|
||||
|
||||
- Per-module unit tests (field axioms, polynomial round-trips, ML-KEM-768 KAT, encode/decode round-trips).
|
||||
- **End-to-end against real captured messages:** the phone's PreKeySignalMessages are saved as
|
||||
`%LOCALAPPDATA%\Packages\cdb9e5d5-..._cnsc1k9bd01st\LocalCache\Local\Wingnal\failed-envelope-*.bin`
|
||||
and keep redelivering (ChatReceiver doesn't ack on failure). `LiveChatConnectTests` [Category=Live]
|
||||
decrypts them; success = "bad MAC" disappears and we surface the text. This is the real proof.
|
||||
|
||||
## Key facts learned
|
||||
|
||||
- `SignalMessage.pq_ratchet` = field 5, `addresses` = field 6 (libsignal wire.proto). Our ProtoReader
|
||||
now skips unknown length-delimited fields correctly (fixed `_pos += ReadVarint()` eval-order bug).
|
||||
- KEM ciphertext on the wire is `0x08 || raw` (Kyber-1024 type byte); SPQR uses ML-KEM-768 internally.
|
||||
- Chat socket auth = `Authorization: Basic {aci.deviceId:password}` header (NOT query params).
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
# Message sync & history download (Task 3)
|
||||
|
||||
Status as of 2026-06-16. Scope: backfill contacts/groups/config on a freshly linked device, plus the
|
||||
foundational attachment-download primitive and the link'n'sync message-history transfer. Beyond the
|
||||
original MVP (link + 1:1 text) — built incrementally; the heavy parts are documented here.
|
||||
|
||||
## What's built
|
||||
|
||||
### 1. SyncMessage.Request on connect ✅
|
||||
`ChatPage.RequestSyncAsync` sends a `SyncMessage.Request` to our own ACI for **CONTACTS**, **BLOCKED**,
|
||||
and **CONFIGURATION** right after the chat socket connects (`MessageSender.SendSyncRequestsAsync`). This
|
||||
asks the primary to push account state; responses arrive as inbound sync messages on the socket. GROUPS
|
||||
is intentionally omitted — it was removed from the sync protocol (`reserved /*GROUPS*/ 2` in
|
||||
`SyncMessage.Request.Type`); groups now live in the storage service (see docs/GROUPS.md).
|
||||
|
||||
### 2. Attachment-download primitive ✅ (offline-tested)
|
||||
`Wingnal.Service/Attachments/`:
|
||||
- `AttachmentCipher` — decrypts a CDN blob: verifies the whole-blob **SHA-256 digest**, verifies
|
||||
**HMAC-SHA256** over `iv‖ciphertext` (macKey = key[32..64]), **AES-256-CBC** decrypts (cipherKey =
|
||||
key[0..32]), and truncates to the declared plaintext `size` (strips bucket padding). Also `Encrypt`
|
||||
for tests. Validated by `AttachmentCipherTests` (round-trip, bucket truncation, digest/MAC/keylen
|
||||
rejection).
|
||||
- `AttachmentDownloader` — `GET {cdnUrl(cdnNumber)}/attachments/{cdnKey|cdnId}` (cert-pinned), then
|
||||
`AttachmentCipher.Decrypt`. Reused by contacts sync now and media later.
|
||||
|
||||
This is the foundational primitive (used by contacts/groups sync and, later, media + the history
|
||||
archive). Only the decrypt half is offline-testable; the CDN GET is live-only.
|
||||
|
||||
### 3. Inbound Contacts sync → ContactsStore ✅ (parse/persist offline-tested)
|
||||
- `ContactRecordStream.Parse` — parses the decrypted contacts blob: a flat stream of
|
||||
`[varint length][ContactDetails]` records, each optionally followed by `[avatar.length]` inline avatar
|
||||
bytes (Signal's DeviceContacts stream format).
|
||||
- `ContactsStore` (SQLite `contacts.db`) — upserts ACI → name/number/inboxPosition.
|
||||
- `SyncProcessor` — on an inbound `SyncMessage.Contacts`, downloads the blob (primitive #2), imports it
|
||||
(`ImportContacts`), and persists. Read receipts (`SyncMessage.Read`) raise `ReadReceiptReceived`.
|
||||
- `ChatReceiver` now decrypts to a `Result{Content, Message}` and routes sync messages to `SyncProcessor`
|
||||
via an `onSync` callback; `ChatPage` refreshes conversation **titles** from `ContactsStore` so the list
|
||||
shows names instead of raw ACIs.
|
||||
- Tested: `SyncContactsTests` (stream parse incl. inline avatar; import → names persisted).
|
||||
|
||||
Read state is surfaced (`ReadReceiptReceived`) but not yet persisted to a read column — a follow-up.
|
||||
|
||||
## link'n'sync message-history backfill — BUILT (engine offline-validated; live path needs a re-link)
|
||||
|
||||
Signal's "A Synchronized Start for Linked Devices." The import engine (key derivation → decrypt →
|
||||
gunzip → parse `backup.proto` → populate stores) is implemented and offline-tested; the only
|
||||
live-untestable parts are the CDN poll/download and the actual re-link (Signal doesn't store history
|
||||
server-side, so old messages only transfer **at link time** — an already-linked device must re-link,
|
||||
removing the Wingnal device on the phone and tapping "transfer/sync messages").
|
||||
|
||||
### Built components (`Wingnal.Service/Sync/`, `Attachments/`)
|
||||
- `BackupKey` — HKDF chain ephemeralBackupKey → backup_id → MessageBackupKey (hmac[32]‖aes[32]).
|
||||
**Validated byte-exact against libsignal v0.96.1's own test vector** (`BackupKeyTests`).
|
||||
- `BackupReader` — container `IV[16]‖AES-256-CBC‖HMAC[32]` → MAC-verify → decrypt → PKCS7 unpad →
|
||||
gzip inflate → varint-delimited `BackupInfo` + `Frame`s. **Frame parser validated against libsignal's
|
||||
canonical-backup.binproto**; container round-trip + tamper rejection tested (`BackupReaderTests`).
|
||||
- `BackupImporter` — `Recipient`→contacts, `Chat`+`ChatItem`(StandardMessage text)→per-peer messages
|
||||
(correct direction; Self→"Note to Self"). End-to-end offline test (`BackupImporterTests`).
|
||||
- `Protos/Backup.proto` — vendored libsignal `backup.proto` (csharp_namespace `…Protos.Backup`).
|
||||
- `MessageHistoryImporter` — orchestrates poll → download → derive → read → import.
|
||||
- `AttachmentDownloader.DownloadRawAsync` — raw CDN GET for the archive (decrypted by BackupReader, not
|
||||
AttachmentCipher).
|
||||
- `SignalRestClient.WaitForTransferArchiveAsync` — `GET /v1/devices/transfer_archive` long-poll.
|
||||
- Linking: QR now advertises the `backup5` capability (`ProvisioningManager.LinkAndSyncCapability`);
|
||||
`LinkingManager` captures `ProvisionMessage.ephemeralBackupKey` into `SignalAccount.EphemeralBackupKey`.
|
||||
- `ChatPage` — on first connect after a link+sync re-link, runs the backfill once, then clears the
|
||||
one-time key, persists, and reloads the conversation list.
|
||||
|
||||
### Remaining (live-only / caveats)
|
||||
- **Needs a real re-link to exercise** the poll/download path (offline tests cover crypto+parse+import).
|
||||
- **Transfer-archive descriptor shape** (`RemoteAttachment` cdn/key) and the CDN object **path**
|
||||
(`/attachments/{key}`) are taken from Signal-Server source but untested live — verify on first re-link.
|
||||
- **CDN cert pinning** — see SHORTCUTS.md; the CDN may chain to a public CA, not the bundled Signal CA.
|
||||
- **`backup_id`/derivation assumes no forward-secrecy token** (OLD_DST), which is the link'n'sync case.
|
||||
- Importer covers 1:1 text only (groups/attachments/reactions/other ChatItem types skipped by design).
|
||||
|
||||
### Historical: original plan (now implemented)
|
||||
|
||||
### The flow (confirmed from Signal-Android Provisioning.proto + Signal-Server DeviceController)
|
||||
1. **Capability at link.** The new device advertises the link+sync capability in the QR/link request.
|
||||
The primary then includes `ephemeralBackupKey` (32 bytes, **Provisioning.proto field 14**) in the
|
||||
encrypted `ProvisionMessage` (also `accountEntropyPool` 15, `mediaRootBackupKey` 16, `aciBinary` 17,
|
||||
`pniBinary` 18; `masterKey` 13 is deprecated in favor of `accountEntropyPool`). Wingnal's
|
||||
`Provisioning.proto` now carries these fields.
|
||||
2. **Primary exports + uploads.** The primary serializes its history as a **Signal Backup** file
|
||||
(`backup.proto`: a `BackupInfo` header then a length-delimited stream of `Frame`s —
|
||||
`AccountData`, `Recipient`, `Chat`, `ChatItem`, `StickerPack`, `AdHocCall`, …), **gzip**-compresses
|
||||
it, encrypts it (AES-CBC + HMAC, keys derived from `ephemeralBackupKey`), uploads it to the CDN, and
|
||||
calls `PUT /v1/devices/transfer_archive` (`TransferArchiveUploadedRequest{destinationDeviceId,
|
||||
destinationDeviceRegistrationId, transferArchive}`).
|
||||
3. **New device downloads + imports.** The new device long-polls
|
||||
`GET /v1/devices/transfer_archive?timeout=…` → a `RemoteAttachment{cdn, key}` (or a
|
||||
`RemoteAttachmentError`), downloads the blob (primitive #2), decrypts with the
|
||||
`ephemeralBackupKey`-derived keys, gunzips, parses the backup frames, and imports into the local
|
||||
stores.
|
||||
|
||||
### Client scaffold in place
|
||||
- `Provisioning.proto` — link'n'sync fields added (step 1), so a future linking flow can capture
|
||||
`ephemeralBackupKey`.
|
||||
- `SignalRestClient.WaitForTransferArchiveAsync` — the step-3 long-poll, returning
|
||||
`TransferArchiveDescriptor` (cdn/key or error), or null on a 204 timeout.
|
||||
- `Wingnal.Service/Sync/MessageHistoryImporter` — the seam: `DeriveBackupKey` and `ImportAsync` throw
|
||||
`NotImplementedException` pointing here.
|
||||
|
||||
### What's left to implement (the heavy parts)
|
||||
- **A: capture `ephemeralBackupKey` at link.** Have `LinkDevicePage` advertise the link+sync capability
|
||||
and stash `ProvisionMessage.ephemeralBackupKey` (the linking layer already decrypts the
|
||||
ProvisionMessage — just read the new field).
|
||||
- **B: backup key derivation.** Port libsignal's `MessageBackupKey` derivation
|
||||
(HKDF over `ephemeralBackupKey`/`accountEntropyPool` → `aesKey` + `hmacKey`; info strings per the
|
||||
`libsignal/rust/message-backup` crate). Implement `MessageHistoryImporter.DeriveBackupKey`.
|
||||
- **C: backup container decrypt + gunzip.** The archive is `iv‖AES-256-CBC(ct)‖HMAC` (same shape as the
|
||||
attachment primitive — likely reuse `AttachmentCipher` with the derived 64-byte key), then gzip-inflate.
|
||||
- **D: `backup.proto` frame import.** Add `backup.proto` to `Wingnal.Service/Protos/`, stream-parse the
|
||||
length-delimited `Frame`s, and map: `Recipient`→`ContactsStore`/conversation, `Chat`+`ChatItem`→
|
||||
`MessageStore` (with proper timestamps/authors/threads). This is the bulk of the work (the backup
|
||||
schema is large) and should be staged: header → recipients → chats → chat items.
|
||||
- **E: wire `MessageHistoryImporter` into the post-link path** (poll → download → B/C/D → populate
|
||||
stores), with progress UI.
|
||||
|
||||
Until A–E land, a freshly linked Wingnal shows live messages from connect onward + synced contact names,
|
||||
but does not backfill historical messages. (Note: link'n'sync is also **opt-in and time-bounded** on the
|
||||
primary — the archive is only offered briefly right after linking.)
|
||||
|
||||
## Endpoints / protos referenced
|
||||
- `GET /v1/devices/transfer_archive?timeout=` → `RemoteAttachment | RemoteAttachmentError` (new device).
|
||||
- `PUT /v1/devices/transfer_archive` (primary, not us).
|
||||
- `SignalService.proto`: `SyncMessage.{Request,Contacts,Blocked,Configuration,Read}`, `ContactDetails`,
|
||||
`AttachmentPointer`.
|
||||
- `Provisioning.proto`: `ProvisionMessage.{ephemeralBackupKey=14, accountEntropyPool=15,
|
||||
mediaRootBackupKey=16, aciBinary=17, pniBinary=18}`.
|
||||
- `backup.proto` (libsignal `proto/backup.proto`) — NOT yet vendored; needed for step D.
|
||||
Reference in New Issue
Block a user