feat(storage): add semantic transcript position snapshots

[Sun-GLiang]

Summary

Adds the Storage-only semantic transcript position producer required by the later Runtime Host pager/subscription work. One exact snapshot generation atomically materializes two indexed projections:

• owner: canonical raw logical membership;
• shared: only shared-visible Turns/notes, with projection-dense ordinals and first-visible durable sequence anchors.

Positions use explicit tagged keys (turn, note, empty), so consumers never infer identity from session: or session-note: prefixes. Shared visibility is a closed, versioned scalar policy in Core; the existing Runtime Host projector reuses it for include/drop while retaining sole ownership of body sanitization.

Refs #4123

Plan: issue-4123-pr1-failure-pair-matrix-v1.3.5-controller-approved-2026-08-31
PR: 1 of 4
Exact implementation base: 8c491e64b35cd7af66437efc8020a88b171e2bb6
Projection-aware amendment start: 21137b50e4f7172b6fdadcac96f8ab2ba2089b1c
Final exact head: 1c79bad38d12cffabfd329be8bfd83baf8728561

Verification

• npm --workspace @maka/core test — 715 passed, 0 failed
• npm --workspace @maka/storage test — 1,065 passed, 7 skipped, 0 failed (1,072 total)
• npm --workspace @maka/runtime-host test — 1,414 passed, 9 skipped, 0 failed (1,423 total)
• npm run format:check — 1,781 files checked, no fixes
• npm run lint — 2,965 files checked, no fixes
• npm run typecheck — passed
• npm test — exit 0, all workspace tests passed
• npm run check:asf-headers — 3,131 covered, 169 excluded, passed
• npm run check:third-party-notices — passed
• npm run check:cli-third-party-notices — passed
• git diff --check 21137b50e4f7172b6fdadcac96f8ab2ba2089b1c...HEAD — passed
• git diff --check 8c491e64b35cd7af66437efc8020a88b171e2bb6...HEAD — passed
• Independent v1.3.5 fixed-point Standards review — no findings
• Independent v1.3.5 fixed-point Spec review — no findings

Deterministic fixtures exercise exactly 1,024 completed recovery records and 4,194,304 source bytes in one step, with larger records resuming from a monotonic byte offset. The 10,000-position owner/shared fixture asserts indexed keyset plans without OFFSET, ROW_NUMBER, COUNT DISTINCT, temporary B-trees, or transcript-wide hot scans. Exact 128-position/64-KiB page, 256-record, and 16-MiB raw body boundaries are covered.

Migration and durability

The final additive, lazy v35 migration creates:

• session_turn_authority_revisions: authority revision, monotonic next generation, shared policy version;
• session_turn_index_state: indexed watermark, cumulative source counters, persisted bounded admission-recovery fixed point, and the terminal failure triple with explicit transcript/admission provenance;
• session_turn_identity_recovery: resumable per-record hash/scanner cursor;
• session_turn_metadata: composite (position_kind, position_id) identity and owner/shared first sequence;
• session_turn_memberships: sequence-to-composite-key membership and shared visibility;
• session_turn_position_snapshots: exact key, named build state, dual projection counters/totals;
• session_turn_snapshot_positions: one union row with nullable owner/shared ordinal and sequence columns;
• session_turn_snapshot_leases: projection-bound consumer ownership.

The migration does not decode bodies at open. Its bounded canonical admission reconciliation recovers legacy v34 bodyless admissions lazily in indexed batches, persists a reusable fixed point across ordinary snapshots, and resets that fixed point only for genuine reconstruction. Admission purge propagates canonical deletion, atomically invalidates building generations, resets recovery, and advances authority exactly once while preserving leased ready snapshots as exact old-authority materializations.

Failure provenance is persisted atomically with reason and sequence. The final v35 CHECK matrix permits only transcript corrupt_source|incompatible_identity and admission corrupt_source|hybrid_missing_admission; partial triples and invalid cross-pairs are rejected. Admission purge clears admission-origin failures without masking transcript corruption, while explicit index invalidation and visibility-policy reset clear both origins.

Ready rows are protected by database immutability guards. Owner restart reclaims snapshot/lease rows while preserving normalized recovery state and monotonic next_snapshot_generation. Shared-policy version drift bumps authority, invalidates old snapshots, resets normalized recovery including the admission fixed point/failure state, and lazily rebuilds through the bounded scanner.

Recovery persists hash state envelope version 1 (sha256, hash-wasm@4.12.0) and scanner state version 1. It validates physical chunks/digests, UTF-8/JSON structure, and only the top-level identity envelope; decodeStoredMessage remains the sole complete body-schema decoder. Public terminal recovery reasons remain corrupt_source, incompatible_identity, and hybrid_missing_admission.

Final amendment chain

• v1.3.1: lazy bounded recovery of legacy v34 bodyless root admissions before Ready publication
• v1.3.2: session-persisted bounded admission recovery fixed point, avoiding historical rescans for ordinary later snapshots
• v1.3.3: exact admission-purge invalidation, fixed-point reset, and one authority revision bump
• v1.3.4: persisted transcript/admission failure provenance with origin-aware clear semantics
• v1.3.5: database-enforced failure origin/reason pair matrix and reset-path regression coverage

Storage API

All ports are internal Storage contracts and bind every continuation/body/release operation to (snapshotKey, projection, snapshotLeaseId):

• readTurnPositionPageSnapshot({ sessionId, projection, snapshotLeaseId, snapshotKey?, throughSequence?, anchor, maxPositions })
• readTranscriptRecordsByPositionKeysSnapshot({ sessionId, projection, snapshotLeaseId, snapshotKey, positionKeys, maxBytes, maxRecords })
• releaseTurnPositionSnapshot({ sessionId, projection, snapshotLeaseId, snapshotKey })

The page result is the explicit building | capacity | page union and uses totalPositions. Position keys are { kind: 'turn'|'note', id } | { kind: 'empty' }; body keys exclude empty. Body results carry the resolved position key, are grouped by semantic ordinal and raw sequence, and reuse the existing complete-record reader/decoder. A changed valid body that no longer resolves to the frozen key fails closed.

Resource bounds

• Recovery: at most 4 MiB actual source payload and 1,024 completed records per invocation
• Admission reconciliation: at most 1,024 canonical scalar admissions per invocation using the canonical keyset index
• Persisted scanner state: below 64 KiB; captured identity: below 32 KiB UTF-8; JSON depth: below 4,096
• Snapshot build: at most 1,024 canonical candidates per step, assigning both projections in one pass
• Position page: at most 128 positions and 64 KiB encoded metadata
• Retained generations: two per Session/owner epoch across both projections
• Body lookup: at most 128 unique tagged keys, 256 complete raw records, and 16 MiB raw stored bytes, with no partial result

Shared body reads intentionally account for and return all raw membership of a selected shared position; the later Runtime Host consumer applies the existing body projector and must not expose raw counts, bytes, or hidden records.

Review boundaries and deferred work

This PR does not add Runtime Host public operations, wire schemas, signed cursors, subscriptions, Desktop Main/Preload/IPC/Renderer/UI behavior, scrolling/navigation/export behavior, compatibility epochs, or payload-limit changes. PR1 alone does not prove Runtime Host consumer validity or end-to-end composition.

• PR2: consume this producer in the Runtime Host, add the public cursor/subscription/protocol path, and provide real composition proof
• PR3 and PR4: unchanged from the approved four-PR plan
• PR1+PR2 composition review and human review before this Draft can be marked ready

AI use

Select exactly one:

• No generative tool made a substantive contribution
• Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex implemented and tested the Storage/Core producer seam, existing Runtime Host projector parity refactor, migration, recovery, projection snapshots/leases, bounded body lookup, and review fixes. The commits retain the Generated-by: Codex attribution.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[Astro-Han]

Thanks for taking on this architectural follow-up. I verified that #4123 is a real, reproduced product problem: durable Turns remain in Storage while non-resident Turns lose their logical positions in the Renderer. The core direction here is also sound—Storage/Runtime Host should own one lightweight semantic position model, without duplicating transcript bodies.

P1 (reachability ①, normal Session import/append/admission paths): this PR is not an independently complete slice yet. Although described as Storage-only groundwork, it is not inert: the exact head immediately adds eight durable tables, a resumable scanner/hash dependency, recovery/build/snapshot/lease state, and synchronous projection work to every import, append, and root-Turn admission path. At the same time, no production Runtime Host consumer exists in this PR, so merging it cannot improve the user-visible failure or satisfy any end-to-end acceptance criterion by itself. We would be taking the schema, write amplification, migration surface, and long-lived state machine now, while all product value remains in PR2–PR4.

Please keep this as a stacked/draft dependency until the first real Runtime Host consumer is available, then review and land the minimum coherent Storage + consumer slice together (or reshape the stack so the first merge independently exercises the new authority in production). That composition review is also where we can prove which of the current recovery, two-generation retention, lease, and persisted-hash machinery is actually necessary; reviewing 5k+ lines of hand-written production/test code in isolation cannot establish that these are the simplest required semantics.

I am not asking to move ordering or transcript authority into Desktop, and I am not suggesting increasing the byte limit. The concern is the delivery boundary: the demonstrated problem and owner are correct, but this exact PR takes on permanent complexity before it delivers a usable capability.

AI-assisted review (OpenAI Codex). I verified the issue reproduction, exact-head diff, current production write composition, authority boundary, and the absence of a production consumer in this PR.

中文对照

感谢你继续推进这个架构改进。我确认 #4123 是真实且已经复现的问题：持久层里的 Turns 仍然完整，但未驻留的 Turns 在 Renderer 中失去了逻辑位置。核心方向也是正确的——应由 Storage/Runtime Host 维护一份轻量的语义位置模型，而不是复制 transcript 正文。

P1（可达类别 ①，正常的 Session 导入、追加和 root Turn admission 路径）：这个 PR 目前还不是一个可以独立交付的完整切片。 它并非休眠的 Storage 基础设施：当前 head 会立即增加 8 张持久化表、可恢复 scanner/hash 依赖、recovery/build/snapshot/lease 状态，并在所有 import、append 和 root Turn admission 路径上同步维护投影。但本 PR 没有生产环境中的 Runtime Host 消费者，因此单独合并既不能改善用户看到的问题，也不能满足任何端到端验收项。我们会先承担 schema、写放大、迁移面和长期状态机，实际产品价值却全部留到 PR2–PR4。

建议先把它保持为 stack/draft 依赖，等第一个真实 Runtime Host 消费者就绪后，再把最小且连贯的 Storage + consumer 切片放在一起审查和落地；或者重新调整 stack，使第一次合并就能在生产组合中实际使用这份新 authority。届时也才能从真实调用链验证目前的 recovery、双 generation 保留、lease 和持久化 hash 状态中哪些确实不可少。孤立审查 5k+ 行手写生产代码和测试，无法证明这些已经是最简单的必要语义。

我并不是建议把排序或 transcript authority 下放到 Desktop，也不是建议提高字节上限。问题和 owner 都定义正确；需要调整的是交付边界——这个 PR 在交付可用能力之前就先引入了永久复杂度。

本次为 AI 辅助审查（OpenAI Codex）。我核对了 issue 复现、exact-head diff、当前生产写入组合、authority 边界，以及本 PR 中没有生产消费者这一事实。

[Sun-GLiang]

I have converted #4258 back to draft and will keep it unmerged while developing PR2.

PR2 will remain within the approved four-PR checklist and will be built on the exact PR1 head. Once the production Runtime Host implementation consumes the new Storage APIs and the Storage–Runtime Host integration coverage is ready, I will request a composition review of both the PR2 layer and the combined main...PR2 diff.

PR1 will not be marked ready or merged before that review. The composition review will be used to verify whether the current recovery, two-generation retention, lease, and persisted-hash machinery is required by the real Runtime Host lifecycle and protocol guarantees. Anything that cannot be justified by the production call path and corresponding tests will be simplified or deferred.

After the composition is approved, I will reshape and land the minimum coherent Storage + Runtime Host slice together, with the exact merge procedure agreed with the maintainers before PR1 is marked ready.

This stage validates the Storage–Runtime Host contract and provides the merge gate for PR1 and PR2. The remaining Desktop behavior and final end-to-end acceptance will still be verified in PR3 and PR4.

简体中文

我已把 #4258 改回 draft，并将在开发 PR2 期间保持其未合入状态。

PR2 将继续遵循已经批准的四 PR checklist，并基于 PR1 的精确 head 开发。真实 Runtime Host 实现开始消费新的 Storage API，且 Storage–Runtime Host 集成测试完成后，我会同时提交 PR2 层级 diff 和完整的 main...PR2 diff 进行组合审查。

在这次审查完成前，PR1 不会被标记为 ready，也不会单独合入。组合审查将验证当前的 recovery、双 generation、lease 和持久化 hash 机制是否确实由 Runtime Host 的真实生命周期与协议保证所需要。无法通过生产调用链和对应测试证明必要性的机制将被简化或后移。

组合设计获得认可后，我会将最小且连贯的 Storage + Runtime Host 切片一起落地，并在 PR1 被标记为 ready 前与维护者确认具体合入方式。

这一阶段验证 Storage–Runtime Host 契约，并作为 PR1 和 PR2 的合入门槛。其余 Desktop 行为和最终端到端验收仍将在 PR3 和 PR4 中完成。