fix+feat(memory): roadmap phases 1-4 — tails, smartness, proof, operability#90
Merged
Conversation
…d extraction, curator provenance - memory add_atom now reports 'quarantined for human review (reason: ...)' instead of 'added atom' when the guard scan routed the atom to quarantine, so the agent (and user) are not misled about what landed in the live store - session Store.Save no longer holds the store mutex during embedding (a potential 10s HTTP call); vector-index add happens after the lock is released, persistence semantics unchanged - session-end extraction uses ONE combined LLM call (summary + facts in a single JSON response) when both episode and fact extraction are enabled, with automatic fallback to the two single-purpose calls on parse failure; all existing safety filters preserved - curator MergeSkills keeps the WORSE provenance of the two inputs (Untrusted/NeedsReview = OR, Sources = deduped union) instead of inheriting the keeper's trust while concatenating a tainted body
…consolidation, index fingerprint, stats API - semantic dedup on add: after the exact-match tier, atoms whose top-1 cosine similarity to a live atom exceeds semantic_dedup_threshold (default 0.92, 0 disables) refresh the existing atom instead of duplicating; in-batch duplicates handled via direct embedding - predicted-intent confidence now weights recall scores; intents below 0.3 confidence are skipped entirely - extractor confidence default is now 0.7 (was 1.0) so unconfident extractions no longer get maximal retention and eviction resistance - new ConsolidateAtoms: greedy similarity clustering (default 0.9) + one LLM merge call per group; originals kept on any failure; merged atoms keep highest confidence and union of associations; live store only, quarantine untouched - vector index persists a corpus fingerprint (count + FNV-1a of sorted atom IDs) in vectors_meta.json; a mismatch on load forces a rebuild instead of silently serving a stale corpus (legacy files rebuild once) - new Stats() API: live/quarantined atoms, quarantine reason breakdown, index vectors/dirty, store size, and recall timeout/failure counters for degradation observability
…anic fix)
- new guard E2E test (env-gated ODEK_E2E_GUARD=1) exercises the real
sidecar through guard.New over HTTP and socket_path: confident-BENIGN
regression, injection detection, batch classification
- new piguard-e2e CI workflow: builds the vendored daemon + gateway,
caches the ONNX model by pinned revision, runs the E2E on
ubuntu-latest (socket mode included), always tears down
- fuzz suites for extractJSON, SKILL.md parsing, and session loading
- fix a REAL panic found by the fuzzer: parseYAMLValue sliced
s[1:len(s)-1] on a 1-char quoted string (' or "), crashing on any
SKILL.md frontmatter line like 'key: "'
…'consolidate' - stats: store/index sizes, quarantine reason breakdown, recall timeout/failure counters with a degradation warning - makes guard false-positive rates and recall health visible to operators - consolidate: merges near-duplicate live atoms via the operator's configured LLM backend (10-minute bounded run)
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
odek | 4c4010d | Commit Preview URL Branch Preview URL |
Jul 22 2026, 12:07 PM |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
- CONFIG.md: add semantic_dedup_threshold and consolidate_similarity_threshold to the extended config example and field reference - MEMORY.md: combined session-end extraction call, bounded drain at close (episode extraction no longer fire-and-forget), quarantine-not-reject guard behavior with the honest add_atom report, stats in the CLI surface and observability sections, complete memory tool action enum - EXTENDED_MEMORY.md: consolidate/stats now have CLI commands - AGENTS.md: PIGuard E2E and fuzz soak commands in Testing
The GitHub Actions job was too slow to keep (a ~735 MB ONNX model download plus two image builds per run). docker/piguard-e2e.sh now provisions the same stack locally (build-if-missing, health wait, env-gated test, always-tear-down) with an optional --linux flag that runs the test binary inside a container for full socket-mode coverage (the host socket subtest skips on macOS because unix sockets do not cross the Docker Desktop VM boundary). Verified locally: host run passes, container run passes all 6 subtests including socket mode.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delivers the four-phase memory roadmap. Split into four per-phase commits for reviewability. Note for reviewers: the one roadmap item not in this PR is upstreaming the PiGuard fixes (arm64 base digests, linux-arm64 Makefile checksums, gateway JSON handling) to
go-prompt-injection-guard— separate repo.Phase 1 — known tails (
18c21d6)memory add_atomreported "added atom" even when the guard had quarantined the atom. It now reports "quarantined for human review (reason: ...)" by diffing quarantine before/after.Saveno longer embeds under the store mutex — a potential 10s HTTP call no longer serializes all concurrent session Loads/Lists/Saves. Persistence semantics unchanged.{summary, facts[]}extraction with automatic fallback to the two single-purpose calls on parse failure; all safety filters (FactLooksUnsafe, scan, taint gates) preserved.Phase 2 — memory smartness (
2447958)semantic_dedup_threshold, default 0.92) refresh the existing atom instead of accumulating — e.g. "My favorite editor is Emacs" vs "Roland's favorite editor is Emacs" now merge. Exact-match tier from fix(memory): efficiency and correctness overhaul for extended + legacy memory #88 remains the fast path; in-batch duplicates handled.ConsolidateAtoms: similarity clustering + one LLM merge per group; originals kept on any failure; quarantine untouched.vectors_meta.jsonnow carries a corpus fingerprint (count + FNV-1a of sorted IDs); mismatch on load forces rebuild instead of silently serving a stale corpus.Stats()API with recall timeout/failure counters (backs Phase 4).Phase 3 — proof (
1f68b9a)piguard-e2eCI workflow: builds the vendored daemon + gateway, caches the ONNX model by pinned revision, runs an env-gated guard E2E (confident-BENIGN regression, injection detection, batch, and socket mode on Linux) on every push/PR to main.extractJSON, SKILL.md parsing, and session loading.parseYAMLValuepanicked (slice out of range) on a 1-char quoted value — any SKILL.md frontmatter line likekey: "crashed the parser.Verified locally before commit: the E2E passes on this machine against the real stack (including a Linux-conditions container run where socket mode passes; macOS host can't cross the VM boundary for unix sockets, CI covers it); fuzz soaks of 20k–1.8M execs each, clean after the fix.
Phase 4 — operability (
5ada47d)odek memory extended stats: live/quarantined atoms, quarantine reason breakdown (tainted vs scan_rejected — the guard FP signal), index vectors/dirty, store size, recall timeout/failure counters with a degradation warning.odek memory extended consolidate: runsConsolidateAtomsagainst the operator's configured backend (10-minute bound).Verification
go test ./...— all green;-racegreen on memory/extended/session/skills/guard.golangci-lint run ./...— 0 issues.statssmoke-tested against a live~/.odek.