Build the failure views from the fused checkpoint's predictions - #81
Open
duckyquang wants to merge 1 commit into
Open
Build the failure views from the fused checkpoint's predictions#81duckyquang wants to merge 1 commit into
duckyquang wants to merge 1 commit into
Conversation
The taxonomy, complementarity matrix and dropout profile were only ever produced by the harness's Ridge, so the project's headline contribution was demonstrated on a model no reported table uses. The two disagree badly enough that it matters: drop-echo MAE 117.5 against the checkpoint's 20.55. With two modalities the harness only asks for the full set and each singleton, and the missing-modality eval has already scored exactly those three conditions and writes them per example. So --predictions re-reads that dump instead of fitting anything, and the views describe the checkpoint the paper reports. The harness takes modality names directly now -- it never read the embedding values, only the keys, and a caller holding predictions has no arrays to pass. Guarded the two ways this route can silently lie: the sanitized bundle copy has its per-example blocks stripped, and conditions whose labels drift out of order would compare different patients.
duckyquang
requested review from
gullyboyslok,
kevzho,
noahdanehebdon,
paiksca,
sebasmos and
tigerkrittaphas
August 28, 2026 06:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The taxonomy, complementarity matrix and loud-vs-silent dropout profile — the three views that are the point of
src/primed_ai/failure/— were only ever produced by the harness's own Ridge. So the project's headline contribution was demonstrated on a model none of the reported tables use, and the two disagree badly enough that it matters: the ridge harness puts drop-echo MAE at117.5where the fused checkpoint gives20.55on the same 245 rows.This adds a
--predictionsroute that reads the per-example dumpevaluate_missing_modality.pyalready writes, so the views describe the checkpoint the paper reports. Nothing is refit. With two modalities the harness only ever asks for the full set and each singleton, and the eval has already scored exactly those three conditions, so it's a re-reading of the reported predictions rather than a second model:full→{echo, ecg}echo_dropped→{ecg}ecg_dropped→{echo}analyze_modality_failuretakes modality names directly now. It never read the embedding values, only the keys, and a caller holding predictions has no arrays to pass —list(embeddings)already did the right thing for both, so it's a rename and a docstring.Two ways this route could quietly lie, both guarded and tested: the sanitized
docs/results/copy has its per-example blocks stripped (it raises instead of half-running), and conditions whose label vectors drift out of order would silently compare different patients. The fused report also gets its own default output dir, so it can't land on top of the ridge one.Refs #79. I'd leave that issue open until the artifacts are regenerated — see Notes.
Notes
feat/eval-artifactsand the diff here is one commit. It depends on theprovenanceblock and thefailure_report.ridge.jsonnaming from that PR. GitHub will retarget the base tomainwhen Real fairness, calibration, and baseline-gap artifacts from the canonical run #74 merges.docs/results/still has nofailure_report.fused.json— the export path and the naming are in place, but the file itself needs a run. That's why I'm not auto-closing [E14] Failure harness on the fused checkpoint's per-example predictions #79: its first acceptance criterion is that the three views exist for the fused checkpoint on the real test fold, and that isn't true until someone reruns.python scripts/run_failure_analysis.py --predictions results/missing_modality.jsonthenpython scripts/export_result_bundle.pywould produce it. No existing metric should move.