Skip to content

Build the failure views from the fused checkpoint's predictions - #81

Open
duckyquang wants to merge 1 commit into
feat/eval-artifactsfrom
feat/fused-failure-views
Open

Build the failure views from the fused checkpoint's predictions#81
duckyquang wants to merge 1 commit into
feat/eval-artifactsfrom
feat/fused-failure-views

Conversation

@duckyquang

Copy link
Copy Markdown
Member

Summary

The taxonomy, complementarity matrix and loud-vs-silent dropout profile — the three views that are the point of src/primed_ai/failure/ — were only ever produced by the harness's own Ridge. So the project's headline contribution was demonstrated on a model none of the reported tables use, and the two disagree badly enough that it matters: the ridge harness puts drop-echo MAE at 117.5 where the fused checkpoint gives 20.55 on the same 245 rows.

This adds a --predictions route that reads the per-example dump evaluate_missing_modality.py already writes, so the views describe the checkpoint the paper reports. Nothing is refit. With two modalities the harness only ever asks for the full set and each singleton, and the eval has already scored exactly those three conditions, so it's a re-reading of the reported predictions rather than a second model:

  • full{echo, ecg}
  • echo_dropped{ecg}
  • ecg_dropped{echo}

analyze_modality_failure takes modality names directly now. It never read the embedding values, only the keys, and a caller holding predictions has no arrays to pass — list(embeddings) already did the right thing for both, so it's a rename and a docstring.

Two ways this route could quietly lie, both guarded and tested: the sanitized docs/results/ copy has its per-example blocks stripped (it raises instead of half-running), and conditions whose label vectors drift out of order would silently compare different patients. The fused report also gets its own default output dir, so it can't land on top of the ridge one.

Refs #79. I'd leave that issue open until the artifacts are regenerated — see Notes.

Notes

The taxonomy, complementarity matrix and dropout profile were only ever produced
by the harness's Ridge, so the project's headline contribution was demonstrated
on a model no reported table uses. The two disagree badly enough that it matters:
drop-echo MAE 117.5 against the checkpoint's 20.55.

With two modalities the harness only asks for the full set and each singleton,
and the missing-modality eval has already scored exactly those three conditions
and writes them per example. So --predictions re-reads that dump instead of
fitting anything, and the views describe the checkpoint the paper reports.

The harness takes modality names directly now -- it never read the embedding
values, only the keys, and a caller holding predictions has no arrays to pass.
Guarded the two ways this route can silently lie: the sanitized bundle copy has
its per-example blocks stripped, and conditions whose labels drift out of order
would compare different patients.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant