Keep every column of probe output in golden packs - #1712
Conversation
The packer treats each .dat file under D/ as a spatial field of the form
<x> [<y> <z>] <value>, inferring the dimensionality from the first line and
keeping only the last column of each row:
ndims = len(_extract_doubles(content.split(chr(10), 1)[0])) - 1
doubles = _extract_doubles(content)[ndims :: ndims + 1]
Probe output is not a spatial field. It is a multi-column time series whose
columns are distinct physical quantities, and the set varies by configuration:
1D, general nondim_time, rho, vel(1), pres
bubbles nondim_time, rho, vel(1), pres, alf, R, Rdot, nR, nRdot
hypoelastic nondim_time, rho, vel(1), vel(2), pres, tau_e(1..3)
3D nondim_time, rho, vel(1..3), pres, gamma, pi_inf, qv, c, accel
Under the field interpretation only the final column survived, so the 3D golden
validated the acceleration magnitude alone and discarded density, velocity,
pressure and the sound speed. lag_bubble files were already special-cased for
the same reason; probe files are the remaining case.
Goldens regenerate in a follow-up commit, from a clean tree.
5CAA4E68 and FBB296DA go from 50 stored values to 450 (nine columns per row instead of one); AE9A7D73 from 1 to 9. The added values are the columns the field interpretation was discarding: density, velocity, pressure, void fraction, and the bubble radius/velocity moments. The recorded provenance says (dirty) because regenerating tracked goldens dirties the tree before the metadata is stamped; this is inherent to regenerating existing goldens rather than adding new ones.
There was a problem hiding this comment.
Pull request overview
This PR updates the golden packer so probe .dat outputs under D/ are treated as multi-column time series (retaining all columns) instead of being misinterpreted as spatial fields (which previously retained only the last column). This improves regression coverage for probe diagnostics that are not in the final column.
Changes:
- Special-case probe outputs in
toolchain/mfc/packer/pack.pyto retain all numeric columns. - Regenerate affected probe-based golden packs to include the full probe column set.
- Update golden metadata files produced during regeneration.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/packer/pack.py | Adds probe-specific parsing path to retain all columns for probe outputs. |
| tests/FBB296DA/golden.txt | Updated golden pack content reflecting full probe columns. |
| tests/FBB296DA/golden-metadata.txt | Updated provenance metadata for regenerated golden. |
| tests/AE9A7D73/golden.txt | Updated golden pack content reflecting full probe columns. |
| tests/AE9A7D73/golden-metadata.txt | Updated provenance metadata for regenerated golden. |
| tests/5CAA4E68/golden-metadata.txt | Updated provenance metadata for regenerated golden. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| lines = content.splitlines() | ||
| content = "\n".join(lines[1:]) # Skip the first line | ||
| doubles = _extract_doubles(content) | ||
| elif "probe" in short_filepath: |
| OpenMP : OFF | ||
|
|
||
| Fypp : /home/bok/dev/MFC/build/venv/bin/fypp | ||
| Fypp : /private/tmp/claude-501/-Users-spencer-Downloads/2d95ba68-dea2-407b-8791-a954495b3fb2/scratchpad/mfc/build/venv/bin/fypp |
| OpenMP : OFF | ||
|
|
||
| Fypp : /home/bok/dev/MFC/build/venv/bin/fypp | ||
| Fypp : /private/tmp/claude-501/-Users-spencer-Downloads/2d95ba68-dea2-407b-8791-a954495b3fb2/scratchpad/mfc/build/venv/bin/fypp |
| OpenMP : OFF | ||
|
|
||
| Fypp : /Users/hyeoksu/MyWork/MFC-local/MFC/bubnorm/build/venv/bin/fypp | ||
| Fypp : /private/tmp/claude-501/-Users-spencer-Downloads/2d95ba68-dea2-407b-8791-a954495b3fb2/scratchpad/mfc/build/venv/bin/fypp |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1712 +/- ##
=======================================
Coverage 60.77% 60.77%
=======================================
Files 83 83
Lines 20872 20872
Branches 3101 3101
=======================================
Hits 12685 12685
Misses 6121 6121
Partials 2066 2066 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Fixes #1711.
The defect
toolchain/mfc/packer/pack.pytreats every.datfile underD/as a spatial field of the form<x> [<y> <z>] <value>, infers dimensionality from the first line, and keeps only the last column of each row:Probe output is not a spatial field. It is a multi-column time series whose columns are distinct physical quantities, and the set varies by configuration:
m_data_output.fpp:1519)nondim_time, rho, vel(1), presnondim_time, rho, vel(1), pres, alf, R, Rdot, nR, nRdotnondim_time, rho, vel(1), vel(2), pres, tau_e(1..3)nondim_time, rho, vel(1..3), pres, gamma, pi_inf, qv, c, accelUnder the field interpretation only the final column survived. In 3D that means the golden validated the acceleration magnitude and discarded density, velocity, pressure,
gamma,pi_inf,qv, and the sound speed.lag_bubblefiles were already special-cased for exactly this reason. Probe files are the remaining case.Change
Retain every column for probe files. No header line to skip —
s_write_probe_fileswrites data rows only.Effect on existing goldens
tests/5CAA4E68(1D exp_bubscreen)tests/FBB296DA(1D bubblescreen)tests/AE9A7D73(1D poly_bubscreen)Nine columns per row instead of one. The added values are what was being dropped: density, velocity, pressure, void fraction, and the bubble radius/velocity moments. That widening is the point of the change — it is coverage these cases should always have had.
The regenerated metadata records
(dirty). That is inherent to regenerating tracked goldens: writing them dirties the tree before the metadata is stamped. Newly added goldens do not have this problem.Verification
The three probe cases pass against their regenerated goldens. The change is gated on
"probe" in short_filepath, so non-probe files take the original code path unchanged and no other golden should move. A full local suite run confirms it: 627 passed, 0 failed.An earlier run of the same suite reported the three probe cases failing with "Variable count didn't match". That did not reproduce — the three pass in isolation, pass on consecutive repeat runs, and pass in the clean full-suite run above. It matches non-reproducible flakiness seen on an unrelated branch this session (two chemistry cases, likewise green individually), and appears to be local contention at high
-jrather than anything in this change. Worth knowing it has been seen, in case CI shows it.Why now
#1707 is a defect in the probe sound speed. It could not be given a regression test through the normal golden path, because
cis never the last column in any configuration — a case could exercise the defective code, emit visibly wrong output, and still pass. This PR is a prerequisite for testing that fix, and stands on its own regardless.Note that even with this change, observing
cspecifically requires a 3D probe case; the 1D writes do not emit it at all.