interpretability refactor: attention/ → interpretability/ (publish diffae core, quarantine _internal) - #18
Open
gav-sturm wants to merge 13 commits into
Open
Conversation
…ility refactor) Brings the self-contained DiffEx interpretability pipeline (classifier/diffae/ directions + figures/viewer/kyle_pcs) onto the public-release branch. No conflict: refactor had no diffex/ dir. Reorg (rename, _internal quarantine, setacc subdir) follows in subsequent commits.
Whole-dir rename; 52 intra-tree module-path refs updated. 0 external ops_model importers (blast radius nil). Core diffex subpackages import-verified.
- interpretability/diffex/ -> interpretability/diffae/ (the DiffEx->DiffAE pipeline) - inner diffae/ (stage-2 diffusion generator) -> generator/ (resolves the name clash) - patched module paths in .py (5 inner-refs, 46 parent-refs, 17 relative ..diffae->..generator) and doc run-commands; preserved the DiffEx method name, DiffAE class, /models/diffex/ output paths, and diffex-viewer infra names. Core pipeline import-verified (directions -> ..generator wiring intact).
Moved (0 core dependencies, 0 residual refs): atlas, shap, titration, embedding, weighted_aggregation, diffae/kyle_pcs, and the SHAP RUNBOOK.md. Fixed titration's one docstring -m path. viewer + figures NOT yet moved — they're coupled (see next).
…/, viewer -> _internal Core figures depend on 5 viewer modules (catalog, precompute, morpho_pipeline, _fluor_topcells, build_pc_crops_masked) that form a closed set over the core stages. Promoted them to diffae/traversal/ (same depth -> their ..classifier/etc. relatives survive). Remaining ~43 viewer files (webapp/deploy/one-offs/diagnostics) -> _internal/viewer/, with 83 cross-pkg relatives + 24 relatives-to-the-5 rewritten to absolute imports. Core stages' 2 lazy ..viewer.<5> refs -> ..traversal. Import-verified: core stages, traversal (all 5), and figures/_setacc_common.
…validation Diagnostic/validation probes (bag sweeps, centroid recovery, valid200) — not published figures. Absolute imports only, so clean move; self-path -m refs repointed. NOTE: 3 figure scripts (mtor_lyso_loc_traversal, mtor_lysosome_localization, real_gen_measure) are untracked WIP on diffex-interpretability -> not vendored.
…ckage No duplicate code to extract — make_panel (in figure4_setacc_panel) is already the single shared renderer; the 3 variants are thin config wrappers. Moved all 4 into figures/setacc/ with __init__; converted their cross-refs (make_panel, _setacc_common, _setacc_phase) + cis_golgi_alternatives' make_panel import + _setacc_phase's own _setacc_common import to absolute (dual-mode: works under -m and flat). Shared _setacc_common/_setacc_phase stay at figures/ level (used by ~11 scripts). Verified: all 4 panels import as package modules.
hatch wheel target now excludes the not-for-release analysis/paper tooling. Verified via 'uv build --wheel': 0 _internal entries, 73 core diffae files shipped.
Match the ops_process convention: not-for-release tooling lives in toolkit/ on the refactor line (present in source, excluded from the wheel). 20 intra refs repointed, pyproject exclude updated. The public-release branch deletes toolkit/ entirely.
…p PLAN.md Minibinder-free public release: - data_loader: guide-col back-compat bridge -> self.guide_col (was minibinder_perturbation) - cp_extraction / anndata_validator: generic docstrings - diffae core: removed the 'minibinder' GRAIN + its catalog/precompute handling (_minibinder_meta.json / grain=='minibinder' branches) -> geneKO+complex only - 3 guide_col tests: fixtures use a generic custom_perturbation column - dropped diffae/PLAN.md (internal living design doc; also carried latent-lens prose) Entire public-release repo is now minibinder-free (git grep -il minibinder = 0).
…ent-lens from METHODS - virtstain_multi: removed the personal slurm_logs glob fallback (NaN->None instead) - figures/METHODS_*: montage described generically (dropped latent-lens package name+URL)
Single env-overridable BASE_PATH (default /hpc/projects/icd.fast.ops); 25 files' path
literals -> f"{BASE_PATH}/..." + import. One OPS_BASE_PATH swap relocates the whole
data/model/analysis tree. Defaults byte-identical (verified); env-swap verified.
(4 residuals are docstring examples, left as illustrative defaults.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
models/attention/→models/interpretability/refactorRestructures the attention/DiffEx code for the public release: publish only the core
DiffAE interpretability pipeline; quarantine paper-figure/analysis/tooling into a
release-excluded
_internal/.Strategy
diffex/is self-contained (imports nothing from the other attention dirs), so it wasvendored as a clean subtree onto this public-release branch (which had no
diffex/) —no 238-commit merge, inheriting the already-cleaned analysis dirs.
Published tree (
uv build --wheelverified: 0_internalentries, 73 core files)Commits
diffex/baselineattention→interpretability(0 external importers; 52 intra refs)diffex→diffae, innerdiffaestage →generator(module paths only — method name "DiffEx",DiffAEclass,/models/diffex/output paths,diffex-viewerinfra all preserved)_internal/diffae/traversal/; the other ~43 viewer files →_internal/viewer/(83 cross-pkg + 24 relative imports rewritten to absolute)figures/gen_validation(27 diagnostics) →_internal/figures/setacc/subpackage (no dup —make_panelalready shared)_internal/from the wheelVerified
Import-smoke: core stages,
traversal(all 5),figures/_setacc_common, all 4 setacc panels. Wheel excludes_internal/, ships 73 core files.Flags / follow-ups
diffex-interpretabilitywere NOT vendored (git only takes tracked files):mtor_lyso_loc_traversal.py,mtor_lysosome_localization.py,real_gen_measure.py. To include: commit them there, then re-vendor._internal/viewer/build_pcs.pyhas a function-level import ofcombination.embedding_overlays(removed on this branch) — runtime-only, internal tool.diffae/depends onorganelle_profiler.*/ops_utils.*— need their public-release branches.