feat: Variant Level CSV Download - #804
Open
bencap wants to merge 5 commits into
Open
Conversation
This was
linked to
issues
Aug 6, 2026
Move all CSV-generation code for score set variant downloads out of score_sets.py into a new dedicated mavedb/lib/score_set_csv.py module, splitting it into pure functions (column planning, header assembly, row formatting) and DB-bound fetching to keep the logic testable in isolation. - Add parse_clinvar_namespace() to clinvar/utils.py, replacing the inline regex parsing that lived in score_sets.py - Add is_csv_output_null() to mave/utils.py to centralize null-value detection used when formatting CSV output - Move variant_to_csv_row, variants_to_csv_rows, get_score_set_variants_as_csv, and drop_na_columns_from_csv_file_rows into score_set_csv.py, and update routers/score_sets.py and scripts/export_public_data.py to import from the new location - Move corresponding tests into tests/lib/test_score_set_csv.py and add new unit tests for parse_clinvar_namespace and is_csv_output_null
Split the monolithic score_set_csv.py into mavedb/lib/csv/, a package organized by responsibility (namespaces, columns, entries, fetch, annotations, specs, deprecated_params) instead of one file mixing column planning, row formatting, and DB fetching for a single record type. score_set.py and variant.py sit on top as the two record-specific entry points, sharing the namespace/column machinery beneath them. This is the foundation for variant-level CSV export and namespace discovery (following commits): the old module had no way to answer "what does this record have data for" or to serve a single variant, so extending it in place would have meant bolting both concepts onto code that already conflated planning and fetching. - Introduce namespace discovery as first-class: CsvNamespace enum, CsvNamespaceGroup, and available_score_set_csv_namespaces / available_variant_csv_namespaces, replacing the hand-maintained CLINVAR_NS_PATTERN validity check in the router - Add lib/annotation/flatten.py to project a variant's VA-Spec interpretation onto scalar CSV columns, sharing acmg.py's new acmg_evidence_outcome_code() with the VA-Spec evidence-line builder so the two representations can't drift - Thread an optional containing_classification_ids set through functional_classification_of_variant and pathogenicity_classification_of_variant, resolving the long-standing TODOs about O(1) membership checks when classifying many variants - Move parse_clinvar_namespace (clinvar/utils.py) and CSV-null-output detection (mave/utils.py) into the csv package, since both were export-side concerns living outside it - Add drop_unused_hgvs_columns, scores_custom, and score_set namespaces; keep drop_na_columns, include_post_mapped_hgvs, and include_custom_columns working as deprecated aliases via resolve_deprecated_csv_params - Add title to ShorterScoreSet so namespace-discovery responses can label a calibration's owning score set without a second lookup BREAKING CHANGE: score_set_csv.py is removed. Callers importing get_score_set_variants_as_csv or variants_to_csv_rows from it must import from mavedb.lib.csv.score_set / mavedb.lib.csv.columns instead.
…points
Add GET /variants/{urn}/csv and GET /variants/{urn}/csv-namespaces,
mirroring the score-set CSV endpoints but scoped to a single variant.
The variant endpoint widens over the variant's equivalent measurements
(as the annotation endpoints already do), so a calibration belonging
to another score set that also measured this allele is included too,
and emits one row per current measurement with the requested variant
first.
Add GET /score-sets/{urn}/csv-namespaces alongside it, and switch
GET /score-sets/{urn}/variants/data to build its namespace list and
validation from discovery instead of a hand-maintained
_VALID_STATIC_NAMESPACES set and the ClinVar regex check.
- Accept drop_unused_hgvs_columns on the score-set CSV endpoints,
keeping drop_na_columns, include_post_mapped_hgvs, and
include_custom_columns working as deprecated query params via
resolve_deprecated_csv_params
- Ask calibration READ permission separately from score-set READ on
every CSV endpoint, since a private calibration's interpretation is
not implied by being able to read the measurement it applies to
- /score-sets/{urn}/scores now includes both score namespaces
("scores" plus "scores_custom"), matching its historical behavior of
returning every score column the investigator uploaded
- Make _stream_generated_annotations resilient to a single variant's
annotation failing mid-stream: log and emit it as unannotated rather
than raising, since the response has already started and an
unhandled exception would silently truncate the file
Replace the hand-maintained list of ClinVar release namespaces in export_public_data.py with annotation_export_namespaces(), which asks available_score_set_csv_namespaces() what the score set actually has and subtracts the score/count/identity groups already covered by their own files in the dump. The previous list named releases one by one, so it silently emitted all-NA columns for a release never ingested and needed a code change for every new one. Discovery-derived namespaces also add score calibration interpretations to the annotations CSV for the first time, since those are now part of what discovery reports. Update scripts/resources/README.md to document the calibration column group and clarify that the ClinVar and calibration groups vary by score set and should be read from the header rather than assumed fixed.
bencap
force-pushed
the
feature/bencap/variant-level-csv-download
branch
from
August 6, 2026 23:09
e39af46 to
c0f0bb9
Compare
bencap
changed the base branch from
release-2026.2.8
to
bugfix/bencap/calibration-permissions
August 6, 2026 23:10
bencap
marked this pull request as ready for review
August 6, 2026 23:12
Coverage Report for CI Build 31130586282Warning No base build found for commit Coverage: 89.286%Details
Uncovered Changes
Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.