feat (closes #7246): add new component: leafcutter/differentialsplicing - #12271
feat (closes #7246): add new component: leafcutter/differentialsplicing#12271JTL-lab wants to merge 13 commits into
Conversation
…ot, set python environment hash variable
| { assert snapshot( | ||
| process.out.findAll { key, val -> key.startsWith('versions') }, | ||
| process.out.cluster_significance.collect { file(it[1]).getName() }, | ||
| process.out.effect_sizes.collect { file(it[1]).getName() } | ||
| ).match() }, | ||
| // Check non-deterministic files exist | ||
| { assert file(process.out.cluster_significance[0][1]).exists() }, | ||
| { assert file(process.out.effect_sizes[0][1]).exists() } |
There was a problem hiding this comment.
Try this:
| { assert snapshot( | |
| process.out.findAll { key, val -> key.startsWith('versions') }, | |
| process.out.cluster_significance.collect { file(it[1]).getName() }, | |
| process.out.effect_sizes.collect { file(it[1]).getName() } | |
| ).match() }, | |
| // Check non-deterministic files exist | |
| { assert file(process.out.cluster_significance[0][1]).exists() }, | |
| { assert file(process.out.effect_sizes[0][1]).exists() } | |
| { assert snapshot(sanitizeOutput(process.out, unstableKeys: ["cluster_significance", "effect_sizes"])).match()} |
| process { | ||
|
|
||
| withName: 'LEAFCUTTER_DIFFERENTIALSPLICING' { | ||
| ext.args = '--min_samples_per_group 2 --min_samples_per_intron 2 --num_threads 1' |
There was a problem hiding this comment.
Shouldn't need num_threads in here, that should already be in the tool?
| def args = task.ext.args ?: '' | ||
| def prefix = task.ext.prefix ?: "${meta.id}" | ||
| """ | ||
| export USER=\${USER:-nobody} |
There was a problem hiding this comment.
Not strictly required, but useful for reproducibility with PyTorch. I have changed this to be set only within main.nf.test rather than main.nf, and opted to include a note in meta.yml to users about this as an FYI.
…sts, removed redundant --num_threads param in test config args, move container options set in main.nf to main.nf.test and add note in meta.yml
|
Hi @SPPearce, thank you very much for your review and suggestions. I have addressed your feedback in my latest commit. Please let me know if anything else is needed to merge. Thank you! |
| maintainers: | ||
| - "@JTL-lab" | ||
| notes: | ||
| - This module uses PyTorch for statistical analysis. When running in containers, you may need to set environment variables in your pipeline configuration for reproducibility. For example, `containerOptions = '--env USER=nobody --env PYTHONHASHSEED=0 --env CUBLAS_WORKSPACE_CONFIG=:4096:8'` (Docker) or `containerOptions = '--env USER=nobody,PYTHONHASHSEED=0,CUBLAS_WORKSPACE_CONFIG=:4096:8'` (Singularity/Apptainer). |
There was a problem hiding this comment.
There are definitely several modules that set PYTHONHASHSEED directly in the code, that seems important.
Can you explain why the others are required?
There was a problem hiding this comment.
CUBLAS_WORKSPACE_CONFIG=:4096:8 is for reproducibility with PyTorch CUDA (more detail here), upon rereview though CUDA isn't currently implemented (might be in the future, in which case this would be best practice to include) so we could omit this from meta.yml for now.
For USER=nobody, this is to resolve an issue to my understanding caused by an internal PyTorch cache initialization call where if the USER env variable isn't set it throws an OSError, which in Leafcutter differential splicing happens when the multiprocessing workers are being called and the cache directory paths are being constructed, e.g.
Test Process LEAFCUTTER_DIFFERENTIALSPLICING │
│ │
│ Test [8fb16eda] 'homo_sapiens - bam - differential splicing' FAILED (19.617s) │
│ │
│ Assertion failed: │
│ │
│ 2 of 2 assertions failed │
│ │
│ Nextflow stdout: │
│ │
│ ERROR ~ Error executing process > 'LEAFCUTTER_DIFFERENTIALSPLICING (test_ds)' │
│ │
│ Caused by: │
│ Process `LEAFCUTTER_DIFFERENTIALSPLICING (test_ds)` terminated with an error exit status (1) │
│ │
│ │
│ Command executed: │
│ │
│ leafcutter-ds \ │
│ test_ds_perind_numers.counts.gz \ │
│ groups_file.txt \ │
│ --output_prefix test_ds_results \ │
│ --num_threads 4 \ │
│ --min_samples_per_group 2 --min_samples_per_intron 2 │
│ │
│ Command exit status: │
│ 1 │
│ │
│ Command output: │
│ Loading counts from test_ds_perind_numers.counts.gz │
│ Loading metadata from groups_file.txt │
│ Settings: Namespace(counts_file='test_ds_perind_numers.counts.gz', groups_file='groups_file.txt', baseline_group='Control', │
│ output_prefix='test_ds_results', max_cluster_size=inf, min_samples_per_intron=2, min_samples_per_group=2, min_coverage=20, min_unique_vals=10, │
│ exon_file=None, init='brr', timeit=False, num_threads=4) │
│ Running differential splicing analysis... │
│ │
│ Command error: │
│ optimizer = MyLBFGS(params, **lbfgs_kwargs) #lr=0.05, max_iter=inner_iterations, tolerance_grad=1e-4, history_size = 20) │
│ File "/usr/local/lib/python3.13/site-packages/torch/optim/lbfgs.py", line 272, in __init__ │
│ super().__init__(params, defaults) │
│ ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ │
│ File "/usr/local/lib/python3.13/site-packages/torch/optim/optimizer.py", line 408, in __init__ │
│ self.add_param_group(cast(dict, param_group)) │
│ ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ │
│ File "/usr/local/lib/python3.13/site-packages/torch/_compile.py", line 47, in inner │
│ import torch._dynamo │
│ File "/usr/local/lib/python3.13/site-packages/torch/_dynamo/__init__.py", line 13, in <module> │
│ from . import ( │
│ ...<6 lines>... │
│ ) │
│ File "/usr/local/lib/python3.13/site-packages/torch/_dynamo/aot_compile.py", line 17, in <module> │
│ from torch._dynamo.package import SystemInfo │
│ File "/usr/local/lib/python3.13/site-packages/torch/_dynamo/package.py", line 1173, in <module> │
│ DynamoCache = DiskDynamoCache(os.path.join(cache_dir(), "dynamo")) │
│ ~~~~~~~~~^^ │
│ File "/usr/local/lib/python3.13/site-packages/torch/_dynamo/package.py", line 1170, in cache_dir │
│ return cache_dir() │
│ File "/usr/local/lib/python3.13/site-packages/torch/_inductor/runtime/cache_dir_utils.py", line 17, in cache_dir │
│ os.environ["TORCHINDUCTOR_CACHE_DIR"] = cache_dir = default_cache_dir() │
│ ~~~~~~~~~~~~~~~~~^^ │
│ File "/usr/local/lib/python3.13/site-packages/torch/_inductor/runtime/cache_dir_utils.py", line 23, in default_cache_dir │
│ sanitized_username = re.sub(r'[\\/:*?"<>|]', "_", getpass.getuser()) │
│ ~~~~~~~~~~~~~~~^^ │
│ File "/usr/local/lib/python3.13/getpass.py", line 175, in getuser │
│ raise OSError('No username set in the environment') from e │
│ OSError: No username set in the environment │
│ """ │
│ │
│ The above exception was the direct cause of the following exception: │
│ │
│ Traceback (most recent call last): │
│ File "/usr/local/bin/leafcutter-ds", line 10, in <module> │
│ sys.exit(leafcutter_ds()) │
│ ~~~~~~~~~~~~~^^ │
│ File "/usr/local/lib/python3.13/site-packages/leafcutter/__main__.py", line 12, in leafcutter_ds │
│ run_path(str(script_pth), run_name="__main__") │
│ ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │
│ File "<frozen runpy>", line 287, in run_path │
│ File "<frozen runpy>", line 98, in _run_module_code │
│ File "<frozen runpy>", line 88, in _run_code │
│ File "/usr/local/lib/python3.13/site-packages/leafcutter/differential_splicing/leafcutter_ds.py", line 128, in <module> │
│ res = differential_splicing(counts, meta["group"], confounders = confounders, max_cluster_size = args.max_cluster_size, │
│ min_samples_per_intron = args.min_samples_per_intron, min_samples_per_group = args.min_samples_per_group, min_coverage = args.min_coverage, init = │
│ args.init, device = "cpu", num_cores = args.num_threads, timeit = args.timeit) │
│ File "/usr/local/lib/python3.13/site-packages/leafcutter/differential_splicing/differential_splicing.py", line 230, in differential_splicing │
│ pool_results = list(pool.imap(task_task, _cluster_iter(), chunksize=max(1, len(cluster_ids) // (num_cores * 8)))) │
│ File "/usr/local/lib/python3.13/multiprocessing/pool.py", line 873, in next │
│ raise value │
│ OSError: No username set in the environment │
I was thinking I could potentially ask the main leafcutter-ds developer to set a fallback for os.environ['USER'] in the code to make it more container-friendly. But for now in the interim maybe we opt to keep PYTHONHASHSEED=0, omit CUBLAS_WORKSPACE_CONFIG=:4096:8, keep --env USER=nobody (could hypothetically be any string val, Pytorch just needs it set to something so just following Unix convention) in containerOptions, and include a clearer note in meta.yml about the purpose of these?
Happy to make these changes asap if you are in agreement, just let me know!
There was a problem hiding this comment.
The more transparent pattern is to set in the bash with a default if it's not set.
export USER="\${USER:-nobody}"
export PYTHONHASHSEED=0
leafcutter-ds \There was a problem hiding this comment.
I have implemented @mahesh-panchal's suggested pattern. Please let me know if there is anything else required to merge. Thank you both for the feedback!
Edit: I am seeing several failing checks for my latest commit 8333804, but these seem unrelated to the refactor and notably the failures are all related to DUCKDB_TABLE2PARQUET snapshot mismatches across all test profiles.. For example, for the nf-test \ x64 | singularity | 1 test:
Run NFT_WORKDIR=~
🚀 nf-test 0.9.5
https://www.nf-test.com/
Please cite: https://doi.org/10.1093/gigascience/giaf130
(c) 2021 - 2026 Lukas Forer and Sebastian Schoenherr
Load .nf-test/plugins/nft-anndata/0.4.1/nft-anndata-0.4.1.jar
Load .nf-test/plugins/nft-bam/0.6.1/nft-bam-0.6.1.jar
Load .nf-test/plugins/nft-csv/0.1.0/nft-csv-0.1.0.jar
Load .nf-test/plugins/nft-compress/0.1.0/nft-compress-0.1.0.jar
Load .nf-test/plugins/nft-fastq/0.1.0/nft-fastq-0.1.0.jar
Load .nf-test/plugins/nft-utils/0.0.9/nft-utils-0.0.9.jar
Load .nf-test/plugins/nft-vcf/1.0.7/nft-vcf-1.0.7.jar
nf-test runs in CI mode.
Test Process ANGSD_DOSAF
Test [5e64d21f] 'angsd - GL 1 - doSAF 1'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-5e64d21f7d64ffcb3f8e7e3193366a29.nf` [pensive_legentil] DSL2 - revision: 4bcd289e53
> Pulling Singularity image https://depot.galaxyproject.org/singularity/angsd:0.940--h13024bc_4 [cache /home/runner/work/modules/modules/.singularity/depot.galaxyproject.org-singularity-angsd-0.940--h13024bc_4.img]
> [df/30feba] Submitted process > ANGSD_DOSAF (FIN)
PASSED (15.793s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Process BISMARK_ALIGN
Test [6e05a7f7] 'bowtie2 | paired-end | sarscov2 genome [fasta]'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-6e05a7f7c8d0af067b53e3954e150b6d.nf` [tiny_gutenberg] DSL2 - revision: 3ebdd28580
> WARN: Unknown directive `params` for process `BISMARK_GENOMEPREPARATION`
> WARN: Unknown directive `params` for process `BISMARK_ALIGN`
> Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/bd/bddea334e6ccbce005ce540214747acf822b040185d2198220dcfbb4b258c331/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-bd-bddea334e6ccbce005ce540214747acf822b040185d2198220dcfbb4b258c331-data.img]
> [6c/3f3545] Submitted process > BISMARK_GENOMEPREPARATION (BismarkIndex/genome.fasta)
> [83/99c9d7] Submitted process > BISMARK_ALIGN (test)
PASSED (12.581s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Process DUCKDB_TABLE2PARQUET
Test [c3b78b2c] 'csv - stub'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-c3b78b2c31a37dfdda2803f306594bc6.nf` [loquacious_sammet] DSL2 - revision: 35f4d8c2b9
> Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/68/68261e24307fdf80b9988d6fd13cf3551735e6dc0e7e38003259f7d8efa84cdb/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-68-68261e24307fdf80b9988d6fd13cf3551735e6dc0e7e38003259f7d8efa84cdb-data.img]
> [21/941646] Submitted process > DUCKDB_TABLE2PARQUET (test)
java.lang.RuntimeException: Different Snapshot:
--- /tmp/expected12306652735755230492json 2026-07-30 15:31:25.890398238 +0000
+++ /tmp/found8756876436986920248json 2026-07-30 15:31:25.891459980 +0000
@@ -1,20 +1,5 @@
[
{
- "0": [
- [
- {
- "id": "test"
- },
- "test.parquet:md5,d41d8cd98f00b204e9800998ecf8427e"
- ]
- ],
- "1": [
- [
- "DUCKDB_TABLE2PARQUET",
- "duckdb",
- "1.5.5"
- ]
- ],
"parquet": [
[
{
FAILED (6.14s)
Assertion failed:
1 of 2 assertions failed
Nextflow stdout:
N E X T F L O W ~ version 25.10.2
Launching `/home/runner/work/modules/modules/.nf-test-c3b78b2c31a37dfdda2803f306594bc6.nf` [loquacious_sammet] DSL2 - revision: 35f4d8c2b9
Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/68/68261e24307fdf80b9988d6fd13cf3551735e6dc0e7e38003259f7d8efa84cdb/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-68-68261e24307fdf80b9988d6fd13cf3551735e6dc0e7e38003259f7d8efa84cdb-data.img]
[21/941646] Submitted process > DUCKDB_TABLE2PARQUET (test)
Nextflow stderr:
Nextflow 26.04.6 is available - Please consider updating your version to it
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Process FGUMI_EXTRACT
Test [ad3f3ca0] 'homo_sapiens - [fastq1, fastq2]'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-ad3f3ca0272bb9684ee513822a93bfef.nf` [angry_avogadro] DSL2 - revision: ce97306aba
> Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/64/64e8f594b6f0dd879bc5abbe4ca70b6b761e1920e407d9e1c7d27b89004aac34/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-64-64e8f594b6f0dd879bc5abbe4ca70b6b761e1920e407d9e1c7d27b89004aac34-data.img]
> [8a/f67cef] Submitted process > FGUMI_EXTRACT (test)
PASSED (7.103s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Process FGUMI_SIMPLEXMETRICS
Test [8b936acc] 'homo_sapiens - grouped bam - stub'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-8b936acccaf2535dcea2923ebc102380.nf` [irreverent_kilby] DSL2 - revision: 9fa204a04e
> [88/9ecde4] Submitted process > FGUMI_SIMPLEXMETRICS (test)
PASSED (5.12s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Process HMMER_ESLSFETCHINDEX
Test [eba2dea3] 'sarscov2 - fasta - stub'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-eba2dea3132534a1882290a6b1163ea8.nf` [elated_murdock] DSL2 - revision: b8c738c3b8
> Pulling Singularity image https://depot.galaxyproject.org/singularity/hmmer:3.4--hb6cb901_4 [cache /home/runner/work/modules/modules/.singularity/depot.galaxyproject.org-singularity-hmmer-3.4--hb6cb901_4.img]
> [0f/529b53] Submitted process > HMMER_ESLSFETCHINDEX (test)
PASSED (25.618s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Process TAR
Test [7c38ef1d] 'sarscov2 - genome - db - kraken2 - .gz'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-7c38ef1def83f33981fdaf869bc19bce.nf` [fervent_albattani] DSL2 - revision: f18e5b15dc
> Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/52/52ccce28d2ab928ab862e25aae26314d69c8e38bd41ca9431c67ef05221348aa/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-52-52ccce28d2ab928ab862e25aae26314d69c8e38bd41ca9431c67ef05221348aa-data.img]
> [59/f6b6a1] Submitted process > UNTAR (kraken2.tar.gz)
> Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/98/98946ea8217c35441352a94f3e0cd1dfa24137c323e8b0f5dfcb3123b465d0b1/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-98-98946ea8217c35441352a94f3e0cd1dfa24137c323e8b0f5dfcb3123b465d0b1-data.img]
> [dc/ed3da4] Submitted process > TAR (test)
PASSED (6.991s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
Test Subworkflow FASTQ_ALIGN_DEDUP_BISMARK
Test [650a6bfe] 'Params: bismark paired-end | cytosine_report'
> Nextflow 26.04.6 is available - Please consider updating your version to it
> N E X T F L O W ~ version 25.10.2
> Launching `/home/runner/work/modules/modules/.nf-test-650a6bfed4b9797d6c7a7fe77946f6e3.nf` [exotic_faggin] DSL2 - revision: a06673991a
> [8b/ac253c] Submitted process > BOWTIE2 (Bowtie2_Index.tar.gz)
> [d1/2b4ace] Submitted process > HISAT2 (Hisat2_Index.tar.gz)
> [d7/237ec6] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:BISMARK_ALIGN (test)
> [b6/ac7ace] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:BISMARK_DEDUPLICATE (test)
> [d7/b6a8d6] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:BISMARK_METHYLATIONEXTRACTOR (test)
> Pulling Singularity image https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/e9/e994bf4eb3731150511a14f5706b7bdfd64df1b6d40898fff334286c027e0859/data [cache /home/runner/work/modules/modules/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-e9-e994bf4eb3731150511a14f5706b7bdfd64df1b6d40898fff334286c027e0859-data.img]
> [9f/28c498] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:BISMARK_REPORT (test)
> [58/48d703] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:BISMARK_SUMMARY
> [59/6565e5] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:BISMARK_COVERAGE2CYTOSINE (test)
> [f7/ab0431] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:SAMTOOLS_SORT (test)
> [bc/04756b] Submitted process > FASTQ_ALIGN_DEDUP_BISMARK:SAMTOOLS_INDEX (test)
PASSED (12.713s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
FAILURE: Executed 8 tests in 92.073s (1 failed)
Wrote TAP report to file test.tap
Error: Process completed with exit code 1.
Just wanted to flag in case there is something broader going on with CI.
There was a problem hiding this comment.
Thanks for flagging the duckdb issue btw, I fixed that in a separate PR after you highlighted it.
…nf for better transparency as recommended, update meta.yml with note on what is set and why
… main.nf.test per nf-core docs best practices
Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
PR Description
This PR adds a new module
leafcutter/differentialsplicingfor running Leafcutter differential splicing analysis to determine statistically significant junction usage changes between biological conditions of interest. This module relies on the new Python reimplementation of Leafcutter (leafcutter-ds) here (also documented in themeta.yml).PR checklist
Closes #7246
topic: versions- See version_topicslabelnf-core modules test <MODULE> --profile dockernf-core modules test <MODULE> --profile singularitynf-core modules test <MODULE> --profile condanf-core subworkflows test <SUBWORKFLOW> --profile dockernf-core subworkflows test <SUBWORKFLOW> --profile singularitynf-core subworkflows test <SUBWORKFLOW> --profile conda