ci: install PR batchgen into per-worktree venv instead of PYTHONPATH override - #168
Open
Andrewxu313 wants to merge 1 commit into
Open
ci: install PR batchgen into per-worktree venv instead of PYTHONPATH override#168Andrewxu313 wants to merge 1 commit into
Andrewxu313 wants to merge 1 commit into
Conversation
…override The previous PYTHONPATH=$WORKTREE_PATH approach silently broke GPU CI: workers spawned by launch_http_server crashed at import time when the PR's Python source disagreed with the conda env's pre-compiled C++/CUDA extensions (ABI mismatch, missing symbols, drifted module APIs). The parent then deadlocked in _wait_for_workers_ready since no worker ever reported healthy. Symptom: server log frozen at _wait_for_workers_ready, 0% GPU utilization, port 33001 never opened, 30-min HEALTH_TIMEOUT hit. Switch to per-worktree venv: - python -m venv $WORKTREE/.venv-ci --system-site-packages inherits torch + CUDA deps from the conda env without copying them. - BUILD_OPS=1 pip install -e . --no-deps --no-build-isolation -v rebuilds batchgen's C++/CUDA extensions against the PR source so imports match the runtime classes used by the launcher. - venv lives inside the worktree; git worktree remove cleans it up. POIS's interactive conda env stays untouched (this was the original reason for the PYTHONPATH design).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace the PYTHONPATH-override in
pr-gpu-smoke-incontainer.shwith a per-worktree venv that pip-installs the PR's batchgen with extensions rebuilt against the PR source.Motivation
Run 26448889229 hung at
_wait_for_workers_readyfor 30+ minutes with zero GPU activity. Diagnosis: the launcher setPYTHONPATH=$WORKTREE_PATHso the PR's Python source overrode imports, but batchgen ships compiled C++/CUDA extensions (built April 29 in the conda env). When the PR's Python classes / function signatures drift from the cached.soABI, workers crash silently at module import. The parent then deadlocks waiting for "workers ready" signals that never arrive.The fix mirrors how
build.ymlalready handles the source: build batchgen from the worktree usingBUILD_OPS=1 pip install ... --no-deps --no-build-isolation. But install into a venv with--system-site-packagesso torch + CUDA deps come from the conda env (no 30 GB duplication) and POIS's interactive conda env stays untouched.Per-run cost: one C++/CUDA build per node (~2-3 min based on build.yml's wall time), then cached at
~/.cache/torch_extensions/for subsequent runs of the same source.Type of Change
Checklist