Skip to content

find_nvidia_binary_utility() violates deterministic-search contract on Windows (CWD leakage via shutil.which) #2119

Description

@rparolin

Summary

cuda.pathfinder.find_nvidia_binary_utility() currently delegates to
shutil.which(name, path=trusted_dirs) after assembling a list of trusted
search directories (NVIDIA wheel bin/, CONDA_PREFIX, CUDA_PATH /
CUDA_HOME, etc.).

On Windows, CPython's shutil.which() may prepend the current working
directory
to executable resolution even when the caller supplies an
explicit path=. As a result, a binary located in the process CWD can be
returned in preference to the trusted CUDA / Conda / wheel binary that
pathfinder is supposed to discover.

This violates the pathfinder contract — callers expect a deterministic
lookup against a documented, bounded set of trusted roots, not a
shell-style lookup that depends on ambient process state.

Reproduction (conceptual)

  1. On Windows, install nvcc via the NVIDIA wheel (so it lives under a
    site-packages/.../bin directory pathfinder considers trusted).
  2. Set CWD to a directory containing a different nvcc.exe (e.g.
    %CONDA_PREFIX%\Library\bin, or any attacker-writable location).
  3. Call cuda.pathfinder.find_nvidia_binary_utility("nvcc").
  4. Expected: the wheel-resolved nvcc.exe (deterministic, contract-
    honoring).
    Actual: the CWD nvcc.exe is returned because shutil.which
    prepended CWD ahead of the supplied path=.

Leo's framing (#2111 comment):

Assuming a pathological case where my CWD is %CONDA_PREFIX%\Library\bin
[...] Following the pathfinder contract I would expect to get the latter
NVCC. But because shutil.which prepends CWD (TIL!), I would actually
get the former.

Impact

  • Contract correctness: the public API silently returns a binary that
    was never in the documented trust set.
  • Security-adjacent: a planted nvcc.exe (or other supported utility)
    in an attacker-controlled CWD is executed downstream by callers that
    invoke the returned path. Originally flagged by an internal security
    scan.
  • Cross-platform inconsistency: Linux/macOS do not exhibit the CWD
    prepend, so behavior diverges by platform in a way users cannot predict.

Root cause

shutil.which() is a "what would the shell run?" primitive, not a "search
exactly these directories" primitive. It carries ambient OS/Python policy:

  • Windows: CWD prepended in certain Python versions (observed on 3.12).
  • PATHEXT expansion is implicit and version-dependent.
  • Empty/relative entries in path= become CWD-sensitive.
  • Return value is not guaranteed to be an absolute, normalized path.

None of these are appropriate for a deterministic discovery API.

Proposed direction

Replace the shutil.which delegation with an explicit, bounded directory
walk (see PR #2111 for a first cut). Key properties:

  • Search only directories pathfinder constructs:
    1. NVIDIA Python wheel bin/ under site-packages
    2. CONDA_PREFIX bin layout
    3. Explicit CUDA_PATH / CUDA_HOME
    4. CTK root resolved via the existing canary mechanism, then CTK bin
  • Preserve Windows extension normalization (.exe, PATHEXT handling
    done explicitly, not inherited from the environment).
  • Preserve Unix executable-bit checks.
  • Return absolute, normalized paths.
  • Do not consult ambient PATH or CWD by default.
  • Document the search order and the non-inclusion of PATH/CWD.

Open design question — ambient PATH

Keith raised (#2111 comment) that the official Linux CUDA install guide
instructs users to add the CTK bin/ to PATH:
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#environment-setup

Users who follow that guide — setting LD_LIBRARY_PATH for libraries and
PATH for executables, without setting CUDA_PATH/CUDA_HOME — would
need to set an explicit env var for pathfinder to discover their CTK
binaries once shutil.which is removed.

Options to weigh:

  1. No PATH fallback (PR [codex] Avoid untrusted binary lookup paths #2111 as written). Maximally deterministic;
    requires affected users to set CUDA_PATH/CUDA_HOME. Cleanest
    contract.
  2. PATH fallback on Linux/macOS only. Sidesteps the Windows CWD
    issue (the actual bug) while preserving the documented install path
    for the platform where the docs apply. Asymmetric but pragmatic.
  3. Opt-in PATH search via parameter or dedicated env var (e.g.
    CUDA_PATHFINDER_ALLOW_PATH=1). Explicit, non-default, auditable.

Leo's call requested before landing the fix.

Regression coverage

A test that plants a Windows CWD binary and asserts it is not returned
is required (already drafted in PR #2111).

References

Metadata

Metadata

Assignees

Labels

P0High priority - Must do!bugSomething isn't workingcuda.pathfinderEverything related to the cuda.pathfinder module

Type

Fields

No fields configured for Bug.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions