Skip to content

gh-154208: Fix Windows handle leak when a child process dies early#154365

Open
gianghungtien wants to merge 1 commit into
python:mainfrom
gianghungtien:gh-154208-win-spawn-handle-leak
Open

gh-154208: Fix Windows handle leak when a child process dies early#154365
gianghungtien wants to merge 1 commit into
python:mainfrom
gianghungtien:gh-154208-win-spawn-handle-leak

Conversation

@gianghungtien

@gianghungtien gianghungtien commented Jul 21, 2026

Copy link
Copy Markdown

On Windows, multiprocessing.reduction.DupHandle duplicates a handle in the parent process and relies on the receiving process to steal it later, via DuplicateHandle(..., DUPLICATE_CLOSE_SOURCE) in DupHandle.detach():

if pid is None:
    # We just duplicate the handle in the current process and
    # let the receiving process steal the handle.
    pid = os.getpid()

If the child process is terminated before it unpickles its arguments, it never steals those handles, and the parent's duplicates are leaked for the rest of its lifetime.

This is what the issue reports: creating a Pool and immediately terminating it leaks 4 handles per pool — the two ends of _inqueue and the two ends of _outqueue, which are the four PipeConnection objects pickled into the worker's arguments. (_change_notifier is not passed to workers, and the queue locks are unaffected because SemLock.__getstate__ already uses duplicate_for_child().)

The leak only occurs when the worker dies before unpickling. If the pool runs a task first, the child steals the handles and nothing leaks — which is why the issue's reproducer, which terminates immediately, leaks reliably.

Fix

When a handle is duplicated for a child process that is currently being spawned, duplicate it directly into the child instead. The child then uses the handle as-is, and if it dies early the operating system releases the handle along with the rest of its handle table.

This is not a new mechanism: it is exactly the fix made in bpo-33929 for the pipe used to send the child its parameters, as the comment in popen_spawn_win32.py records:

# bpo-33929: Previously, the read end of pipe was "stolen" by the child
# process, but it leaked a handle if the child process had been
# terminated before it could steal the handle from the parent process.

multiprocessing.synchronize.SemLock already passes its handle to children the same way, through Popen.duplicate_for_child().

The handle is duplicated with the requested access rather than DUPLICATE_SAME_ACCESS, so a read-only pipe end stays read-only in the child, exactly as before. Handles sent to an already running process (send_handle(), or a connection sent through a queue) have no spawning popen and keep using the existing steal mechanism, so that path is unchanged.

Verification

Reproducer from the issue, before:

Initial handle count: 116
Handle count: 123
Handle count: 127
Handle count: 131
Handle count: 135

After (the initial rise is one-time warm-up of the spawn machinery, then it is flat — it stays at 132 over 100 iterations too):

initial: 128
  after  1 pools: 131
  after  2 pools: 131
  after  3 pools: 132
  after  4 pools: 132
  ...
  after 10 pools: 132

The added regression test spawns a child with a pipe end in its arguments, terminates it before it can unpickle them, and checks the process handle count does not grow. It fails on main (AssertionError: 10 not less than or equal to 4) and passes with this change.

test_multiprocessing_spawn (448 tests), test_concurrent_futures and test_multiprocessing_main_handling all pass on Windows.

Fixes #154208

On Windows, ``multiprocessing.reduction.DupHandle`` duplicated a handle in
the parent process and relied on the receiving process to steal it with
``DUPLICATE_CLOSE_SOURCE``. When the child was terminated before it could
unpickle its arguments, that duplicate was never claimed and leaked in the
parent for the rest of its lifetime.

Creating and immediately terminating a ``multiprocessing.pool.Pool`` leaked
four handles (both ends of the input and output queues) per pool.

When a child process is being spawned, duplicate the handle straight into
it instead, so the operating system releases the handle if the child dies
early. This is the same approach already used for the pipe that sends the
child its parameters (bpo-33929) and for semaphores. Handles sent to an
already running process still use the steal mechanism.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

On Windows, creating a multiprocessing pool leaks handles

1 participant