Keep the CUDA memory pool warm between delegates

[shoumikhin]

The CUDA delegate allocates through the stream ordered allocator, whose pool
hands physical memory back to the driver whenever a synchronization observes a
pending free. With the default release threshold of zero that happens repeatedly
during one inference, so nearly every allocation has to map memory again.

Raising the threshold lets the pool keep the memory it already has. The threshold
caps what the pool may keep rather than granting extra headroom, so any fixed
value silently stops helping once a model's footprint passes it: a 64 MiB setting
against a 256 MiB working set still costs about 259 microseconds per allocation,
and at 1 GiB it is no better than no threshold at all. The maximum makes the win
independent of model size, and it is what PyTorch's own async allocator sets.

Because the memory is then held rather than returned at each synchronize, the
backend gives it back explicitly when the last delegate handle is destroyed.

The pool being configured is the device default pool, which is shared with every
other user of the async allocator in this process. So this reads the current
threshold, only ever raises it, remembers the previous value per device, and puts
it back on release, rather than overwriting a setting another user chose.

Two details that decide whether any of it works. The release trims every device
this code raised, taken from its own record, not whichever device the destroying
thread happens to be current on, which on a multi GPU host is often not the
device the model ran on. And it synchronizes before trimming, because the pool
can only release a free the driver has already observed and nothing else on the
teardown path waits.

A failure to configure the pool only costs speed, so it is logged and execution
continues, but the CUDA error is cleared rather than left latched for the next
kernel launch check to report against an unrelated kernel.

Test plan:

Three tests in backends/cuda/runtime/test/test_cuda_allocator.cpp. Each one
allocates, frees, synchronizes, and asserts the pool retained the block before
releasing it, so none of them can pass when the change is absent. The live
allocation test also checks that reserved memory falls but stays above the block
still in use, rather than asserting a driver guarantee that holds either way.

Measured, per allocation, allocating and freeing with a synchronize between:

Orin Nano 3790.79 us before, 2.34 us after
Thor 363.00 us before, 1.51 us after
H100 40.30 us before, 1.01 us after
A100 18.60 us before, 1.03 us after

A model split into 25 delegates went from about 714 to about 518 microseconds
median on an H100. Peak reserved memory is unchanged on a model whose footprint
fits in one pool block.

The pool calls have no equivalent in the HIP compatibility header, so both the
allocator's pool code and the three tests are compiled out on ROCm, and the
change is a no-op there.

Not measured: Windows, and whether a second copy of this translation unit in one
process keeps its own pool record.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22312

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 4fef258 with merge base c27baa8 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[linux-foundation-easycla]

• ❌ The email address for the commit (4fef258) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.