FEAT: Add reset_conversation_async hook to PromptTarget

[Mahdi Alhakim (mahdi-al-hakim)]

Description

Implements item 3 from #1247, which Roman Lutz (@romanlutz) scoped on 2026-07-03 as:

a standardized conversation-reset hook: a no-op reset_conversation_async(*, conversation_id=...) on base PromptTarget, invoked from attack _teardown_async, that targets holding external state (Playwright page, HTTP/websocket conversation_id) override.

A target that holds state per conversation has nowhere to release it, so the issue reporter monkey-patched _teardown_async to refresh a PlaywrightTarget page. RealtimeTarget and WebsocketTarget have the same shape from the other side: each caches one websocket per conversation id and grew its own cleanup_conversation_async, which nothing calls. A scenario hands one target instance to every atomic attack it runs, so those connections accumulate until someone closes the whole target.

Changes

• PromptTarget.reset_conversation_async(*, conversation_id), a no-op on the base so stateless targets are unaffected.
• AttackStrategy._teardown_async hands the objective target every conversation the run used: the live one plus the ones recorded as PRUNED. That is AttackResult.get_active_conversation_ids() read off the context, and a test pins the two equal. It reads the context because execute_with_context_async re-raises rather than returning, so a failed or cancelled run produces no result to read, and those are the runs that leave connections open.
• Eight attacks had a _teardown_async that did nothing. Each would have needed await super()._teardown_async(...), and forgetting that line would have disabled the reset with a green suite, so they are deleted. All eleven concrete attacks inherit the base now.
• RealtimeTarget and WebsocketTarget implement the hook; on both, cleanup_conversation_async delegates to it and warns, removed_in="1.3.0". cleanup_target_async is untouched, since closing the whole target is a different lifetime.
• Only the objective target is reset. Adversarial, scorer and converter targets have their own lifetimes, which is why the adversarial conversations an attack records are skipped. Nothing is removed or renamed from the public API.

TAP kept conversations nothing could name

A PromptSendingAttack retry, a Crescendo backtrack and the rotation in MultiTurnAttackStrategy all record the id they supersede. TAP did not: against a single-turn target a node mints a fresh conversation id every turn and dropped the one it replaced, and branches still standing at the end were never recorded. On a real tree at depth 3, a single-turn target served 3 conversations with 1 still reachable. PAIRAttack subclasses TAP and had both gaps.

A node now reports each conversation as its send returns, and the attack records it then. Recording there rather than while building the result is what makes it survive a run that raises: raising in the second iteration of a depth 3 width 3 tree loses 2 of 3 conversations if you wait for the result. The leading branch is taken back out whenever the lead is recomputed, which is the last step of every iteration, so a run that raises does not report its own conversation as pruned as well.

With that fixed the base lookup covers TAP, so its override is gone and no attack overrides it.

Two things worth knowing

A TAP or PAIR result now names conversations it always opened but could not previously reach, so get_pruned_conversation_ids() returns more of them and the backend message count for such a run rises to the real total. Nothing new is sent; the counts were under-reporting.

Anyone who subclassed RealtimeTarget or WebsocketTarget and overrode cleanup_conversation_async will not have that override called by teardown, which calls reset_conversation_async. The old name still works when called directly, and it had no callers here.

Tests and Documentation

Test ids against origin/main: +38, -10. Ten additions are renames, so 28 are new. The renames: three test_teardown_async_is_noop became test_teardown_async_resets_target_conversation, and seven test_cleanup_conversation_async_* across the two websocket targets became test_reset_conversation_async_*.

They cover the id lookup against real contexts, a raising target being swallowed, and a run releasing its conversation when it succeeds, when it raises and when it is cancelled; end-to-end TAP runs with real nodes asserting that everything the target served stays reachable including when the run raises, that a branched node reports its own conversations, and that the leading branch is never also reported as pruned; and the hook on both websocket targets, including the deprecated alias warning.

Documentation: a "Releasing per-conversation state" section in doc/code/targets/0_prompt_targets.md and a matching one in .github/instructions/targets.instructions.md. Both are .md, so there is nothing to run through JupyText.

uv run pre-commit run --all-files                  18 hooks, all Passed
uv run pytest -n 4 --dist=loadfile tests/unit -q   16710 passed, 8 skipped

Tool versions match uv.lock: ruff 0.16.4, ty 0.0.74, pytest 9.1.1. Integration tests need credentials and were not run.

[Mahdi Alhakim (mahdi-al-hakim)]

Hi Richard Lundeen (@rlundeen2) Roman Lutz (@romanlutz), this has been open for a few weeks now. It is a self-contained change (a reset_conversation_async hook on PromptTarget). I recently landed microsoft/RAMPART#141, so I am glad to match whatever conventions you prefer here. Is there anything I can do to help move it along, or someone I should tag for review?

[Richard Lundeen (rlundeen2)]

(GHCP Generated): I think the underlying problem is real, especially for scenarios. A scenario commonly shares one RealtimeTarget across many concurrent attacks. Conversation IDs prevent state from mixing, but RealtimeTarget caches one websocket per ID. Without conversation-level cleanup, a large or long-running scenario can accumulate open connections until cleanup_target_async() runs.

I do not think AttackStrategy._teardown_async should discover conversation IDs from attack contexts, though. That introduces several concerns:

• AttackResult already defines the canonical relationship through conversation_id, related_conversations, and get_active_conversation_ids(). _get_objective_conversation_ids() duplicates this contract with getattr checks and attack-specific overrides.
• TAP demonstrates the fragility: it needs a custom lookup, and its single-turn path replaces a node's conversation ID each turn without recording the old ID as PRUNED, so those earlier resources still leak.
• The current implementation handles only the objective target. Attacks can also use adversarial targets, scorer targets, and converter targets. Some of their IDs are already stored in related_conversations, but the teardown filters them out. This makes a generic attack-level cleanup guarantee incomplete.
• Custom attacks can return a valid AttackResult while storing transient IDs in a different context shape. They will silently skip cleanup unless they know to override the new protected method.

I suggest this design:

1. Keep a no-op conversation-level cleanup hook on PromptTarget, with RealtimeTarget implementing it. This allows a shared target to release one conversation without destroying the target or its other active conversations.
2. Invoke objective-target cleanup at the AttackExecutor boundary after each run. The executor owns per-run execution, has the objective target, and receives the completed AttackResult.
3. Use result.get_active_conversation_ids() instead of reconstructing IDs from context shape. Ensure error results also contain the real multi-turn session.conversation_id and all related objective conversations before cleanup runs.
4. Make the scope explicit: this pass cleans objective-target conversations only. Define adversarial, scorer, and converter target lifecycles separately rather than implying that attack teardown handles every target.
5. Fix TAP and other rotation paths so every replaced objective conversation is added to related_conversations; otherwise neither result-based nor context-based cleanup can release it.

This keeps the useful part of the proposal for shared realtime targets, while preserving AttackResult as the source of truth and avoiding target-lifecycle policy in each attack implementation.

[Mahdi Alhakim (mahdi-al-hakim)]

Richard Lundeen (@rlundeen2) The TAP half is done. A node now reports each objective-target conversation as its send returns and the attack records it there, so the ones a single-turn rotation replaces and the branches a run walks away from stay nameable. Recording at that point rather than at result time is what makes it survive a run that raises, which otherwise loses 2 of 3 conversations on a depth 3 width 3 tree. That made the TAP override unnecessary, and no attack overrides the lookup now. WebsocketTarget has the same per-conversation cache, so it implements the hook too.

I kept the call in _teardown_async rather than the executor. execute_with_context_async re-raises, so a failed run reaches the executor as an exception rather than a result and completed_results is empty. And attack.execute_async never goes through the executor at all: doc/code/targets/realtime_target.py runs attacks three ways, two of which bypass it.

One resolver now, shared with the error-result builder from #2322. It still reads the context rather than the result, with a test pinning the two equal. Happy to move it if you would rather have it at the executor.