Skip to content

FIX: warn when selected attack techniques have no registered factory - #2466

Open
fei (feiiiiii5) wants to merge 8 commits into
microsoft:mainfrom
feiiiiii5:issue2461/silent-technique-drop
Open

FIX: warn when selected attack techniques have no registered factory#2466
fei (feiiiiii5) wants to merge 8 commits into
microsoft:mainfrom
feiiiiii5:issue2461/silent-technique-drop

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

Description

resolve_technique_factories() in pyrit/scenario/core/matrix_atomic_attack_builder.py silently dropped selected techniques whose factory is not registered — the docstring even documented the behavior as "silently dropped". When a scenario (or a user of extra_factories) selects a technique that was never registered, e.g. due to a typo or a custom technique added under a different name, the run quietly proceeds with fewer attacks and there is no signal anywhere that selections were discarded.

This PR emits a single warning naming every missing technique, then proceeds with the remaining ones as before:

WARNING  pyrit.scenario.core.matrix_atomic_attack_builder:matrix_atomic_attack_builder.py:171 Skipping 2 selected attack technique(s) with no registered factory: missing_a, missing_b. Register the technique(s) (or pass them via extra_factories) to include them in the run.

Design notes:

  • Warning, not an exception: dropping is intentional fallback behavior (per the original docstring), and callers may legitimately probe which techniques are available; this only makes it observable. Happy to switch to raising if maintainers prefer strictness.
  • One aggregated warning per call, names deduplicated but kept in selection order, so repeated selections don't spam the log.

Fixes #2461

Tests

Added three regression tests to TestResolveTechniqueFactories:

  • warning is emitted when a selected technique has no factory, and names it
  • no warning when all selected techniques resolve
  • one warning listing each distinct missing name once, in selection order (deduplicated)

All 35 tests in tests/unit/scenario/core/test_matrix_atomic_attack_builder.py pass locally (73 failed / 569 passed elsewhere in tests/unit/scenario/ is identical to the pre-change baseline on this machine — those failures are environment-only, unrelated modules). Ruff check + format pass on both touched files.

…d factory

resolve_technique_factories silently dropped techniques whose factory was
not registered, so a typo or an unregistered custom technique shrank the
run without any signal. Emit one warning naming the missing technique(s)
in selection order (deduplicated), and update the docstring accordingly.

Fixes microsoft#2461
@romanlutz Roman Lutz (romanlutz) changed the title fix(scenario): warn when selected attack techniques have no registered factory FIX: warn when selected attack techniques have no registered factory Aug 24, 2026
Comment thread pyrit/scenario/core/matrix_atomic_attack_builder.py
Comment thread pyrit/scenario/core/matrix_atomic_attack_builder.py
…tion resolves

Review follow-ups on the silent-technique-drop warning:

- A nonempty technique selection that resolves to zero factories now raises
  TechniqueResolutionError instead of running baseline-only with a success
  status — a silently empty evaluation. Partial misses keep the
  warn-and-continue behavior. The error subclasses ValueError so existing
  handlers keep working, mirroring DatasetConstraintError.
- Skipped selections are now visible in normal flows, not just pyrit_backend.log:
  ScenarioRunSummary gains skipped_techniques, populated by comparing the
  scenario identifier's resolved techniques against built display groups
  (execution-progress independent), and pyrit_scan renders a "Skipped:" line
  so users see when selected techniques were left out.

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
Comment thread pyrit/backend/services/scenario_run_service.py Outdated
Comment thread tests/unit/backend/services/test_scenario_run_service_summary.py Outdated
Roman Lutz (romanlutz) and others added 3 commits August 24, 2026 16:44
…olution

Review round two on the skipped-techniques summary:

- resolve_technique_factories() now returns a TechniqueResolution (resolved
  factories + skipped names) instead of a bare dict; jailbreak/adversarial
  record resolution.skipped and Scenario persists it into
  ScenarioResult.metadata["skipped_techniques"] at result creation. The run
  summary reads that authoritative record — display groups are presentation
  data (grouped by dataset/target/template depending on scenario) and cannot
  reveal which factories resolved.
- tests/unit/backend/test_scenario_run_service.py's shared fixture now builds
  a real ScenarioResult (model_construct) carrying scenario_identifier and
  metadata, fixing the 28 spec-mock failures; added a custom-display-group
  regression proving labels don't affect reported skips.

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
… into issue2461/silent-technique-drop

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Both addressed in a1600c2 (+ merge 0fa4881 pulling in main's jailbreak changes):

  1. Authoritative skips: resolve_technique_factories() now returns a TechniqueResolution (resolved factories + skipped names in selection order); jailbreak/adversarial record resolution.skipped and the base Scenario persists it into ScenarioResult.metadata["skipped_techniques"] at result creation. The run summary reads that persisted record — no display-group derivation. Your Rapid Response/Adversarial/Jailbreak examples would previously have mis-reported; a new regression (test_display_group_labels_do_not_affect_reporting) pins that custom labels never change reported skips.

  2. Shared fixture: _make_db_scenario_result now builds a real ScenarioResult via model_construct carrying scenario_identifier + metadata (with MagicMock method overrides for get_techniques_used / objective_achieved_rate / get_display_groups since individual tests configure them), fixing the 28 failures. Also merged main's jailbreak fail-fast for missing techniques with this branch's resolver semantics: resolver raises only when nothing resolves; scenario-layer raises on any partial miss per main's newer behavior.

Full local: builder tests 38/38, service suites green, ruff clean.

Comment thread pyrit/scenario/core/matrix_atomic_attack_builder.py Outdated
Comment thread pyrit/scenario/core/matrix_atomic_attack_builder.py
…ic_attacks and multilingual

Review round two follow-ups: build_matrix_atomic_attacks now returns
(attacks, skipped) so Cyber/Leakage/RapidResponse persist the authoritative
skip record before ScenarioResult creation, and Multilingual consumes
resolution.resolved while recording resolution.skipped (its dict-style use of
the old return type caused 9 TypeErrors after the TechniqueResolution change).
Jailbreak's mocked resolver updated to the new contract.

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Both gaps closed in 6b67551:

  1. build_matrix_atomic_attacks now returns (attacks, skipped); Cyber / Leakage / RapidResponse unpack it and set self._skipped_techniques before the base Scenario persists metadata["skipped_techniques"] — so those scenarios' summaries and CLI output no longer lose partial-miss information.

  2. Multilingual consumes resolution.resolved (its dict-style iteration was the source of the 9 TypeErrors) and records resolution.skipped. The Jailbreak mock now returns TechniqueResolution(resolved={}, skipped=[]) per the new contract.

Local: multilingual 14/14 (was 9 failed pre-fix), builder 38/38, service suites green; ruff clean on all touched files.

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one more minor comment, then it should be ready

return TechniqueResolution(resolved=resolved, skipped=missing)


def build_matrix_atomic_attacks(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function now returns (attacks, skipped) but is still annotated as list[AtomicAttack], which causes four ty errors in this helper and its callers. It also breaks the documented custom-scenario pattern in doc/code/scenarios/0_scenarios.py, where _build_atomic_attacks_async() directly returns this helper and now returns a tuple instead of the required attack list. Please fix the return contract and update both synchronized documentation files, or preserve the existing list-returning API.

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
build_matrix_atomic_attacks returns (attacks, skipped technique names) but
was still annotated list[AtomicAttack], so ty reported an invalid return
type in the helper plus one in each Cyber/Leakage/RapidResponse override
that unpacks it. The annotation now matches the code.

The documented custom-scenario pattern showed the helper being returned
directly from _build_atomic_attacks_async, which no longer type-checks.
Updated the percent-format example, its .ipynb twin, and the scenario
contributor instructions to unpack the tuple and record the skip list on
self._skipped_techniques, since that is what makes a partial technique
miss visible in ScenarioResult metadata and the CLI summary.

Also drops a duplicated patch_central_database marker and the two call
sites ruff-format wants joined.

Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Addressed in 1b30e56:

  • build_matrix_atomic_attacks is now annotated tuple[list[AtomicAttack], list[str]], which also clears the four ty errors (the one in the helper plus one in each of the Cyber / Leakage / RapidResponse overrides that unpack it).
  • The documented custom-scenario pattern in doc/code/scenarios/0_scenarios.py and its .ipynb twin now unpacks the tuple and records the skip list on self._skipped_techniques — that is the path that surfaces a partial technique miss in ScenarioResult.metadata and the run summary, so the example teaches the same contract the built-in scenarios use. The scenario contributor instructions (.github/instructions/scenarios.instructions.md) describe the second return value the same way.
  • Two call sites ruff-format wants joined, and a duplicated usefixtures marker in the builder tests, dropped.

If you'd rather keep the one-line helper for custom scenarios, I'm happy to preserve a list-returning API instead — but since surfacing the skip list is the point of the change, unpacking felt like the honest shape for the documented pattern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resolve_technique_factories silently drops selected techniques with no registered factory

2 participants