FIX: warn when selected attack techniques have no registered factory - #2466
FIX: warn when selected attack techniques have no registered factory#2466fei (feiiiiii5) wants to merge 8 commits into
Conversation
…d factory resolve_technique_factories silently dropped techniques whose factory was not registered, so a typo or an unregistered custom technique shrank the run without any signal. Emit one warning naming the missing technique(s) in selection order (deduplicated), and update the docstring accordingly. Fixes microsoft#2461
…tion resolves Review follow-ups on the silent-technique-drop warning: - A nonempty technique selection that resolves to zero factories now raises TechniqueResolutionError instead of running baseline-only with a success status — a silently empty evaluation. Partial misses keep the warn-and-continue behavior. The error subclasses ValueError so existing handlers keep working, mirroring DatasetConstraintError. - Skipped selections are now visible in normal flows, not just pyrit_backend.log: ScenarioRunSummary gains skipped_techniques, populated by comparing the scenario identifier's resolved techniques against built display groups (execution-progress independent), and pyrit_scan renders a "Skipped:" line so users see when selected techniques were left out. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
…olution Review round two on the skipped-techniques summary: - resolve_technique_factories() now returns a TechniqueResolution (resolved factories + skipped names) instead of a bare dict; jailbreak/adversarial record resolution.skipped and Scenario persists it into ScenarioResult.metadata["skipped_techniques"] at result creation. The run summary reads that authoritative record — display groups are presentation data (grouped by dataset/target/template depending on scenario) and cannot reveal which factories resolved. - tests/unit/backend/test_scenario_run_service.py's shared fixture now builds a real ScenarioResult (model_construct) carrying scenario_identifier and metadata, fixing the 28 spec-mock failures; added a custom-display-group regression proving labels don't affect reported skips. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
… into issue2461/silent-technique-drop Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
|
Both addressed in a1600c2 (+ merge 0fa4881 pulling in main's jailbreak changes):
Full local: builder tests 38/38, service suites green, ruff clean. |
…ic_attacks and multilingual Review round two follow-ups: build_matrix_atomic_attacks now returns (attacks, skipped) so Cyber/Leakage/RapidResponse persist the authoritative skip record before ScenarioResult creation, and Multilingual consumes resolution.resolved while recording resolution.skipped (its dict-style use of the old return type caused 9 TypeErrors after the TechniqueResolution change). Jailbreak's mocked resolver updated to the new contract. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
|
Both gaps closed in 6b67551:
Local: multilingual 14/14 (was 9 failed pre-fix), builder 38/38, service suites green; ruff clean on all touched files. |
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
one more minor comment, then it should be ready
| return TechniqueResolution(resolved=resolved, skipped=missing) | ||
|
|
||
|
|
||
| def build_matrix_atomic_attacks( |
There was a problem hiding this comment.
This function now returns (attacks, skipped) but is still annotated as list[AtomicAttack], which causes four ty errors in this helper and its callers. It also breaks the documented custom-scenario pattern in doc/code/scenarios/0_scenarios.py, where _build_atomic_attacks_async() directly returns this helper and now returns a tuple instead of the required attack list. Please fix the return contract and update both synchronized documentation files, or preserve the existing list-returning API.
Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
build_matrix_atomic_attacks returns (attacks, skipped technique names) but was still annotated list[AtomicAttack], so ty reported an invalid return type in the helper plus one in each Cyber/Leakage/RapidResponse override that unpacks it. The annotation now matches the code. The documented custom-scenario pattern showed the helper being returned directly from _build_atomic_attacks_async, which no longer type-checks. Updated the percent-format example, its .ipynb twin, and the scenario contributor instructions to unpack the tuple and record the skip list on self._skipped_techniques, since that is what makes a partial technique miss visible in ScenarioResult metadata and the CLI summary. Also drops a duplicated patch_central_database marker and the two call sites ruff-format wants joined. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
|
Addressed in 1b30e56:
If you'd rather keep the one-line helper for custom scenarios, I'm happy to preserve a list-returning API instead — but since surfacing the skip list is the point of the change, unpacking felt like the honest shape for the documented pattern. |
Description
resolve_technique_factories()inpyrit/scenario/core/matrix_atomic_attack_builder.pysilently dropped selected techniques whose factory is not registered — the docstring even documented the behavior as "silently dropped". When a scenario (or a user ofextra_factories) selects a technique that was never registered, e.g. due to a typo or a custom technique added under a different name, the run quietly proceeds with fewer attacks and there is no signal anywhere that selections were discarded.This PR emits a single warning naming every missing technique, then proceeds with the remaining ones as before:
Design notes:
Fixes #2461
Tests
Added three regression tests to
TestResolveTechniqueFactories:All 35 tests in
tests/unit/scenario/core/test_matrix_atomic_attack_builder.pypass locally (73 failed / 569 passedelsewhere intests/unit/scenario/is identical to the pre-change baseline on this machine — those failures are environment-only, unrelated modules). Ruff check + format pass on both touched files.