FEAT: Add --output console | json flag in scenario-results CLI command

[Justin Song (jsong468)]

This is phase 4 of the goal of letting users dig into individual attack results after running scenarios in pyrit_scan or pyrit_shell. See gist here. This PR adds a machine-readable output format to the scenario-results command in both pyrit_scan and pyrit_shell, so a run can be inspected as JSON, not just human-readable console text.

JSON attacks view

JSON conversations view

JSON overview view (see "Less-obvious decisions / Open to feedback" section)

Description

• New --output {console,json} flag on scenario-results (both front-ends). console (default) keeps today's behavior; json serializes the view's typed payload via model_dump_json(). No backend changes.
• Format is orthogonal to view. Each --view builds one typed payload; --output only decides how to serialize it, so every view works with every format without an N×M explosion of renderers:
  • overview → ScenarioOverviewPayload (aggregates + per-group rates)
  • attacks → AttacksTablePayload
  • conversations → ConversationsPayload
  • full → FullPayload { attacks, conversations } (one well-formed JSON document rather than two concatenated ones)
• One shared builder — build_results_payload_async returns the right payload per view, hiding the sync (overview/attacks, embedded data) vs. async (conversations/full, per-attack fetch) split behind a single call, so each front-end collapses to "build the payload, then switch on format."
• Informational notes route to stderr. apply_view_limit_policy messages now print to stderr, so --output json keeps stdout a clean, parseable document (e.g. pyrit_scan ... --output json > out.json captures only JSON while the note still shows on screen).
• shown sibling field added to the attacks/conversations payloads next to the pre-limit total. Console shows "Showing N of M"; JSON consumers get the same signal as data (shown < total ⇒ truncated). It is a computed field (len(...)) so it can't drift from what was actually rendered.

Less-obvious decisions / Open to feedback

• Console overview reuses the framework pretty printer; JSON uses a CLI-owned payload. The scenario printer is a pure render over a ScenarioResult the CLI already holds, so reusing it gives cross-surface consistency (CLI matches the notebook/library output) with no memory coupling. (Conversations can't reuse the framework conversation printer because it's coupled to CentralMemory + Message domain objects the thin REST client doesn't have — hence CLI-owned transcript renderers there.)
• Overview JSON is a compact aggregate, not a mirror of the richer console summary. They're treated as different-purpose artifacts sharing a core (the aggregate numbers): the console serves a human (richness aids skimming), the JSON serves machines (small, stable schema). Keeping it lean minimizes the API contract surface, avoids pulling extra couplings (e.g. the scorer-eval registry and target/scorer identifier properties) into a pure builder, and shrinks the console/JSON drift surface. Fields can be added later without breaking consumers; parity can be revisited if a real consumer needs it. Since Overview is the one view whose console and JSON come from different renderers, they can drift. But, we added a unit test which renders the framework printer and asserts the payload's aggregates appear in its output, failing if the printer's math changes.
• One possible alternative approach would be to not use PrettyScenarioResultMemoryPrinter at all in CLI and just add a new renderer for the payload, similar to print_attacks_table or print_conversations (comes at cost of extra code and repeated logic). This question can also be revisited if consumers need changes.
• Another possible approach would be to just to have ScenarioOverviewPayload mirror all the information presented by the PrettyScenarioResultMemoryPrinter (including retrieving scorer metrics, parsing target/scorer identifier information) while still using the framework printer for the console overview view and enhance the unit test to enforce consistency (comes at cost of extra code, maintenance to keep the two views in line, repeated logic)

Next Phase:

• --output html and file-destination flag (--output-file) are deferred to phase 5; JSON goes to stdout and relies on shell redirection for now (i.e., pyrit_scan scenario-results 5cdb4b76-b1a6-4fc4-9cfb-d0cc85bb8424 --view attacks --output json >output.json)

Tests and Documentation

• Payloads / builders (test_results.py): OutputFormat values + parse_output_format (valid/invalid); --output registered with default console; build_overview_payload aggregates + per-group rates; the characterization test locking overview aggregates to the framework printer's rendered output; build_results_payload_async returns the correct payload per view and composes FullPayload under one shared limit; shown tracks --limit truncation while total stays pre-limit.
• Renderers (test_output.py): print_results_json emits a single parseable document equal to model_dump_json (including the computed shown); print_results_console_async dispatches each payload to the right renderer (overview → framework printer, attacks → table, conversations → transcripts, full → both).
• Front-ends (test_pyrit_scan.py, test_pyrit_shell.py): --output json per view emits parseable JSON on stdout with exit 0; the console path is unchanged; limit-policy notes land on stderr; existing error paths preserved.
• In-code docs: docstrings on the new models, builders, and renderers; --output help text notes JSON is machine-readable with informational notes on stderr so stdout stays parseable.
• Full tests/unit/cli suite green; ruff (incl. import sorting) and the async-suffix build check clean.
• Follow-up: Notebook docs changes to come after phase 5 lands.

[Richard Lundeen (rlundeen2)]

should we not have anything new here and just match the output module?

[Richard Lundeen (rlundeen2)]

hmmm, is there a reason why we're not using pyrit.output? This seems to dupe a lot of code here.

[Richard Lundeen (rlundeen2)]

Similarly, I don't love this code being in cli module