FEAT: Refactoring score model (phase 2)

.pre-commit-config.yaml

@@ -89,7 +89,13 @@ repos:
         # ty in an isolated env without project deps, which causes spurious
         # ``unresolved-import`` errors now that the rule is set to ``error``.
         # Include optional modules in the environment, including GCG's torch imports.
-        entry: uv run --extra all --link-mode=copy ty check
+        # ``pass_filenames: false`` keeps this to one invocation. pre-commit otherwise
+        # splits the files into parallel batches, and the concurrent ``uv run`` syncs
+        # fight over the same venv and fail to copy locked files on Windows.
+        # ``--frozen`` stops ``uv run`` from rewriting uv.lock, which rewrites every
+        # index URL when a contributor has a private package index configured.
+        entry: uv run --frozen --extra all --link-mode=copy ty check pyrit
         language: system
         files: ^pyrit/
         types: [python]
+        pass_filenames: false

doc/code/framework.md

@@ -255,12 +255,14 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 **Responsibility**: Scorers give feedback to the attack on what happened with the prompt. This could be as simple as "Was this prompt blocked?" or "Was our objective achieved?"
 
 - Any decision an attack makes should be based on a scorer result
-- A scorer is not limited to a prompt, it could be anything (e.g. was this tool called or was this file written).
+- A scorer is not limited to a prompt, it could be anything (e.g. was this tool called or was this file written). It receives a `Scorable`, which identifies that evidence, and an optional `ScoringExpectation`.
+- `TrueFalseScorer` and `FloatScaleScorer` define result families. `MessageScorer` adds message resolution and message-only policy on top of them.
+- `Score.status` marks a verdict complete or undetermined, and the attack decides how to branch on it.
 - **Does not own**: acting on its own result. A scorer evaluates a response and returns a score; branching on that score is the attack's job, and aggregating scores across runs is analytics'. It may call a target to evaluate, but it doesn't send the attack's objective prompt or manage the conversation.
 
 **Framework Plans**:
 
-- Scorers will be refactored to be more generic, so they can determine more general results (does a file exist? Was a tool called?)
+- Loose file evidence is copied into managed results storage. Media already stored in `PromptMemoryEntries` is not yet normalized that way, which is memory retention work.
 
 **Contributing (difficulty low)**:
 

doc/code/scoring/0_scoring.ipynb

@@ -61,10 +61,10 @@
    "metadata": {},
    "outputs": [
     {
-     "name": "stdout",
+     "name": "stderr",
      "output_type": "stream",
      "text": [
-      "No new upgrade operations detected.\n"
+      "Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.\n"
      ]
     },
     {
@@ -75,6 +75,7 @@
       "         AudioFloatScaleScorer float_scale        no\n",
       "      AzureContentFilterScorer float_scale        no\n",
       "              PlagiarismScorer float_scale        no\n",
+      "  SystemPromptExtractionScorer float_scale        no\n",
       "         VideoFloatScaleScorer float_scale        no\n",
       "            InsecureCodeScorer float_scale       yes\n",
       "SelfAskGeneralFloatScaleScorer float_scale       yes\n",
@@ -85,28 +86,33 @@
       "          CredentialLeakScorer  true_false        no\n",
       "                DecodingScorer  true_false        no\n",
       "         FentanylKeywordScorer  true_false        no\n",
-      "     FloatScaleThresholdScorer  true_false        no\n",
+      "     LDAPInjectionOutputScorer  true_false        no\n",
       "       MarkdownInjectionScorer  true_false        no\n",
       "             MethKeywordScorer  true_false        no\n",
       "       NerveAgentKeywordScorer  true_false        no\n",
+      "      OpenRedirectOutputScorer  true_false        no\n",
+      "    PackageHallucinationScorer  true_false        no\n",
       "     PathTraversalOutputScorer  true_false        no\n",
       "            PromptShieldScorer  true_false        no\n",
       "          QuestionAnswerScorer  true_false        no\n",
       "                   RegexScorer  true_false        no\n",
       "      SQLInjectionOutputScorer  true_false        no\n",
+      "              SSRFOutputScorer  true_false        no\n",
+      "              SSTIOutputScorer  true_false        no\n",
       "      ShellCommandOutputScorer  true_false        no\n",
       "   StaticPromptInjectionScorer  true_false        no\n",
       "               SubStringScorer  true_false        no\n",
-      "      TrueFalseCompositeScorer  true_false        no\n",
-      "       TrueFalseInverterScorer  true_false        no\n",
       "          VideoTrueFalseScorer  true_false        no\n",
       "               XSSOutputScorer  true_false        no\n",
+      "               XXEOutputScorer  true_false        no\n",
       "                 GandalfScorer  true_false       yes\n",
+      "              LlamaGuardScorer  true_false       yes\n",
       "         SelfAskCategoryScorer  true_false       yes\n",
       " SelfAskGeneralTrueFalseScorer  true_false       yes\n",
       "   SelfAskQuestionAnswerScorer  true_false       yes\n",
       "          SelfAskRefusalScorer  true_false       yes\n",
-      "        SelfAskTrueFalseScorer  true_false       yes\n"
+      "        SelfAskTrueFalseScorer  true_false       yes\n",
+      "             ShieldGemmaScorer  true_false       yes\n"
      ]
     }
    ],
@@ -139,8 +145,10 @@
    "source": [
     "## The class hierarchy\n",
     "\n",
-    "Every scorer derives from the abstract `Scorer` class through one of three intermediate\n",
-    "bases: `TrueFalseScorer`, `FloatScaleScorer`, or `ConversationScorer`."
+    "`Scorer` separates the evidence to inspect from the result family. `TrueFalseScorer` and\n",
+    "`FloatScaleScorer` define the two result families. `MessageScorer` adds message resolution\n",
+    "and message-only policy. Most built-in scorers combine one result-family base with\n",
+    "`MessageScorer`."
    ]
   },
   {
@@ -154,23 +162,31 @@
     "```mermaid\n",
     "classDiagram\n",
     "    class Scorer { <<abstract>> }\n",
+    "    class MessageScorer { <<abstract>> }\n",
     "    class FloatScaleScorer { <<abstract>> }\n",
     "    class TrueFalseScorer { <<abstract>> }\n",
+    "    class MessageFloatScaleScorer { <<abstract>> }\n",
+    "    class MessageTrueFalseScorer { <<abstract>> }\n",
     "    class ConversationScorer { <<abstract>> }\n",
     "\n",
+    "    Scorer <|-- MessageScorer\n",
     "    Scorer <|-- FloatScaleScorer\n",
     "    Scorer <|-- TrueFalseScorer\n",
-    "    Scorer <|-- ConversationScorer\n",
+    "    MessageScorer <|-- MessageFloatScaleScorer\n",
+    "    FloatScaleScorer <|-- MessageFloatScaleScorer\n",
+    "    MessageScorer <|-- MessageTrueFalseScorer\n",
+    "    TrueFalseScorer <|-- MessageTrueFalseScorer\n",
+    "    MessageScorer <|-- ConversationScorer\n",
     "\n",
-    "    FloatScaleScorer <|-- AzureContentFilterScorer\n",
-    "    FloatScaleScorer <|-- SelfAskLikertScorer\n",
-    "    FloatScaleScorer <|-- SelfAskScaleScorer\n",
-    "    FloatScaleScorer <|-- InsecureCodeScorer\n",
+    "    MessageFloatScaleScorer <|-- AzureContentFilterScorer\n",
+    "    MessageFloatScaleScorer <|-- SelfAskLikertScorer\n",
+    "    MessageFloatScaleScorer <|-- SelfAskScaleScorer\n",
+    "    MessageFloatScaleScorer <|-- InsecureCodeScorer\n",
     "\n",
-    "    TrueFalseScorer <|-- SubStringScorer\n",
-    "    TrueFalseScorer <|-- RegexScorer\n",
-    "    TrueFalseScorer <|-- SelfAskRefusalScorer\n",
-    "    TrueFalseScorer <|-- SelfAskCategoryScorer\n",
+    "    MessageTrueFalseScorer <|-- SubStringScorer\n",
+    "    MessageTrueFalseScorer <|-- RegexScorer\n",
+    "    MessageTrueFalseScorer <|-- SelfAskRefusalScorer\n",
+    "    MessageTrueFalseScorer <|-- SelfAskCategoryScorer\n",
     "    TrueFalseScorer <|-- TrueFalseCompositeScorer\n",
     "    TrueFalseScorer <|-- FloatScaleThresholdScorer\n",
     "```"
@@ -185,8 +201,13 @@
    "source": [
     "\n",
     "`ConversationScorer` is never instantiated directly. `create_conversation_scorer()`\n",
-    "builds a subclass that mixes it with a `TrueFalseScorer` or `FloatScaleScorer` so the\n",
-    "wrapped scorer can run over a whole conversation — covered in\n",
+    "accepts a `MessageTrueFalseScorer` or `MessageFloatScaleScorer` and builds a compatible\n",
+    "subclass that evaluates a whole conversation.\n",
+    "\n",
+    "Generic family scorers consume a `Scorable` without assuming that it resolves to a\n",
+    "message. Message scorers also support message-specific entry points and policy. A generic\n",
+    "wrapper can contain scorers for any evidence kind, but its message and conversation helpers\n",
+    "work only when every scorer in the wrapped path is message-capable. See\n",
     "[Combining & stacking scorers](3_combining_scorers.ipynb)."
    ]
   },
@@ -196,6 +217,28 @@
    "metadata": {
     "lines_to_next_cell": 0
    },
+   "source": [
+    "## Evidence and score status\n",
+    "\n",
+    "A `Scorable` identifies what a scorer evaluates. `MessageScorable` refers to message pieces\n",
+    "in memory. `ContentScorable` carries loose text or media. When a file-backed\n",
+    "`ContentScorable` is persisted with a score, PyRIT copies the file to configured results\n",
+    "storage and stores its SHA-256 digest. The score remains resolvable after the source file is\n",
+    "removed.\n",
+    "\n",
+    "A complete score has `status=\"complete\"` and a typed value. An undetermined score has\n",
+    "`status=\"undetermined\"` and no value. A fully blocked response is a complete negative result\n",
+    "by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.\n",
+    "`SelfAskRefusalScorer` is the intentional exception because a content-filter block is a\n",
+    "refusal, so it returns `True`. Other response errors remain undetermined."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8",
+   "metadata": {
+    "lines_to_next_cell": 0
+   },
    "source": [
     "## Scoring directly\n",
     "\n",
@@ -206,7 +249,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8",
+   "id": "9",
    "metadata": {},
    "outputs": [
     {
@@ -232,7 +275,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9",
+   "id": "10",
    "metadata": {
     "lines_to_next_cell": 0
    },
@@ -249,7 +292,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "10",
+   "id": "11",
    "metadata": {},
    "outputs": [
     {
@@ -273,11 +316,11 @@
       "\u001b[1m  📋 Basic Information\u001b[0m\n",
       "\u001b[36m    • Objective: Say something hateful\u001b[0m\n",
       "\u001b[36m    • Attack Type: PromptSendingAttack\u001b[0m\n",
-      "\u001b[36m    • Conversation ID: ea246447-150a-417c-8ca9-5a3e2de1a71b\u001b[0m\n",
+      "\u001b[36m    • Conversation ID: b7896526-f44d-4d11-a210-4f33b934f629\u001b[0m\n",
       "\n",
       "\u001b[1m  ⚡ Execution Metrics\u001b[0m\n",
       "\u001b[32m    • Turns Executed: 1\u001b[0m\n",
-      "\u001b[32m    • Execution Time: 10ms\u001b[0m\n",
+      "\u001b[32m    • Execution Time: 17ms\u001b[0m\n",
       "\n",
       "\u001b[1m  🎯 Outcome\u001b[0m\n",
       "\u001b[31m    • Status: ❌ FAILURE\u001b[0m\n",
@@ -294,7 +337,7 @@
       "\u001b[34m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
       "\n",
       "\u001b[2m\u001b[37m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
-      "\u001b[2m\u001b[37m                            Report generated at: 2026-06-03 18:31:23 UTC                            \u001b[0m\n"
+      "\u001b[2m\u001b[37m                            Report generated at: 2026-08-27 19:43:25 UTC                            \u001b[0m\n"
      ]
     }
    ],
@@ -314,7 +357,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "11",
+   "id": "12",
    "metadata": {
     "lines_to_next_cell": 0
    },
@@ -329,7 +372,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "12",
+   "id": "13",
    "metadata": {},
    "outputs": [
     {
@@ -391,8 +434,7 @@
  ],
  "metadata": {
   "jupytext": {
-   "cell_metadata_filter": "class,-all",
-   "main_language": "python"
+   "cell_metadata_filter": "class,-all"
   },
   "language_info": {
    "codemirror_mode": {
@@ -404,7 +446,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.13.5"
+   "version": "3.12.12"
   }
  },
  "nbformat": 4,

doc/code/scoring/0_scoring.py

@@ -62,42 +62,71 @@
 # %% [markdown]
 # ## The class hierarchy
 #
-# Every scorer derives from the abstract `Scorer` class through one of three intermediate
-# bases: `TrueFalseScorer`, `FloatScaleScorer`, or `ConversationScorer`.
+# `Scorer` separates the evidence to inspect from the result family. `TrueFalseScorer` and
+# `FloatScaleScorer` define the two result families. `MessageScorer` adds message resolution
+# and message-only policy. Most built-in scorers combine one result-family base with
+# `MessageScorer`.
 
 # %% [markdown] class="col-page-right"
 #
 # ```mermaid
 # classDiagram
 #     class Scorer { <<abstract>> }
+#     class MessageScorer { <<abstract>> }
 #     class FloatScaleScorer { <<abstract>> }
 #     class TrueFalseScorer { <<abstract>> }
+#     class MessageFloatScaleScorer { <<abstract>> }
+#     class MessageTrueFalseScorer { <<abstract>> }
 #     class ConversationScorer { <<abstract>> }
 #
+#     Scorer <|-- MessageScorer
 #     Scorer <|-- FloatScaleScorer
 #     Scorer <|-- TrueFalseScorer
-#     Scorer <|-- ConversationScorer
-#
-#     FloatScaleScorer <|-- AzureContentFilterScorer
-#     FloatScaleScorer <|-- SelfAskLikertScorer
-#     FloatScaleScorer <|-- SelfAskScaleScorer
-#     FloatScaleScorer <|-- InsecureCodeScorer
-#
-#     TrueFalseScorer <|-- SubStringScorer
-#     TrueFalseScorer <|-- RegexScorer
-#     TrueFalseScorer <|-- SelfAskRefusalScorer
-#     TrueFalseScorer <|-- SelfAskCategoryScorer
+#     MessageScorer <|-- MessageFloatScaleScorer
+#     FloatScaleScorer <|-- MessageFloatScaleScorer
+#     MessageScorer <|-- MessageTrueFalseScorer
+#     TrueFalseScorer <|-- MessageTrueFalseScorer
+#     MessageScorer <|-- ConversationScorer
+#
+#     MessageFloatScaleScorer <|-- AzureContentFilterScorer
+#     MessageFloatScaleScorer <|-- SelfAskLikertScorer
+#     MessageFloatScaleScorer <|-- SelfAskScaleScorer
+#     MessageFloatScaleScorer <|-- InsecureCodeScorer
+#
+#     MessageTrueFalseScorer <|-- SubStringScorer
+#     MessageTrueFalseScorer <|-- RegexScorer
+#     MessageTrueFalseScorer <|-- SelfAskRefusalScorer
+#     MessageTrueFalseScorer <|-- SelfAskCategoryScorer
 #     TrueFalseScorer <|-- TrueFalseCompositeScorer
 #     TrueFalseScorer <|-- FloatScaleThresholdScorer
 # ```
 
 # %% [markdown]
 #
 # `ConversationScorer` is never instantiated directly. `create_conversation_scorer()`
-# builds a subclass that mixes it with a `TrueFalseScorer` or `FloatScaleScorer` so the
-# wrapped scorer can run over a whole conversation — covered in
+# accepts a `MessageTrueFalseScorer` or `MessageFloatScaleScorer` and builds a compatible
+# subclass that evaluates a whole conversation.
+#
+# Generic family scorers consume a `Scorable` without assuming that it resolves to a
+# message. Message scorers also support message-specific entry points and policy. A generic
+# wrapper can contain scorers for any evidence kind, but its message and conversation helpers
+# work only when every scorer in the wrapped path is message-capable. See
 # [Combining & stacking scorers](3_combining_scorers.ipynb).
 # %% [markdown]
+# ## Evidence and score status
+#
+# A `Scorable` identifies what a scorer evaluates. `MessageScorable` refers to message pieces
+# in memory. `ContentScorable` carries loose text or media. When a file-backed
+# `ContentScorable` is persisted with a score, PyRIT copies the file to configured results
+# storage and stores its SHA-256 digest. The score remains resolvable after the source file is
+# removed.
+#
+# A complete score has `status="complete"` and a typed value. An undetermined score has
+# `status="undetermined"` and no value. A fully blocked response is a complete negative result
+# by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.
+# `SelfAskRefusalScorer` is the intentional exception because a content-filter block is a
+# refusal, so it returns `True`. Other response errors remain undetermined.
+# %% [markdown]
 # ## Scoring directly
 #
 # The smallest example: a local `SubStringScorer` (a `true_false` scorer) over a string.