FEAT: Adding Garak prompt injection scenario

doc/code/datasets/1_loading_datasets.ipynb

@@ -64,7 +64,8 @@
     "(`garak_pypi_packages`, `garak_npm_packages`, `garak_crates_packages`,\n",
     "`garak_rubygems_packages`, `garak_dart_packages`, `garak_perl_packages`,\n",
     "`garak_raku_packages`), system-prompt libraries (`garak_drh_system_prompts`,\n",
-    "`garak_tm_system_prompts`), an audio jailbreak set\n",
+    "`garak_tm_system_prompts`), PromptInject context and technique templates\n",
+    "(`prompt_inject_contexts`, `prompt_inject_techniques`), an audio jailbreak set\n",
     "(`garak_audio_achilles_heel`), and visual jailbreak sets (`figstep`, `figstep_pro`)."
    ]
   },
@@ -157,6 +158,8 @@
        " 'or_bench_hard',\n",
        " 'or_bench_toxic',\n",
        " 'pku_safe_rlhf',\n",
+       " 'prompt_inject_contexts',\n",
+       " 'prompt_inject_techniques',\n",
        " 'promptintel',\n",
        " 'psfuzz_steal_system_prompt',\n",
        " 'pyrit_example_dataset',\n",

doc/code/datasets/1_loading_datasets.py

@@ -68,7 +68,8 @@
 # (`garak_pypi_packages`, `garak_npm_packages`, `garak_crates_packages`,
 # `garak_rubygems_packages`, `garak_dart_packages`, `garak_perl_packages`,
 # `garak_raku_packages`), system-prompt libraries (`garak_drh_system_prompts`,
-# `garak_tm_system_prompts`), an audio jailbreak set
+# `garak_tm_system_prompts`), PromptInject context and technique templates
+# (`prompt_inject_contexts`, `prompt_inject_techniques`), an audio jailbreak set
 # (`garak_audio_achilles_heel`), and visual jailbreak sets (`figstep`, `figstep_pro`).
 
 # %%

doc/scanner/garak.ipynb

@@ -10,7 +10,8 @@
     "The Garak scenario family implements probes inspired by the\n",
     "[Garak](https://github.com/NVIDIA/garak) framework. These include encoding-based probes (which\n",
     "test whether a target can be tricked into producing harmful content when prompts are encoded in\n",
-    "various formats), web-injection probes (which test whether a target emits markdown\n",
+    "various formats), prompt-injection probes (which embed override commands in benign tasks),\n",
+    "web-injection probes (which test whether a target emits markdown\n",
     "data-exfiltration or cross-site-scripting payloads), a doctor probe (which applies the Policy\n",
     "Puppetry universal bypass), system-prompt-extraction probes (which test whether a target can be\n",
     "coaxed into revealing its own system prompt), package-hallucination probes (which test whether a\n",
@@ -22,10 +23,35 @@
     "[Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb)."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## PromptInject\n",
+    "\n",
+    "PromptInject tests whether a target follows injected instructions hidden in benign tasks.\n",
+    "`--techniques` selects the injection wording, and `--goal-texts` selects the requested output.\n",
+    "The default run uses all five techniques, the three Garak goal texts, and up to 60 attempts.\n",
+    "\n",
+    "**CLI examples:**\n",
+    "\n",
+    "```bash\n",
+    "pyrit_scan run garak.prompt_inject --target openai_chat\n",
+    "pyrit_scan run garak.prompt_inject --target openai_chat \\\n",
+    "  --techniques ignore_print \\\n",
+    "  --goal-texts \"Custom trigger text\" \\\n",
+    "  --max-dataset-size 16\n",
+    "```\n",
+    "\n",
+    "**Techniques:** `ignore_print`, `ignore_say`, `ignore_print_upper`, `nevermind`, and\n",
+    "`screaming_stop`."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1",
+   "id": "2",
    "metadata": {
     "lines_to_next_cell": 0
    },
@@ -55,7 +81,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2",
+   "id": "3",
    "metadata": {},
    "source": [
     "## Encoding\n",
@@ -86,7 +112,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3",
+   "id": "4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -111,7 +137,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "4",
+   "id": "5",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -120,7 +146,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5",
+   "id": "6",
    "metadata": {},
    "source": [
     "## FigStep\n",
@@ -148,7 +174,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6",
+   "id": "7",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -173,7 +199,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7",
+   "id": "8",
    "metadata": {},
    "source": [
     "## WebInjection\n",
@@ -199,7 +225,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8",
+   "id": "9",
    "metadata": {},
    "source": [
     "## Doctor\n",
@@ -223,7 +249,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9",
+   "id": "10",
    "metadata": {},
    "source": [
     "## SystemPromptExtraction\n",
@@ -256,7 +282,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "10",
+   "id": "11",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -278,7 +304,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "11",
+   "id": "12",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -287,7 +313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12",
+   "id": "13",
    "metadata": {},
    "source": [
     "## PackageHallucination\n",
@@ -322,7 +348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "13",
+   "id": "14",
    "metadata": {},
    "source": [
     "## AudioAchillesHeel\n",
@@ -350,7 +376,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "14",
+   "id": "15",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -378,7 +404,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "15",
+   "id": "16",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -387,7 +413,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16",
+   "id": "17",
    "metadata": {},
    "source": [
     "For more details, see the [Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb) and\n",

doc/scanner/garak.py

@@ -14,7 +14,8 @@
 # The Garak scenario family implements probes inspired by the
 # [Garak](https://github.com/NVIDIA/garak) framework. These include encoding-based probes (which
 # test whether a target can be tricked into producing harmful content when prompts are encoded in
-# various formats), web-injection probes (which test whether a target emits markdown
+# various formats), prompt-injection probes (which embed override commands in benign tasks),
+# web-injection probes (which test whether a target emits markdown
 # data-exfiltration or cross-site-scripting payloads), a doctor probe (which applies the Policy
 # Puppetry universal bypass), system-prompt-extraction probes (which test whether a target can be
 # coaxed into revealing its own system prompt), package-hallucination probes (which test whether a
@@ -25,6 +26,26 @@
 # For full programming details, see the
 # [Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb).
 
+# %% [markdown]
+# ## PromptInject
+#
+# PromptInject tests whether a target follows injected instructions hidden in benign tasks.
+# `--techniques` selects the injection wording, and `--goal-texts` selects the requested output.
+# The default run uses all five techniques, the three Garak goal texts, and up to 60 attempts.
+#
+# **CLI examples:**
+#
+# ```bash
+# pyrit_scan run garak.prompt_inject --target openai_chat
+# pyrit_scan run garak.prompt_inject --target openai_chat \
+#   --techniques ignore_print \
+#   --goal-texts "Custom trigger text" \
+#   --max-dataset-size 16
+# ```
+#
+# **Techniques:** `ignore_print`, `ignore_say`, `ignore_print_upper`, `nevermind`, and
+# `screaming_stop`.
+
 # %%
 from pathlib import Path