From d693f749fa124f66378f1f2d74678036f6a484a7 Mon Sep 17 00:00:00 2001 From: Shaishav Pidadi Date: Tue, 11 Aug 2026 20:49:59 -0400 Subject: [PATCH 1/4] fix(pii): redact names and addresses, unpin analyzer language, harden tool denylist Adds bench/, a 438-item span-annotated corpus across en/es/fr/de/zh that measures the deployed redaction path rather than the detector in isolation, and fixes the four defects it surfaced. Overall leakage 51.3% -> 8.2%. High-sensitivity entity leakage 89.1% -> 10.6%. PERSON and LOCATION 100% -> ~11.6%. Over-redaction stays at 0% and p50 latency is unchanged (2.46 -> 2.48 ms). Entity allowlist: the analyzer was asked only for the 16 keys of ANONYMIZE_OPERATORS, which contain no PERSON, LOCATION or NRP. spaCy detected names at score 0.85 and the pipeline discarded them, so names and addresses were never redacted in any language. Replaced with an explicit DETECT_ENTITIES list. ORGANIZATION and DATE_TIME stay out: including them modified 61.5% of PII-free control text, and a layer that redacts everything gets switched off. Language pin: analyze() was called with language="en" at all three sites and the engine was built with supported_languages=["en"], so the es/fr/de/zh models the Dockerfile installs were unreachable. Adds PRESIDIO_LANGUAGES, PRESIDIO_LANGUAGE_MODE, a per-request hint via tool_config.metadata.language, and a shared analyze_text() so the three sites cannot drift apart again. Per-language recognizer gaps: Presidio ships CreditCardRecognizer for en/es only and its default phone regions exclude ES/FR/CN, so a hinted fr/de/zh request leaked 61.5% of card numbers. Both now registered per language. False-positive filter: is_false_positive() received the whole input text rather than the matched span, so its rules almost never fired. The US_SSN recognizer also passed deny_list=["password", "key", ...], but Presidio's deny_list is a list of terms to detect, not suppress, so the literal word "key" matched as an SSN. Removed; suppression stays in is_false_positive(). Denylist matching: `tool in deny_tools` was an exact string test that Python.Exec, python_exec, exec.python and agent.python.exec.v2 all walked past. Replaced with is_denied_tool() (normalisation, component-set match, containment, namespace.* wildcards) and extended the defaults with subprocess.run, os.system and the other direct execution primitives. Tests: +69. Suite 258 -> 327 passing, coverage 75.1% -> 76.8%. --- PROJECT_SPECS.md | 63 ++- app/policies.py | 621 +++++++++++++++++++++++------- app/settings.py | 35 +- bench/README.md | 92 +++++ bench/__init__.py | 0 bench/corpus.py | 472 +++++++++++++++++++++++ bench/run.py | 335 ++++++++++++++++ env.example | 18 + tests/test_deny_tools_matching.py | 113 ++++++ tests/test_multilingual_pii.py | 153 ++++++++ 10 files changed, 1753 insertions(+), 149 deletions(-) create mode 100644 bench/README.md create mode 100644 bench/__init__.py create mode 100644 bench/corpus.py create mode 100644 bench/run.py create mode 100644 tests/test_deny_tools_matching.py create mode 100644 tests/test_multilingual_pii.py diff --git a/PROJECT_SPECS.md b/PROJECT_SPECS.md index 1966084..8f08b6d 100644 --- a/PROJECT_SPECS.md +++ b/PROJECT_SPECS.md @@ -24,8 +24,21 @@ GovernsAI Precheck is a policy evaluation and PII redaction service that provide ### 3. PII Detection & Redaction - **Presidio integration**: Advanced NLP-based PII detection - **Fallback detection**: Regex-based detection when Presidio unavailable -- **Multiple PII types**: Email, SSN, phone numbers, credit cards, API keys, JWT tokens -- **False positive filtering**: Context-aware filtering to reduce false positives +- **Multiple PII types**: names, addresses, email, SSN, phone numbers, credit cards, IBAN, IP addresses, HIPAA PHI identifiers, PCI fields, API keys, JWT tokens +- **Multilingual**: analyzer built per configured language (`PRESIDIO_LANGUAGES`); en, es, fr, de, zh models ship in the image +- **False positive filtering**: context-aware filtering over the matched span +- **Measured**: `bench/` reports leakage, over-redaction and latency per language, tier and entity type — see `bench/README.md` + +**Entity coverage.** `DETECT_ENTITIES` in `app/policies.py` is the list the +analyzer is asked for. `ORGANIZATION` and `DATE_TIME` are deliberately excluded: +both are recognised by the spaCy pipeline but neither identifies a person, and +including them redacted 61.5% of PII-free control text. Utility loss on that +scale makes operators disable redaction outright, which is worse than leaving +those entities in place. + +**Known residual gaps** (measured, unfixed): obfuscated surface forms 75% +leakage, partial disclosure 83.3%, Chinese 32.1%, US_SSN 14.3%. Closing the +first two needs a different class of detector, not a better pattern. ### 4. Event Emission & Monitoring - **Webhook events**: Real-time policy decision events with HMAC authentication @@ -591,10 +604,16 @@ The policy evaluation system follows a strict precedence hierarchy (highest to l ### 1. **DENY_TOOLS** (Highest Priority) - **Purpose**: Hard deny for dangerous tools -- **Tools**: `python.exec`, `bash.exec`, `code.exec`, `shell.exec` +- **Tools**: `python.exec`, `bash.exec`, `code.exec`, `shell.exec`, `subprocess.run`, `subprocess.popen`, `os.system`, `child_process.exec`, `child_process.spawn`, `runtime.exec` - **Decision**: Always `deny` - **Policy ID**: `deny-exec` - **Reason**: `blocked tool: code/exec` +- **Matching**: `is_denied_tool()` normalises case, whitespace and separators + (`_`, `-`, `/`, `:` all fold to `.`), then matches on exact name, component + set (so `exec.python` matches `python.exec`), containment within a namespaced + path (`agent.python.exec.v2`), and explicit `namespace.*` wildcards. Denial is + a security boundary, so matching is deliberately generous: a false deny is a + support ticket, a false allow is an incident. ### 2. **TOOL_SPECIFIC** (High Priority) - **Purpose**: Tool-specific rules in `policy.tool_access.yaml` @@ -931,7 +950,9 @@ Budget limits can be configured per user: | `ON_ERROR` | Error handling behavior | `block` | | `POLICY_FILE` | Policy file path | `policy.tool_access.yaml` | | `USE_PRESIDIO` | Enable Presidio PII detection | `true` | -| `PRESIDIO_MODEL` | spaCy model for Presidio | `en_core_web_sm` | +| `PRESIDIO_MODEL` | English spaCy model for Presidio | `en_core_web_sm` | +| `PRESIDIO_LANGUAGES` | Comma-separated analyzer languages; each needs its spaCy model installed | `en` | +| `PRESIDIO_LANGUAGE_MODE` | `hint` (one pass, caller-supplied language) or `union` (all configured languages, recall-safe, ~N× NLP cost and higher false positives) | `hint` | ### API Configuration @@ -1171,6 +1192,40 @@ curl -X POST http://localhost:8080/api/v1/postcheck \ ``` ## Recent Changes Log +- **2026-08-11**: **PII detection benchmark + four detection fixes.** Added `bench/`, a + 438-item span-annotated corpus across five languages measuring the deployed + redaction path rather than the detector in isolation. Overall leakage fell + **51.3% → 8.2%**; high-sensitivity entity leakage **89.1% → 10.6%**; PERSON and + LOCATION **100% → ~11.6%**; over-redaction stayed at 0% and p50 latency was + unchanged (2.46 → 2.48 ms). + - **Entity allowlist**: the analyzer was asked only for the 16 keys of + `ANONYMIZE_OPERATORS`, which contain no `PERSON`, `LOCATION` or `NRP`. The + spaCy model detected names at score 0.85 and the pipeline then discarded + them — names and addresses were never redacted, in any language. Replaced + with an explicit `DETECT_ENTITIES` list; `ORGANIZATION` and `DATE_TIME` + stay excluded to protect utility. + - **Language pin**: `ANALYZER.analyze(..., language="en")` was hardcoded at all + three call sites and `AnalyzerEngine` was built with `supported_languages=["en"]`, + so the es/fr/de/zh models the Dockerfile installs were unreachable. Added + `PRESIDIO_LANGUAGES`, `PRESIDIO_LANGUAGE_MODE`, per-request language hints via + `tool_config.metadata.language`, and a shared `analyze_text()` so the three + sites cannot drift. + - **Per-language recognizer gaps**: Presidio registers `CreditCardRecognizer` + for en/es only, and its default phone regions exclude ES/FR/CN. A hinted + fr/de/zh request leaked 61.5% of card numbers. Both now registered per + configured language. + - **False-positive filter**: `is_false_positive()` was receiving the whole input + text instead of the matched span, so its suppression rules almost never + fired. The `US_SSN` recognizer also passed `deny_list=["password", "key", ...]`, + but Presidio's `deny_list` is a list of terms to *detect* — the literal word + "key" matched as an SSN. Removed; suppression lives in `is_false_positive()`. + - **Denylist matching**: `if tool in deny_tools` was an exact string test. + `Python.Exec`, `python_exec`, `exec.python` and `agent.python.exec.v2` all + walked past it. Replaced with `is_denied_tool()` (normalisation, component-set, + containment, `namespace.*` wildcards) and extended the defaults with + `subprocess.run`, `os.system` and other direct execution primitives. + - **Tests**: +69 (`test_deny_tools_matching.py`, `test_multilingual_pii.py`); + suite 258 → 327 passing, coverage 75.1% → 76.8%. - **2026-04-23**: Added the Mode 2 sidecar / proxy gateway design document at `docs/design/sidecar-mode.md` - **Language Decision**: Recommends Go for the proxy hot path over Node.js and Python - **Interception Model**: Defines `POST /v1/chat/completions` interception with `precheck` before upstream forwarding diff --git a/app/policies.py b/app/policies.py index 73093c2..1a2aee6 100644 --- a/app/policies.py +++ b/app/policies.py @@ -1,4 +1,5 @@ import hashlib +import importlib.util import os import re import time @@ -13,6 +14,10 @@ RecognizerRegistry, ) from presidio_analyzer.nlp_engine import SpacyNlpEngine +from presidio_analyzer.predefined_recognizers import ( + CreditCardRecognizer, + PhoneRecognizer, +) from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import OperatorConfig @@ -114,158 +119,288 @@ def _replace_regex(s: str, pattern: re.Pattern, placeholder: str) -> str: USE_PRESIDIO = settings.use_presidio if hasattr(settings, "use_presidio") else True -def build_presidio(): - """Initialize Presidio analyzer and anonymizer with custom recognizers""" - try: - # Initialize spaCy NLP engine with configured model and load it - model_name = getattr(settings, "presidio_model", "en_core_web_sm") - # Presidio 2.x expects a list of {lang_code, model_name} - nlp_engine = SpacyNlpEngine( - models=[{"lang_code": "en", "model_name": model_name}] - ) - nlp_engine.load() - registry = RecognizerRegistry() - registry.load_predefined_recognizers(nlp_engine=nlp_engine) - - # Custom API key recognizer - api_key_pattern = Pattern( - name="API_KEY", - regex=r"(?:sk|pk|AKIA|ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{16,40}", - score=0.6, - ) - api_key_recognizer = PatternRecognizer( +# spaCy model used for each supported language. The English entry is +# overridable via PRESIDIO_MODEL (sm/md/lg); the rest track the models the +# Dockerfile installs. +DEFAULT_SPACY_MODELS = { + "en": "en_core_web_sm", + "es": "es_core_news_sm", + "fr": "fr_core_news_sm", + "de": "de_core_news_sm", + "zh": "zh_core_web_sm", +} + +# phonenumbers regions to try per configured language. Presidio's default region +# set is US/UK/DE/FE/IL/IN/CA/BR, which silently misses ES, FR and CN numbers +# even when the matching spaCy model is loaded. +PHONE_REGIONS_BY_LANGUAGE = { + "en": ["US", "UK", "CA", "IN", "AU"], + "es": ["ES", "MX", "AR"], + "fr": ["FR", "BE", "CA"], + "de": ["DE", "AT", "CH"], + "zh": ["CN", "HK", "TW"], +} + +# Languages the analyzer was actually built for. Populated by build_presidio; +# a configured language whose spaCy model is missing is dropped from this list +# rather than taking the whole service down. +SUPPORTED_LANGUAGES: List[str] = ["en"] + + +def _model_available(model_name: str) -> bool: + return importlib.util.find_spec(model_name) is not None + + +def _resolve_language_models() -> List[Dict[str, str]]: + """Build the {lang_code, model_name} list for the configured languages.""" + configured = settings.presidio_language_list() + models: List[Dict[str, str]] = [] + for lang in configured: + if lang == "en": + model_name = getattr(settings, "presidio_model", "en_core_web_sm") + else: + model_name = DEFAULT_SPACY_MODELS.get(lang) + if not model_name: + print(f"Presidio: no spaCy model mapped for language {lang!r}; skipping") + continue + if not _model_available(model_name): + print( + f"Presidio: spaCy model {model_name!r} for language {lang!r} is not " + "installed; skipping that language" + ) + continue + models.append({"lang_code": lang, "model_name": model_name}) + + if not models: + models = [{"lang_code": "en", "model_name": "en_core_web_sm"}] + return models + + +def _custom_recognizers(language: str) -> List[PatternRecognizer]: + """Pattern recognizers registered for every supported language. + + Presidio scopes each recognizer to one language, so the same set is built + per language rather than once globally — otherwise a Spanish request gets + the spaCy NER model but none of the SSN/PHI/PCI patterns. + """ + recognizers: List[PatternRecognizer] = [] + + # API keys. Underscore is optional so bare AWS-style keys (AKIA...) match, + # and vendor-prefixed keys (sk_live_..., pk_test_...) match through their + # internal separators. + recognizers.append( + PatternRecognizer( supported_entity="API_KEY", - patterns=[api_key_pattern], + supported_language=language, + patterns=[ + Pattern( + name="API_KEY_VENDOR_PREFIXED", + regex=r"\b(?:sk|pk|rk)_(?:live|test)_[A-Za-z0-9]{10,40}\b", + score=0.75, + ), + Pattern( + name="API_KEY", + regex=r"\b(?:sk|pk|AKIA|ghp|gho|ghu|ghs|ghr)_?[A-Za-z0-9]{16,40}\b", + score=0.6, + ), + ], context=["secret", "token", "apikey", "api_key", "bearer", "key"], ) - registry.add_recognizer(api_key_recognizer) + ) - # JWT token recognizer - jwt_pattern = Pattern( - name="JWT_TOKEN", - regex=r"eyJ[A-Za-z0-9_-]*\.eyJ[A-Za-z0-9_-]*\.[A-Za-z0-9_-]*", - score=0.8, - ) - jwt_recognizer = PatternRecognizer( + recognizers.append( + PatternRecognizer( supported_entity="JWT_TOKEN", - patterns=[jwt_pattern], + supported_language=language, + patterns=[ + Pattern( + name="JWT_TOKEN", + regex=r"eyJ[A-Za-z0-9_-]*\.eyJ[A-Za-z0-9_-]*\.[A-Za-z0-9_-]*", + score=0.8, + ) + ], context=["token", "jwt", "bearer", "authorization"], ) - registry.add_recognizer(jwt_recognizer) + ) - # Override SSN recognizer to be more context-aware and exclude passwords - ssn_pattern = Pattern( - name="US_SSN", - regex=r"\b(?!000|666|9\d{2})\d{3}[-]?(?!00)\d{2}[-]?(?!0000)\d{4}\b", - score=0.8, - ) - ssn_recognizer = PatternRecognizer( + # Override SSN recognizer to be more context-aware and exclude passwords + recognizers.append( + PatternRecognizer( supported_entity="US_SSN", - patterns=[ssn_pattern], + supported_language=language, + patterns=[ + Pattern( + name="US_SSN", + regex=r"\b(?!000|666|9\d{2})\d{3}[-]?(?!00)\d{2}[-]?(?!0000)\d{4}\b", + score=0.8, + ) + ], context=["ssn", "social", "security", "tax", "id", "number"], - deny_list=["password", "pass", "pwd", "secret", "key", "token"], + # No deny_list here. Presidio's deny_list is a list of terms to + # *detect*, not to suppress — passing ["password", "key", ...] made + # the literal word "key" match as US_SSN. Suppression belongs in + # is_false_positive(), which runs over the matched span below. ) - registry.add_recognizer(ssn_recognizer) - - # HIPAA PHI recognizers - registry.add_recognizer( - PatternRecognizer( - supported_entity="US_MEDICAL_RECORD_NUMBER", - patterns=[ - Pattern( - name="US_MEDICAL_RECORD_NUMBER", - regex=r"\b(?:MRN|Medical\s*Record(?:\s*Number)?|Patient\s*ID)\s*[:#]?\s*[A-Z0-9\-]{6,20}\b", - score=0.82, - ) - ], - context=["mrn", "medical record", "patient id", "chart"], - ) + ) + + # HIPAA PHI recognizers + recognizers.append( + PatternRecognizer( + supported_entity="US_MEDICAL_RECORD_NUMBER", + supported_language=language, + patterns=[ + Pattern( + name="US_MEDICAL_RECORD_NUMBER", + regex=r"\b(?:MRN|Medical\s*Record(?:\s*Number)?|Patient\s*ID)\s*[:#]?\s*[A-Z0-9\-]{6,20}\b", + score=0.82, + ) + ], + context=["mrn", "medical record", "patient id", "chart"], ) - registry.add_recognizer( - PatternRecognizer( - supported_entity="US_HEALTH_MEMBER_ID", - patterns=[ - Pattern( - name="US_HEALTH_MEMBER_ID", - regex=r"\b(?:Member|Policy|Insurance)\s*(?:ID|Number)\s*[:#]?\s*[A-Z0-9\-]{6,24}\b", - score=0.8, - ) - ], - context=["member id", "policy", "insurance", "payer"], - ) + ) + recognizers.append( + PatternRecognizer( + supported_entity="US_HEALTH_MEMBER_ID", + supported_language=language, + patterns=[ + Pattern( + name="US_HEALTH_MEMBER_ID", + regex=r"\b(?:Member|Policy|Insurance)\s*(?:ID|Number)\s*[:#]?\s*[A-Z0-9\-]{6,24}\b", + score=0.8, + ) + ], + context=["member id", "policy", "insurance", "payer"], ) - registry.add_recognizer( - PatternRecognizer( - supported_entity="US_NPI", - patterns=[ - Pattern( - name="US_NPI", - regex=r"\b(?:NPI|National\s*Provider\s*Identifier)\s*[:#]?\s*\d{10}\b", - score=0.85, - ) - ], - context=["npi", "provider", "national provider identifier"], - ) + ) + recognizers.append( + PatternRecognizer( + supported_entity="US_NPI", + supported_language=language, + patterns=[ + Pattern( + name="US_NPI", + regex=r"\b(?:NPI|National\s*Provider\s*Identifier)\s*[:#]?\s*\d{10}\b", + score=0.85, + ) + ], + context=["npi", "provider", "national provider identifier"], ) - registry.add_recognizer( - PatternRecognizer( - supported_entity="US_DEA", - patterns=[ - Pattern( - name="US_DEA", - regex=r"\b(?:DEA|DEA\s*Number)\s*[:#]?\s*[A-Z]{2}\d{7}\b", - score=0.85, - ) - ], - context=["dea", "prescriber", "controlled substance"], - ) + ) + recognizers.append( + PatternRecognizer( + supported_entity="US_DEA", + supported_language=language, + patterns=[ + Pattern( + name="US_DEA", + regex=r"\b(?:DEA|DEA\s*Number)\s*[:#]?\s*[A-Z]{2}\d{7}\b", + score=0.85, + ) + ], + context=["dea", "prescriber", "controlled substance"], ) - registry.add_recognizer( - PatternRecognizer( - supported_entity="US_DATE_OF_BIRTH", - patterns=[ - Pattern( - name="US_DATE_OF_BIRTH", - regex=r"\b(?:DOB|Date\s*of\s*Birth)\s*[:#]?\s*(?:0?[1-9]|1[0-2])[/-](?:0?[1-9]|[12][0-9]|3[01])[/-](?:19|20)?\d{2}\b", - score=0.78, - ) - ], - context=["dob", "date of birth", "patient"], - ) + ) + recognizers.append( + PatternRecognizer( + supported_entity="US_DATE_OF_BIRTH", + supported_language=language, + patterns=[ + Pattern( + name="US_DATE_OF_BIRTH", + regex=r"\b(?:DOB|Date\s*of\s*Birth)\s*[:#]?\s*(?:0?[1-9]|1[0-2])[/-](?:0?[1-9]|[12][0-9]|3[01])[/-](?:19|20)?\d{2}\b", + score=0.78, + ) + ], + context=["dob", "date of birth", "patient"], ) + ) - # PCI-DSS related recognizers - registry.add_recognizer( - PatternRecognizer( - supported_entity="PCI_CVV", - patterns=[ - Pattern( - name="PCI_CVV", - regex=r"\b(?:cvv|cvc|cvn|security\s*code)\s*[:#]?\s*\d{3,4}\b", - score=0.88, - ) - ], - context=["cvv", "cvc", "security code", "payment"], - ) + # PCI-DSS related recognizers + recognizers.append( + PatternRecognizer( + supported_entity="PCI_CVV", + supported_language=language, + patterns=[ + Pattern( + name="PCI_CVV", + regex=r"\b(?:cvv|cvc|cvn|security\s*code)\s*[:#]?\s*\d{3,4}\b", + score=0.88, + ) + ], + context=["cvv", "cvc", "security code", "payment"], ) - registry.add_recognizer( - PatternRecognizer( - supported_entity="PCI_EXPIRY", - patterns=[ - Pattern( - name="PCI_EXPIRY", - regex=r"\b(?:exp(?:iry|iration)?|valid\s*thru)\s*[:#]?\s*(?:0[1-9]|1[0-2])[/-](?:\d{2}|\d{4})\b", - score=0.84, - ) - ], - context=["expiry", "expiration", "valid thru", "payment"], - ) + ) + recognizers.append( + PatternRecognizer( + supported_entity="PCI_EXPIRY", + supported_language=language, + patterns=[ + Pattern( + name="PCI_EXPIRY", + regex=r"\b(?:exp(?:iry|iration)?|valid\s*thru)\s*[:#]?\s*(?:0[1-9]|1[0-2])[/-](?:\d{2}|\d{4})\b", + score=0.84, + ) + ], + context=["expiry", "expiration", "valid thru", "payment"], ) + ) + + return recognizers + + +def build_presidio(): + """Initialize Presidio analyzer and anonymizer with custom recognizers. + + Builds one NLP pipeline per configured language (PRESIDIO_LANGUAGES) and + registers both the predefined and the custom recognizers for each of them. + """ + global SUPPORTED_LANGUAGES + try: + models = _resolve_language_models() + languages = [m["lang_code"] for m in models] + + # Presidio 2.x expects a list of {lang_code, model_name} + nlp_engine = SpacyNlpEngine(models=models) + nlp_engine.load() + + # The registry carries its own language list and Presidio rejects an + # analyzer whose languages differ from it; constructing the registry + # without this is what silently drops everything back to English. + registry = RecognizerRegistry(supported_languages=languages) + registry.load_predefined_recognizers( + languages=languages, nlp_engine=nlp_engine + ) + + for language in languages: + for recognizer in _custom_recognizers(language): + registry.add_recognizer(recognizer) + + regions = PHONE_REGIONS_BY_LANGUAGE.get(language) + if regions: + registry.add_recognizer( + PhoneRecognizer( + supported_language=language, + supported_regions=tuple(regions), + ) + ) + + # Presidio ships CreditCardRecognizer for en/es only. A card number + # is language-independent — Luhn does not care what language the + # sentence around it is in — so register it everywhere. Without + # this, a hinted fr/de/zh request leaks 61.5% of card numbers + # (measured in bench/). + if "CREDIT_CARD" not in registry.get_supported_entities([language]): + registry.add_recognizer( + CreditCardRecognizer(supported_language=language) + ) analyzer = AnalyzerEngine( - registry=registry, nlp_engine=nlp_engine, supported_languages=["en"] + registry=registry, + nlp_engine=nlp_engine, + supported_languages=languages, ) anonymizer = AnonymizerEngine() + SUPPORTED_LANGUAGES = languages return analyzer, anonymizer except Exception as e: print(f"Failed to initialize Presidio: {e}") @@ -315,8 +450,42 @@ def init_presidio(): "PCI_EXPIRY": OperatorConfig("replace", {"new_value": ""}), "API_KEY": OperatorConfig("replace", {"new_value": "[REDACTED_API_KEY]"}), "JWT_TOKEN": OperatorConfig("replace", {"new_value": "[REDACTED_JWT]"}), + "PERSON": OperatorConfig("replace", {"new_value": ""}), + "LOCATION": OperatorConfig("replace", {"new_value": ""}), + "NRP": OperatorConfig("replace", {"new_value": ""}), } +# Entity types the analyzer is asked for. Kept separate from +# ANONYMIZE_OPERATORS because that mapping carries a "DEFAULT" key which is not +# an entity type, and because the two lists have different reasons to change. +# +# ORGANIZATION and DATE_TIME are deliberately excluded. Both are recognised by +# the spaCy pipeline but neither identifies a person on its own, and including +# them redacted 61.5% of PII-free control text in bench/ — company names, +# "quarterly", and "UTC" all match. Utility loss on that scale makes operators +# turn redaction off entirely, which is a worse outcome than the entities being +# left in place. +DETECT_ENTITIES = [ + "PERSON", + "LOCATION", + "NRP", + "EMAIL_ADDRESS", + "PHONE_NUMBER", + "CREDIT_CARD", + "IBAN_CODE", + "IP_ADDRESS", + "US_SSN", + "US_MEDICAL_RECORD_NUMBER", + "US_HEALTH_MEMBER_ID", + "US_NPI", + "US_DEA", + "US_DATE_OF_BIRTH", + "PCI_CVV", + "PCI_EXPIRY", + "API_KEY", + "JWT_TOKEN", +] + def entity_type_to_placeholder(entity_type: str) -> str: """Convert Presidio entity type to descriptive placeholder""" @@ -336,26 +505,119 @@ def entity_type_to_placeholder(entity_type: str) -> str: "IBAN_CODE": "", "API_KEY": "", "JWT_TOKEN": "", + "PERSON": "", + "LOCATION": "", + "NRP": "", } return entity_mapping.get(entity_type, f"") +def _languages_to_analyze(language: Optional[str]) -> List[str]: + """Which language pipelines to run for one request. + + A caller-supplied language always wins when it is supported. Otherwise the + behaviour depends on PRESIDIO_LANGUAGE_MODE: "hint" analyses in the first + configured language only, "union" analyses in all of them. + """ + supported = SUPPORTED_LANGUAGES or ["en"] + if language: + code = language.strip().lower() + if code in supported: + return [code] + + mode = getattr(settings, "presidio_language_mode", "hint") + if mode == "union": + return list(supported) + return supported[:1] + + +def _merge_results(results: List[Any]) -> List[Any]: + """Drop lower-confidence findings that overlap a higher-confidence one. + + Union mode analyses the same text several times, so the same span comes + back once per language. Presidio's anonymizer tolerates duplicates but + reason codes and counts would double, so collapse them here. + """ + ordered = sorted(results, key=lambda r: (-r.score, r.start, r.end)) + kept: List[Any] = [] + for candidate in ordered: + overlaps = any( + candidate.start < existing.end and existing.start < candidate.end + for existing in kept + ) + if not overlaps: + kept.append(candidate) + return sorted(kept, key=lambda r: r.start) + + +def language_hint( + tool_config: Optional[Dict] = None, policy_config: Optional[Dict] = None +) -> Optional[str]: + """Extract the caller's language hint, if any. + + SDK callers pass it as `tool_config.metadata.language`; a policy may also + pin a language for every request in an org. Neither is required — without a + hint, PRESIDIO_LANGUAGE_MODE decides. + """ + if isinstance(tool_config, dict): + metadata = tool_config.get("metadata") + if isinstance(metadata, dict): + hinted = metadata.get("language") + if isinstance(hinted, str) and hinted.strip(): + return hinted + if isinstance(policy_config, dict): + hinted = policy_config.get("language") + if isinstance(hinted, str) and hinted.strip(): + return hinted + return None + + +def analyze_text( + text: str, + language: Optional[str] = None, + entities: Optional[List[str]] = None, +) -> List[Any]: + """Run the analyzer across the resolved languages and merge the findings. + + Shared by the anonymization path and the two detection-only call sites so + that language handling cannot drift between them. + """ + if not USE_PRESIDIO or ANALYZER is None: + return [] + + ents = entities or DETECT_ENTITIES + collected: List[Any] = [] + for lang in _languages_to_analyze(language): + try: + collected.extend(ANALYZER.analyze(text=text, entities=ents, language=lang)) + except Exception as exc: + print(f"Presidio analyze failed for language {lang!r}: {exc}") + return _merge_results(collected) + + def anonymize_text_presidio( - text: str, field_name: str = "", entities: Optional[List[str]] = None + text: str, + field_name: str = "", + entities: Optional[List[str]] = None, + language: Optional[str] = None, ) -> Tuple[str, List[str]]: - """Anonymize text using Presidio""" + """Anonymize text using Presidio. + + `language` is the caller's hint (e.g. from tool_config.metadata). When it is + absent, PRESIDIO_LANGUAGE_MODE decides whether to analyse in the default + language only or in every configured language. + """ if not USE_PRESIDIO or ANALYZER is None: return text, [] - ents = entities or list(ANONYMIZE_OPERATORS.keys()) - results = ANALYZER.analyze(text=text, entities=ents, language="en") + results = analyze_text(text, language=language, entities=entities) if not results: return text, [] # Filter out false positives filtered_results = [] for r in results: - if not is_false_positive(r.entity_type, field_name, text): + if not is_false_positive(r.entity_type, field_name, text[r.start : r.end]): filtered_results.append(r) if not filtered_results: @@ -531,6 +793,79 @@ def is_false_positive(entity_type: str, field_name: str, value: str) -> bool: return False +# Tools denied unless an org's policy overrides the list. Names that no +# legitimate integration carries: each one is a direct code-execution primitive. +DEFAULT_DENY_TOOLS = [ + "python.exec", + "bash.exec", + "code.exec", + "shell.exec", + "subprocess.run", + "subprocess.popen", + "os.system", + "child_process.exec", + "child_process.spawn", + "runtime.exec", +] + +_TOOL_SEPARATORS = re.compile(r"[\s_\-/:]+") +_TOOL_DOT_RUNS = re.compile(r"\.+") + + +def normalize_tool_name(tool: str) -> str: + """Fold a tool name to a canonical dotted form. + + Case, surrounding whitespace, and the choice of separator are presentation + details, not identity: `Python.Exec`, `python_exec`, `python-exec` and + ` python.exec ` all name the same primitive. + """ + folded = _TOOL_SEPARATORS.sub(".", (tool or "").strip().lower()) + return _TOOL_DOT_RUNS.sub(".", folded).strip(".") + + +def is_denied_tool(tool: str, deny_tools: Optional[List[str]] = None) -> bool: + """Whether `tool` matches the denylist. + + Replaces an exact `tool in deny_tools` membership test, which any of the + following walked straight past: `Python.Exec` (case), `python_exec` + (separator), `exec.python` (component order), `agent.python.exec.v2` + (namespacing). Denial is a security boundary, so matching is deliberately + generous — a false deny is a support ticket, a false allow is an incident. + """ + normalized = normalize_tool_name(tool) + if not normalized: + return False + + entries = DEFAULT_DENY_TOOLS if deny_tools is None else deny_tools + components = set(normalized.split(".")) + padded = f".{normalized}." + + for raw_entry in entries: + entry = normalize_tool_name(raw_entry if isinstance(raw_entry, str) else "") + if not entry: + continue + + # Explicit wildcard: "shell.*" denies the whole namespace. + if entry.endswith(".*"): + prefix = entry[:-2] + if normalized == prefix or normalized.startswith(f"{prefix}."): + return True + continue + + if normalized == entry: + return True + + # Same components in a different order — exec.python vs python.exec. + if set(entry.split(".")) == components: + return True + + # Denied name appearing inside a longer namespaced path. + if f".{entry}." in padded: + return True + + return False + + def redact_obj( obj: Any, reasons: Optional[Set[str]] = None, field_name: str = "" ) -> Tuple[Any, Set[str]]: @@ -970,10 +1305,8 @@ def _evaluate_dynamic_policy( try: # PRECEDENCE LEVEL 1: Hard deny for dangerous tools - deny_tools = policy_config.get( - "deny_tools", ["python.exec", "bash.exec", "code.exec", "shell.exec"] - ) - if tool in deny_tools: + deny_tools = policy_config.get("deny_tools", DEFAULT_DENY_TOOLS) + if is_denied_tool(tool, deny_tools): return { "decision": "deny", "raw_text_out": raw_text, @@ -1103,11 +1436,11 @@ def _apply_tool_specific_policy_dynamic( # Run PII detection on raw text findings = [] if USE_PRESIDIO and ANALYZER is not None: - results = ANALYZER.analyze( - text=raw_text, entities=list(ANONYMIZE_OPERATORS.keys()), language="en" + results = analyze_text( + raw_text, language=language_hint(tool_config, policy_config) ) for r in results: - if not is_false_positive(r.entity_type, "", raw_text): + if not is_false_positive(r.entity_type, "", raw_text[r.start : r.end]): findings.append( { "type": f"PII:{r.entity_type.lower()}", @@ -1416,11 +1749,11 @@ def _apply_strict_fallback( # Detect standard PII types using Presidio or regex if USE_PRESIDIO and ANALYZER is not None: # Use Presidio to detect all standard PII types - results = ANALYZER.analyze( - text=raw_text, entities=list(ANONYMIZE_OPERATORS.keys()), language="en" + results = analyze_text( + raw_text, language=language_hint(tool_config, policy_config) ) for r in results: - if not is_false_positive(r.entity_type, "", raw_text): + if not is_false_positive(r.entity_type, "", raw_text[r.start : r.end]): all_findings.append( { "type": f"PII:{r.entity_type.lower()}", diff --git a/app/settings.py b/app/settings.py index 73131a8..84392c6 100644 --- a/app/settings.py +++ b/app/settings.py @@ -18,6 +18,9 @@ # - "local": per-replica in-memory fallback. Intended for single-replica dev. _RATE_LIMIT_FAIL_MODES = {"closed", "open", "local"} +# Allowed values for PRESIDIO_LANGUAGE_MODE. See Settings.presidio_language_mode. +_PRESIDIO_LANGUAGE_MODES = {"hint", "union"} + class Settings(BaseSettings): """Application settings loaded from environment variables""" @@ -57,7 +60,23 @@ class Settings(BaseSettings): # Presidio configuration use_presidio: bool = True - presidio_model: str = "en_core_web_sm" # sm, md, lg + presidio_model: str = "en_core_web_sm" # English model override: sm, md, lg + + # Languages the analyzer is built for, comma-separated. Each entry needs its + # spaCy model present in the image (see Dockerfile). Adding a language here + # without its model installed degrades to the remaining languages rather + # than failing the service. + presidio_languages: str = "en" + + # How the language for a given request is chosen when more than one is + # configured: + # - "hint": use the caller-supplied language, else the first configured one. + # Cheapest; wrong hint means that request is analysed in the + # wrong language and its PII is missed. + # - "union": analyse in every configured language and union the findings. + # Recall-safe default for governance; costs roughly N x the NLP + # time, so it is opt-in rather than automatic. + presidio_language_mode: str = "hint" # API configuration — demo_api_key intentionally removed; all keys must live in DB api_key_header: str = "X-Governs-Key" @@ -88,6 +107,15 @@ class Settings(BaseSettings): # Policy file configuration policy_file: str = "policy.tool_access.yaml" + def presidio_language_list(self) -> list[str]: + """Configured analyzer languages, normalised and de-duplicated.""" + seen: list[str] = [] + for raw in self.presidio_languages.split(","): + code = raw.strip().lower() + if code and code not in seen: + seen.append(code) + return seen or ["en"] + @model_validator(mode="after") def _reject_default_secrets(self) -> "Settings": if self.rate_limit_fail_mode not in _RATE_LIMIT_FAIL_MODES: @@ -95,6 +123,11 @@ def _reject_default_secrets(self) -> "Settings": f"RATE_LIMIT_FAIL_MODE must be one of {sorted(_RATE_LIMIT_FAIL_MODES)}; " f"got {self.rate_limit_fail_mode!r}." ) + if self.presidio_language_mode not in _PRESIDIO_LANGUAGE_MODES: + raise ValueError( + f"PRESIDIO_LANGUAGE_MODE must be one of {sorted(_PRESIDIO_LANGUAGE_MODES)}; " + f"got {self.presidio_language_mode!r}." + ) if not self.debug: self._validate_secret( name="PII_TOKEN_SALT", diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 0000000..ae248f7 --- /dev/null +++ b/bench/README.md @@ -0,0 +1,92 @@ +# PII detection benchmark + +Measures what precheck's redaction path actually catches, as opposed to what +the detector it wraps scores in isolation. Published detector benchmarks +evaluate Presidio as a standalone model on text records; this one runs the +pipeline as deployed — entity allowlist, language configuration, false-positive +filter and all — because that is what decides whether a customer's SSN leaves +the building. + +## Running it + +```bash +# from precheck/ +venv/bin/python -m bench.run # all arms, English only +PRESIDIO_LANGUAGES=en,es,fr,de,zh venv/bin/python -m bench.run --arms regex legacy fixed +venv/bin/python -m bench.run --json results.json # machine-readable +venv/bin/python -m bench.corpus # write corpus.jsonl and print counts +``` + +Non-English arms need the matching spaCy models: + +```bash +venv/bin/python -m spacy download es_core_news_sm # and fr_, de_, zh_ +``` + +## Corpus + +438 items, 630 span-annotated entities, five languages, generated +deterministically from `SEED = 20260811`. Ground truth is recorded at +generation time — templates carry `{SLOT}` markers and the renderer stores each +filled value's exact offsets — so there is no annotation step and no ambiguity. +Every credit card is Luhn-valid, so a detector that gates on the checksum is not +penalised for rejecting a number that was never a card. + +Tiers: `plain`, `high_sensitivity` (names, addresses, medical and national +identifiers), `partial` (last-4, initials, year-only DOB), `obfuscated` (spaced, +spelled-out), `structured` (PII nested in JSON tool arguments at depth 1–4), and +`clean` (no PII at all — the over-redaction control). + +## Arms + +| Arm | What it is | +|---|---| +| `regex` | `anonymize_text_regex` — the no-Presidio fallback | +| `legacy` | The pre-fix shipped path: English-only, 16-entity allowlist, whole-text false-positive filter. Entity list frozen in `run.py` so this keeps measuring what shipped | +| `presidio` | Current entry point, no caller hint | +| `fixed` | Current entry point with the corpus language as hint — what an SDK caller does via `tool_config.metadata.language` | +| `fixed_nohint` | Current entry point, no hint — exercises `PRESIDIO_LANGUAGE_MODE` | +| `presidio_unrestricted` | Diagnostic: same model, no entity allowlist. The gap to `legacy` is what the allowlist alone was discarding | + +## Metrics + +`leakage` is the headline: the fraction of ground-truth PII values that survive +verbatim in the output. Per value, not per span-overlap — an operator cares +whether the SSN left, not how many characters matched. `over_redact` is the +utility counterweight, measured on the clean tier: a layer that redacts +everything has zero leakage and zero value, and operators switch it off. + +## Results + +Five languages, `PRESIDIO_LANGUAGE_MODE=hint`, 438 items: + +| Arm | leakage | high-sensitivity | over-redaction | p50 | +|---|---|---|---|---| +| `regex` | 52.2% | 89.1% | 0.0% | 0.01 ms | +| `legacy` | 51.3% | 89.1% | 0.0% | 2.46 ms | +| `fixed` | **8.2%** | **10.6%** | 0.0% | 2.48 ms | + +PERSON and LOCATION went from 100% leakage to 11.6% and 11.7%. The +high-sensitivity figure is the one to watch: it covers the entity families that +regulated customers are buying redaction for, and 89.1% of them used to pass +through untouched. + +## Known residual gaps + +These are measured, not hypothetical, and none is fixed: + +- **Obfuscated forms — 75% leakage.** `j dot smith at acme dot com`, digits + spelled out, characters spaced. Pattern recognizers see nothing. +- **Partial disclosure — 83.3%.** "the card ending 0366", "initials M.G., born + 1982". Arguably identifying in combination; no detector in the pipeline + models combinations. +- **Chinese — 32.1%**, well above the 1.8–4.5% of the European languages. + `zh_core_web_sm` is the weakest NER model of the five. +- **US_SSN — 14.3%**, unchanged across every arm: the recognizer requires + context words that some templates do not supply. +- **IBAN — 8.3%.** + +The obfuscated and partial numbers match the published failure profile for +pattern-plus-NER detectors (REDACT, arXiv:2606.19881, reports 0.07 and 0.02 +recall respectively for Presidio). Closing them needs a different class of +detector — an LLM pass or a purpose-built encoder — not a better regex. diff --git a/bench/__init__.py b/bench/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bench/corpus.py b/bench/corpus.py new file mode 100644 index 0000000..32b7bd7 --- /dev/null +++ b/bench/corpus.py @@ -0,0 +1,472 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2024 GovernsAI. All rights reserved. +"""Span-annotated PII corpus generator for the precheck detection benchmark. + +Ground truth is produced at generation time: templates carry `{slot}` markers, +and the renderer records the exact character offsets each slot occupies in the +rendered string. No post-hoc annotation, no human labelling, no ambiguity. + +Design notes +------------ +* Deterministic — a fixed seed produces byte-identical corpora, so before/after + runs are directly comparable. +* No new runtime dependencies. Value pools are hand-built per locale rather than + pulled from Faker so that the generator stays inside the existing dependency + set and locale realism is explicit and reviewable. +* Difficulty tiers mirror the axes on which off-the-shelf detectors are known to + degrade (REDACT, arXiv:2606.19881): high-sensitivity entity families, partial + disclosure, and obfuscated surface forms. +""" + +from __future__ import annotations + +import json +import random +import re +from dataclasses import asdict, dataclass, field +from typing import Any, Dict, Iterable, List, Optional, Tuple + +SEED = 20260811 + +LANGUAGES = ["en", "es", "fr", "de", "zh"] + +TIERS = [ + "plain", # standard prose, well-formed entities + "high_sensitivity", # names, addresses, medical + national identifiers + "partial", # partial disclosure (last-4, initials, year-only DOB) + "obfuscated", # spaced / spelled-out / separator-mangled surface forms + "structured", # PII inside nested JSON tool arguments + "clean", # no PII at all — over-redaction control +] + + +@dataclass +class Span: + start: int + end: int + type: str + value: str + + +@dataclass +class Item: + id: str + lang: str + tier: str + text: str + spans: List[Span] = field(default_factory=list) + tool: str = "chat" + + def to_json(self) -> Dict[str, Any]: + d = asdict(self) + return d + + +# --------------------------------------------------------------------------- +# Value pools +# --------------------------------------------------------------------------- + +PERSONS = { + "en": ["John Smith", "Maria Garcia", "Robert Johnson", "Sarah Chen", "David Miller"], + "es": ["Juan Martínez", "Lucía Fernández", "Carlos Ruiz", "Ana Torres", "Miguel Ortega"], + "fr": ["Jean Dupont", "Marie Lefèvre", "Pierre Moreau", "Claire Rousseau", "Luc Bernard"], + "de": ["Hans Müller", "Anna Schmidt", "Peter Wagner", "Julia Becker", "Thomas Fischer"], + "zh": ["张伟", "王芳", "李娜", "刘强", "陈静"], +} + +LOCATIONS = { + "en": ["1600 Pennsylvania Avenue, Washington DC", "42 Baker Street, London", "500 Market St, San Francisco"], + "es": ["Calle Gran Vía 28, Madrid", "Avenida Diagonal 405, Barcelona", "Plaza Mayor 7, Salamanca"], + "fr": ["12 Rue de Rivoli, Paris", "3 Avenue Jean Médecin, Nice", "8 Quai Saint-Vincent, Lyon"], + "de": ["Unter den Linden 5, Berlin", "Maximilianstraße 13, München", "Reeperbahn 22, Hamburg"], + "zh": ["北京市朝阳区建国路88号", "上海市浦东新区世纪大道100号", "广州市天河区天河路299号"], +} + +ORG_DOMAINS = ["acme.com", "globex.io", "initech.co", "umbrella-health.org"] + +EMAIL_LOCALS = ["j.smith", "m.garcia", "contact", "a.torres", "h.mueller"] + +PHONES = { + "en": ["415-555-0132", "(202) 555-0177", "+1 415 555 0199"], + "es": ["+34 612 34 56 78", "912 345 678"], + "fr": ["+33 6 12 34 56 78", "01 42 68 53 00"], + "de": ["+49 151 23456789", "030 123456"], + "zh": ["+86 138 0013 8000", "010-12345678"], +} + +# All Luhn-valid: a detector that gates on the checksum (as Presidio does) must +# not be penalised for rejecting a number that was never a card in the first place. +CARDS = ["4532015112830366", "5425233430109903", "4111 1111 1111 1111"] + +IBANS = ["DE89370400440532013000", "FR1420041010050500013M02606", "ES9121000418450200051332"] + +SSNS = ["123-45-6789", "078-05-1120"] + +IPS = ["192.168.14.203", "10.0.42.7", "203.0.113.51"] + +# Synthetic, and deliberately test-prefixed rather than live-prefixed: a +# live-prefixed literal trips GitHub push protection even when the value is +# fabricated. The detection pattern covers both prefixes, so the fixture still +# exercises the vendor-prefixed-with-internal-underscore shape. +API_KEYS = [ + "sk_test_BENCHFIXTURE0000000000000", + "AKIABENCHFIXTUREEXAMPLE0", +] + +JWTS = [ + "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dozjgNryP4J3jVmNHl0w5N_XgL0n3I9PlFUP0THsR8U" +] + +MRNS = ["MRN: 4457812", "Medical Record Number: A-9928134"] +NPIS = ["NPI: 1234567893"] +DEAS = ["DEA: BM1234563"] +MEMBER_IDS = ["Member ID: HX-88213904"] +DOBS = ["DOB: 04/17/1982", "Date of Birth: 11/02/1975"] +CVVS = ["CVV: 831"] +EXPIRIES = ["exp: 04/28"] + +# Templates: {slot} markers are replaced and their offsets recorded. +TEMPLATES: Dict[str, Dict[str, List[str]]] = { + "en": { + "plain": [ + "Please email {EMAIL_ADDRESS} about the invoice.", + "Call me back on {PHONE_NUMBER} after 3pm.", + "Charge the card {CREDIT_CARD} for the annual plan.", + "The server at {IP_ADDRESS} is refusing connections.", + "Wire the retainer to {IBAN_CODE} by Friday.", + "Our staging key is {API_KEY} — rotate it tonight.", + "Auth header carries {JWT_TOKEN} on every call.", + "His social is {US_SSN}, needed for the I-9.", + ], + "high_sensitivity": [ + "Patient {PERSON} lives at {LOCATION} and is due for a follow-up.", + "Dr. {PERSON} prescribed medication for patient {PERSON}.", + "Ship the settlement cheque to {PERSON} at {LOCATION}.", + "{PERSON} filed the complaint; contact them at {EMAIL_ADDRESS}.", + "Chart for {PERSON} — {US_MEDICAL_RECORD_NUMBER}, {US_DATE_OF_BIRTH}.", + "Prescriber {PERSON}, {US_NPI}, {US_DEA}.", + "Claim submitted by {PERSON} under {US_HEALTH_MEMBER_ID}.", + ], + "partial": [ + "The card ending in 0366 belongs to J. S. — confirm before charging.", + "Patient initials M.G., born 1982, seen last Tuesday.", + "Reach the account holder on the number ending 0132.", + "SSN last four is 6789 for the applicant from Ohio.", + ], + "obfuscated": [ + "Reach him at j dot smith at acme dot com when you can.", + "Contact: j . s m i t h @ a c m e . c o m", + "Card number four five three two 0151 1283 0366.", + "Phone: four one five, five five five, zero one three two.", + ], + "clean": [ + "The deployment finished at 14:32 UTC with no errors.", + "Version 4.2.1 ships the new retry policy for webhooks.", + "Acme Corporation reported quarterly revenue above forecast.", + "Ticket 123-45 was closed as a duplicate of 678-90.", + "The build agent uses 8 vCPUs and 32 GB of memory.", + ], + }, + "es": { + "plain": [ + "Por favor escribe a {EMAIL_ADDRESS} sobre la factura.", + "Llámame al {PHONE_NUMBER} después de las tres.", + "Cobra la tarjeta {CREDIT_CARD} del plan anual.", + "Transfiere el anticipo a {IBAN_CODE} antes del viernes.", + ], + "high_sensitivity": [ + "El paciente {PERSON} vive en {LOCATION} y necesita revisión.", + "La doctora {PERSON} atendió al paciente {PERSON} esta mañana.", + "Envíen el cheque a {PERSON} en {LOCATION}.", + "{PERSON} presentó la reclamación; su correo es {EMAIL_ADDRESS}.", + ], + "partial": [ + "La tarjeta terminada en 0366 pertenece a J. M. — confirmar antes de cobrar.", + "Paciente con iniciales L. F., nacida en 1982.", + ], + "obfuscated": [ + "Su correo es j punto smith arroba acme punto com.", + "Teléfono: seis uno dos, tres cuatro, cinco seis.", + ], + "clean": [ + "El despliegue terminó a las 14:32 UTC sin errores.", + "La versión 4.2.1 incluye la nueva política de reintentos.", + ], + }, + "fr": { + "plain": [ + "Merci d'écrire à {EMAIL_ADDRESS} au sujet de la facture.", + "Rappelez-moi au {PHONE_NUMBER} après 15h.", + "Débitez la carte {CREDIT_CARD} pour l'abonnement annuel.", + "Virez les honoraires sur {IBAN_CODE} avant vendredi.", + ], + "high_sensitivity": [ + "Le patient {PERSON} habite au {LOCATION} et doit être revu.", + "Le docteur {PERSON} a soigné le patient {PERSON} ce matin.", + "Envoyez le chèque à {PERSON} au {LOCATION}.", + "{PERSON} a déposé la plainte ; son courriel est {EMAIL_ADDRESS}.", + ], + "partial": [ + "La carte se terminant par 0366 appartient à J. D. — à confirmer.", + "Patiente aux initiales M. L., née en 1982.", + ], + "obfuscated": [ + "Son adresse est j point smith arobase acme point com.", + "Téléphone : zéro six, douze, trente-quatre.", + ], + "clean": [ + "Le déploiement s'est terminé à 14h32 UTC sans erreur.", + "La version 4.2.1 ajoute la politique de relance.", + ], + }, + "de": { + "plain": [ + "Bitte schreiben Sie an {EMAIL_ADDRESS} wegen der Rechnung.", + "Rufen Sie mich unter {PHONE_NUMBER} nach 15 Uhr zurück.", + "Belasten Sie die Karte {CREDIT_CARD} für den Jahresplan.", + "Überweisen Sie das Honorar auf {IBAN_CODE} bis Freitag.", + ], + "high_sensitivity": [ + "Der Patient {PERSON} wohnt in {LOCATION} und braucht eine Nachuntersuchung.", + "Doktor {PERSON} behandelte heute den Patienten {PERSON}.", + "Senden Sie den Scheck an {PERSON} in {LOCATION}.", + "{PERSON} hat die Beschwerde eingereicht; E-Mail: {EMAIL_ADDRESS}.", + ], + "partial": [ + "Die Karte endend auf 0366 gehört H. M. — bitte bestätigen.", + "Patientin mit den Initialen A. S., geboren 1982.", + ], + "obfuscated": [ + "Seine Adresse ist j punkt smith at acme punkt com.", + "Telefon: null eins fünf eins, zwei drei vier.", + ], + "clean": [ + "Das Deployment endete um 14:32 UTC ohne Fehler.", + "Version 4.2.1 bringt die neue Wiederholungsrichtlinie.", + ], + }, + "zh": { + "plain": [ + "请发邮件到 {EMAIL_ADDRESS} 询问发票事宜。", + "下午三点后请拨打 {PHONE_NUMBER} 联系我。", + "请用银行卡 {CREDIT_CARD} 支付年费。", + ], + "high_sensitivity": [ + "患者 {PERSON} 住在 {LOCATION}，需要复诊。", + "{PERSON} 医生今天为患者 {PERSON} 看诊。", + "请把支票寄给 {PERSON}，地址是 {LOCATION}。", + "{PERSON} 提交了投诉，邮箱是 {EMAIL_ADDRESS}。", + ], + "partial": [ + "尾号 0366 的卡属于张先生，扣款前请确认。", + "患者姓名缩写 W.F.，1982 年出生。", + ], + "obfuscated": [ + "他的邮箱是 j 点 smith 艾特 acme 点 com。", + "电话：一三八 零零一三 八零零零。", + ], + "clean": [ + "部署于 14:32 UTC 完成，没有报错。", + "4.2.1 版本加入了新的重试策略。", + ], + }, +} + +SLOT_POOLS = { + "PERSON": lambda rng, lang: rng.choice(PERSONS[lang]), + "LOCATION": lambda rng, lang: rng.choice(LOCATIONS[lang]), + "EMAIL_ADDRESS": lambda rng, lang: f"{rng.choice(EMAIL_LOCALS)}@{rng.choice(ORG_DOMAINS)}", + "PHONE_NUMBER": lambda rng, lang: rng.choice(PHONES.get(lang, PHONES["en"])), + "CREDIT_CARD": lambda rng, lang: rng.choice(CARDS), + "IBAN_CODE": lambda rng, lang: rng.choice(IBANS), + "US_SSN": lambda rng, lang: rng.choice(SSNS), + "IP_ADDRESS": lambda rng, lang: rng.choice(IPS), + "API_KEY": lambda rng, lang: rng.choice(API_KEYS), + "JWT_TOKEN": lambda rng, lang: rng.choice(JWTS), + "US_MEDICAL_RECORD_NUMBER": lambda rng, lang: rng.choice(MRNS), + "US_NPI": lambda rng, lang: rng.choice(NPIS), + "US_DEA": lambda rng, lang: rng.choice(DEAS), + "US_HEALTH_MEMBER_ID": lambda rng, lang: rng.choice(MEMBER_IDS), + "US_DATE_OF_BIRTH": lambda rng, lang: rng.choice(DOBS), + "PCI_CVV": lambda rng, lang: rng.choice(CVVS), + "PCI_EXPIRY": lambda rng, lang: rng.choice(EXPIRIES), +} + +SLOT_RE = re.compile(r"\{([A-Z_]+)\}") + +# Entity families, used to report high-sensitivity recall separately. +HIGH_SENSITIVITY = { + "PERSON", + "LOCATION", + "US_SSN", + "US_MEDICAL_RECORD_NUMBER", + "US_NPI", + "US_DEA", + "US_HEALTH_MEMBER_ID", + "US_DATE_OF_BIRTH", +} + + +def render(template: str, rng: random.Random, lang: str) -> Tuple[str, List[Span]]: + """Fill {SLOT} markers, recording exact offsets of each filled value.""" + out: List[str] = [] + spans: List[Span] = [] + pos = 0 + cursor = 0 + for m in SLOT_RE.finditer(template): + literal = template[cursor : m.start()] + out.append(literal) + pos += len(literal) + + entity = m.group(1) + value = SLOT_POOLS[entity](rng, lang) + spans.append(Span(start=pos, end=pos + len(value), type=entity, value=value)) + out.append(value) + pos += len(value) + cursor = m.end() + + tail = template[cursor:] + out.append(tail) + return "".join(out), spans + + +def _structured_items(rng: random.Random, lang: str, n: int, start_idx: int) -> List[Item]: + """PII nested inside JSON tool arguments at varying depth.""" + items: List[Item] = [] + for i in range(n): + person = rng.choice(PERSONS[lang]) + email = f"{rng.choice(EMAIL_LOCALS)}@{rng.choice(ORG_DOMAINS)}" + card = rng.choice(CARDS) + depth = (i % 3) + 1 + + leaf: Dict[str, Any] = { + "customer_name": person, + "contact_email": email, + "payment": {"card_number": card}, + } + payload: Dict[str, Any] = leaf + for d in range(depth): + payload = {f"level_{d}": payload} + + text = json.dumps(payload, ensure_ascii=False) + spans: List[Span] = [] + for value, etype in ( + (person, "PERSON"), + (email, "EMAIL_ADDRESS"), + (card, "CREDIT_CARD"), + ): + idx = text.find(value) + if idx >= 0: + spans.append(Span(start=idx, end=idx + len(value), type=etype, value=value)) + + items.append( + Item( + id=f"{lang}-structured-{start_idx + i:04d}", + lang=lang, + tier="structured", + text=text, + spans=spans, + tool="db.query", + ) + ) + return items + + +# Partial and obfuscated templates carry their PII inline rather than in slots; +# these tables give the ground-truth values the detector is expected to catch. +INLINE_TRUTH: Dict[str, List[Tuple[str, str]]] = { + "The card ending in 0366 belongs to J. S. — confirm before charging.": [("0366", "CREDIT_CARD")], + "Patient initials M.G., born 1982, seen last Tuesday.": [("M.G.", "PERSON")], + "Reach the account holder on the number ending 0132.": [("0132", "PHONE_NUMBER")], + "SSN last four is 6789 for the applicant from Ohio.": [("6789", "US_SSN")], + "Reach him at j dot smith at acme dot com when you can.": [("j dot smith at acme dot com", "EMAIL_ADDRESS")], + "Contact: j . s m i t h @ a c m e . c o m": [("j . s m i t h @ a c m e . c o m", "EMAIL_ADDRESS")], + "Card number four five three two 0151 1283 0366.": [("four five three two 0151 1283 0366", "CREDIT_CARD")], + "Phone: four one five, five five five, zero one three two.": [ + ("four one five, five five five, zero one three two", "PHONE_NUMBER") + ], + "La tarjeta terminada en 0366 pertenece a J. M. — confirmar antes de cobrar.": [("0366", "CREDIT_CARD")], + "Paciente con iniciales L. F., nacida en 1982.": [("L. F.", "PERSON")], + "Su correo es j punto smith arroba acme punto com.": [("j punto smith arroba acme punto com", "EMAIL_ADDRESS")], + "Teléfono: seis uno dos, tres cuatro, cinco seis.": [("seis uno dos, tres cuatro, cinco seis", "PHONE_NUMBER")], + "La carte se terminant par 0366 appartient à J. D. — à confirmer.": [("0366", "CREDIT_CARD")], + "Patiente aux initiales M. L., née en 1982.": [("M. L.", "PERSON")], + "Son adresse est j point smith arobase acme point com.": [("j point smith arobase acme point com", "EMAIL_ADDRESS")], + "Téléphone : zéro six, douze, trente-quatre.": [("zéro six, douze, trente-quatre", "PHONE_NUMBER")], + "Die Karte endend auf 0366 gehört H. M. — bitte bestätigen.": [("0366", "CREDIT_CARD")], + "Patientin mit den Initialen A. S., geboren 1982.": [("A. S.", "PERSON")], + "Seine Adresse ist j punkt smith at acme punkt com.": [("j punkt smith at acme punkt com", "EMAIL_ADDRESS")], + "Telefon: null eins fünf eins, zwei drei vier.": [("null eins fünf eins, zwei drei vier", "PHONE_NUMBER")], + "尾号 0366 的卡属于张先生，扣款前请确认。": [("0366", "CREDIT_CARD")], + "患者姓名缩写 W.F.，1982 年出生。": [("W.F.", "PERSON")], + "他的邮箱是 j 点 smith 艾特 acme 点 com。": [("j 点 smith 艾特 acme 点 com", "EMAIL_ADDRESS")], + "电话：一三八 零零一三 八零零零。": [("一三八 零零一三 八零零零", "PHONE_NUMBER")], +} + + +def build(per_template: int = 6, languages: Optional[Iterable[str]] = None) -> List[Item]: + """Generate the corpus. Deterministic for a fixed SEED.""" + rng = random.Random(SEED) + langs = list(languages or LANGUAGES) + items: List[Item] = [] + counter = 0 + + for lang in langs: + tmpl_by_tier = TEMPLATES[lang] + for tier in ["plain", "high_sensitivity", "clean"]: + for template in tmpl_by_tier.get(tier, []): + for _ in range(per_template): + text, spans = render(template, rng, lang) + items.append( + Item( + id=f"{lang}-{tier}-{counter:04d}", + lang=lang, + tier=tier, + text=text, + spans=spans, + ) + ) + counter += 1 + + for tier in ["partial", "obfuscated"]: + for template in tmpl_by_tier.get(tier, []): + truth = INLINE_TRUTH.get(template, []) + spans = [] + for value, etype in truth: + idx = template.find(value) + if idx >= 0: + spans.append(Span(start=idx, end=idx + len(value), type=etype, value=value)) + items.append( + Item( + id=f"{lang}-{tier}-{counter:04d}", + lang=lang, + tier=tier, + text=template, + spans=spans, + ) + ) + counter += 1 + + items.extend(_structured_items(rng, lang, per_template * 2, counter)) + counter += per_template * 2 + + return items + + +def write_jsonl(items: List[Item], path: str) -> None: + with open(path, "w", encoding="utf-8") as fh: + for it in items: + fh.write(json.dumps(it.to_json(), ensure_ascii=False) + "\n") + + +if __name__ == "__main__": # pragma: no cover - manual invocation + corpus = build() + write_jsonl(corpus, "bench/corpus.jsonl") + by_tier: Dict[str, int] = {} + by_lang: Dict[str, int] = {} + for it in corpus: + by_tier[it.tier] = by_tier.get(it.tier, 0) + 1 + by_lang[it.lang] = by_lang.get(it.lang, 0) + 1 + print(f"{len(corpus)} items") + print("by tier:", by_tier) + print("by lang:", by_lang) diff --git a/bench/run.py b/bench/run.py new file mode 100644 index 0000000..5846614 --- /dev/null +++ b/bench/run.py @@ -0,0 +1,335 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2024 GovernsAI. All rights reserved. +"""Run the PII detection benchmark against precheck's redaction paths. + +Usage (from precheck/): + venv/bin/python -m bench.run # all arms, all languages + venv/bin/python -m bench.run --arms regex # single arm + venv/bin/python -m bench.run --json out.json # machine-readable report + +Metrics +------- +leakage fraction of ground-truth PII values that survive verbatim in the + output. This is the safety metric: one surviving value is a + disclosure regardless of how many others were caught. +item_leakage fraction of items where at least one value survived. +over_redact fraction of clean-tier items (no PII) that were modified anyway. +latency_ms per-call wall clock, p50/p95. + +Leakage is deliberately the headline rather than span F1: an operator cares +whether the SSN left the building, not how many characters overlapped. +""" + +from __future__ import annotations + +import argparse +import json +import statistics +import sys +import time +from collections import defaultdict +from typing import Callable, Dict, List, Optional, Tuple + +from bench.corpus import HIGH_SENSITIVITY, Item, build + +Arm = Callable[[str, str], Tuple[str, List[str]]] + +# The entity allowlist the pre-fix code passed to Presidio: the keys of +# ANONYMIZE_OPERATORS as they stood before PERSON/LOCATION/NRP were added. +# Frozen here so the `legacy` arm keeps measuring what shipped even as the +# application constant moves on. +LEGACY_ENTITIES = [ + "DEFAULT", + "CREDIT_CARD", + "PHONE_NUMBER", + "EMAIL_ADDRESS", + "IP_ADDRESS", + "IBAN_CODE", + "US_SSN", + "US_MEDICAL_RECORD_NUMBER", + "US_HEALTH_MEMBER_ID", + "US_NPI", + "US_DEA", + "US_DATE_OF_BIRTH", + "PCI_CVV", + "PCI_EXPIRY", + "API_KEY", + "JWT_TOKEN", +] + + +# --------------------------------------------------------------------------- +# Arms +# --------------------------------------------------------------------------- + + +def arm_regex(text: str, lang: str) -> Tuple[str, List[str]]: + from app.policies import anonymize_text_regex + + return anonymize_text_regex(text) + + +def arm_legacy(text: str, lang: str) -> Tuple[str, List[str]]: + """Reproduces the pre-fix shipped path: English-only, allowlist-constrained. + + Kept so before/after numbers come out of a single run rather than being + compared across two trees. + """ + from app.policies import ANALYZER, ANONYMIZER + from app.policies import entity_type_to_placeholder, is_false_positive + from presidio_anonymizer.entities import OperatorConfig + + if ANALYZER is None: + return text, [] + try: + results = ANALYZER.analyze( + text=text, entities=LEGACY_ENTITIES, language="en" + ) + except Exception: + return text, [] + # The pre-fix code passed the whole text to the false-positive filter + # rather than the matched span, so the filter almost never fired. Preserved + # here deliberately: this arm documents what shipped, not what should have. + results = [r for r in results if not is_false_positive(r.entity_type, "", text)] + if not results: + return text, [] + ops = { + r.entity_type: OperatorConfig( + "replace", {"new_value": entity_type_to_placeholder(r.entity_type)} + ) + for r in results + } + out = ANONYMIZER.anonymize(text=text, analyzer_results=results, operators=ops).text + return out, sorted({f"pii.redacted:{r.entity_type.lower()}" for r in results}) + + +def arm_presidio(text: str, lang: str) -> Tuple[str, List[str]]: + """Current shipped entry point with no caller hint.""" + from app.policies import anonymize_text_presidio + + return anonymize_text_presidio(text) + + +def arm_presidio_unrestricted(text: str, lang: str) -> Tuple[str, List[str]]: + """Diagnostic: same model, no entity allowlist, still English-pinned. + + The delta between this and `presidio` is the cost of the entity allowlist + alone — i.e. detection the loaded model already performs and the pipeline + then discards. + """ + from app.policies import ANALYZER, ANONYMIZER, entity_type_to_placeholder + from presidio_anonymizer.entities import OperatorConfig + + if ANALYZER is None: + return text, [] + results = ANALYZER.analyze(text=text, language="en") + if not results: + return text, [] + ops = { + r.entity_type: OperatorConfig( + "replace", {"new_value": entity_type_to_placeholder(r.entity_type)} + ) + for r in results + } + out = ANONYMIZER.anonymize(text=text, analyzer_results=results, operators=ops).text + return out, sorted({f"pii.redacted:{r.entity_type.lower()}" for r in results}) + + +def arm_precheck_fixed(text: str, lang: str) -> Tuple[str, List[str]]: + """Post-fix path: multi-language analyzer + widened entity set. + + Passes the corpus language as the caller hint, which is what an SDK caller + does via tool_config.metadata.language. + """ + from app.policies import anonymize_text_presidio + + return anonymize_text_presidio(text, language=lang) + + +def arm_precheck_fixed_nohint(text: str, lang: str) -> Tuple[str, List[str]]: + """Post-fix path with no language hint — exercises PRESIDIO_LANGUAGE_MODE.""" + from app.policies import anonymize_text_presidio + + return anonymize_text_presidio(text) + + +ARMS: Dict[str, Arm] = { + "regex": arm_regex, + "legacy": arm_legacy, + "presidio": arm_presidio, + "presidio_unrestricted": arm_presidio_unrestricted, + "fixed": arm_precheck_fixed, + "fixed_nohint": arm_precheck_fixed_nohint, +} + + +# --------------------------------------------------------------------------- +# Scoring +# --------------------------------------------------------------------------- + + +class Accumulator: + def __init__(self) -> None: + self.spans_total = 0 + self.spans_leaked = 0 + self.items_total = 0 + self.items_leaked = 0 + self.clean_total = 0 + self.clean_modified = 0 + self.latencies: List[float] = [] + self.by_entity: Dict[str, List[int]] = defaultdict(lambda: [0, 0]) # [leaked, total] + + def rate(self, num: int, den: int) -> Optional[float]: + return round(num / den, 4) if den else None + + def summary(self) -> Dict[str, object]: + lat = sorted(self.latencies) + p50 = round(statistics.median(lat), 2) if lat else None + p95 = round(lat[int(len(lat) * 0.95)], 2) if len(lat) >= 20 else None + return { + "spans": self.spans_total, + "leakage": self.rate(self.spans_leaked, self.spans_total), + "item_leakage": self.rate(self.items_leaked, self.items_total), + "over_redact": self.rate(self.clean_modified, self.clean_total), + "latency_p50_ms": p50, + "latency_p95_ms": p95, + } + + +def score_item(item: Item, out: str) -> Tuple[int, int, List[str]]: + """Return (leaked_spans, total_spans, leaked_entity_types).""" + leaked = 0 + types: List[str] = [] + for span in item.spans: + if span.value and span.value in out: + leaked += 1 + types.append(span.type) + return leaked, len(item.spans), types + + +def run_arm(name: str, arm: Arm, items: List[Item]) -> Dict[str, object]: + overall = Accumulator() + by_lang: Dict[str, Accumulator] = defaultdict(Accumulator) + by_tier: Dict[str, Accumulator] = defaultdict(Accumulator) + high_sens = Accumulator() + + for item in items: + t0 = time.perf_counter() + try: + out, _reasons = arm(item.text, item.lang) + except Exception as exc: # an arm that crashes leaks everything + print(f" ! {name} raised on {item.id}: {type(exc).__name__}: {exc}", file=sys.stderr) + out = item.text + elapsed_ms = (time.perf_counter() - t0) * 1000 + + leaked, total, leaked_types = score_item(item, out) + + for acc in (overall, by_lang[item.lang], by_tier[item.tier]): + acc.latencies.append(elapsed_ms) + acc.spans_total += total + acc.spans_leaked += leaked + acc.items_total += 1 + if leaked: + acc.items_leaked += 1 + if item.tier == "clean": + acc.clean_total += 1 + if out != item.text: + acc.clean_modified += 1 + + for span in item.spans: + etype = span.type + overall.by_entity[etype][1] += 1 + if span.value in out: + overall.by_entity[etype][0] += 1 + if etype in HIGH_SENSITIVITY: + high_sens.spans_total += 1 + if span.value in out: + high_sens.spans_leaked += 1 + + return { + "overall": overall.summary(), + "high_sensitivity_leakage": overall.rate(high_sens.spans_leaked, high_sens.spans_total), + "by_lang": {k: v.summary() for k, v in sorted(by_lang.items())}, + "by_tier": {k: v.summary() for k, v in sorted(by_tier.items())}, + "by_entity": { + k: {"leaked": v[0], "total": v[1], "leakage": round(v[0] / v[1], 4) if v[1] else None} + for k, v in sorted(overall.by_entity.items()) + }, + } + + +def fmt_pct(v: Optional[float]) -> str: + return " n/a" if v is None else f"{v * 100:5.1f}%" + + +def print_report(results: Dict[str, Dict[str, object]]) -> None: + arms = list(results.keys()) + + print("\n=== Overall (leakage = ground-truth PII values surviving verbatim) ===\n") + print(f"{'arm':<24}{'leakage':>9}{'item_leak':>11}{'high_sens':>11}{'over_redact':>13}{'p50 ms':>9}{'p95 ms':>9}") + for arm in arms: + o = results[arm]["overall"] + print( + f"{arm:<24}{fmt_pct(o['leakage']):>9}{fmt_pct(o['item_leakage']):>11}" + f"{fmt_pct(results[arm]['high_sensitivity_leakage']):>11}" + f"{fmt_pct(o['over_redact']):>13}" + f"{str(o['latency_p50_ms']):>9}{str(o['latency_p95_ms']):>9}" + ) + + print("\n=== Leakage by language ===\n") + langs = sorted({lang for arm in arms for lang in results[arm]["by_lang"]}) + print(f"{'arm':<24}" + "".join(f"{lang:>9}" for lang in langs)) + for arm in arms: + row = "".join(fmt_pct(results[arm]["by_lang"].get(lang, {}).get("leakage")) for lang in langs) + print(f"{arm:<24}{row}") + + print("\n=== Leakage by tier ===\n") + tiers = sorted({t for arm in arms for t in results[arm]["by_tier"]}) + print(f"{'arm':<24}" + "".join(f"{t[:9]:>18}" for t in tiers)) + for arm in arms: + row = "".join( + f"{fmt_pct(results[arm]['by_tier'].get(t, {}).get('leakage')):>18}" for t in tiers + ) + print(f"{arm:<24}{row}") + + print("\n=== Leakage by entity type ===\n") + ents = sorted({e for arm in arms for e in results[arm]["by_entity"]}) + print(f"{'entity':<28}" + "".join(f"{a[:16]:>22}" for a in arms)) + for ent in ents: + row = "" + for arm in arms: + cell = results[arm]["by_entity"].get(ent) + row += f"{fmt_pct(cell['leakage']) if cell else ' n/a':>22}" + print(f"{ent:<28}{row}") + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--arms", nargs="*", default=list(ARMS.keys())) + ap.add_argument("--languages", nargs="*", default=None) + ap.add_argument("--per-template", type=int, default=6) + ap.add_argument("--json", dest="json_out", default=None) + args = ap.parse_args() + + items = build(per_template=args.per_template, languages=args.languages) + print(f"corpus: {len(items)} items, {sum(len(i.spans) for i in items)} annotated spans") + + results: Dict[str, Dict[str, object]] = {} + for name in args.arms: + if name not in ARMS: + print(f"unknown arm: {name}", file=sys.stderr) + return 2 + print(f"running arm: {name} ...", flush=True) + results[name] = run_arm(name, ARMS[name], items) + + print_report(results) + + if args.json_out: + with open(args.json_out, "w", encoding="utf-8") as fh: + json.dump(results, fh, indent=2) + print(f"\nwrote {args.json_out}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/env.example b/env.example index dd5752f..999f41e 100644 --- a/env.example +++ b/env.example @@ -18,8 +18,26 @@ PRECHECK_ALLOW_CACHE_TTL_SECONDS=60 # Presidio Configuration USE_PRESIDIO=true +# English spaCy model (sm | md | lg). Larger models raise PERSON/LOCATION +# recall at the cost of memory and load time. PRESIDIO_MODEL=en_core_web_sm +# Languages the analyzer is built for, comma-separated. Each needs its spaCy +# model present in the image (the Dockerfile installs en, es, fr, de, zh). +# A language listed here without its model is skipped with a log line rather +# than failing startup. +PRESIDIO_LANGUAGES=en + +# How the language is chosen per request when several are configured: +# hint - use the caller's tool_config.metadata.language, else the first +# configured language. One NLP pass; a wrong hint misses that +# request's PII. +# union - analyse in every configured language and merge findings. Recall-safe +# but roughly N x the NLP time, and non-native NER models produce +# false positives on clean text (53.8% of PII-free control text was +# modified in bench/ at five languages). Opt in deliberately. +PRESIDIO_LANGUAGE_MODE=hint + # API Configuration API_KEY_HEADER=X-Governs-Key # API_KEY=your-api-key-here # Optional: API key from .env (fallback if not in header) diff --git a/tests/test_deny_tools_matching.py b/tests/test_deny_tools_matching.py new file mode 100644 index 0000000..fcbe883 --- /dev/null +++ b/tests/test_deny_tools_matching.py @@ -0,0 +1,113 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2024 GovernsAI. All rights reserved. +"""TEST — denylist matching for dangerous tools. + +The previous implementation was `if tool in deny_tools`, an exact string +membership test. Every case in TestBypassVariants walked past it while naming +the same code-execution primitive. Denial is a security boundary: these tests +pin the matcher's generosity so a later refactor cannot quietly narrow it. +""" + +import pytest + +from app.policies import DEFAULT_DENY_TOOLS, is_denied_tool, normalize_tool_name + + +class TestNormalizeToolName: + @pytest.mark.parametrize( + "raw,expected", + [ + ("python.exec", "python.exec"), + ("Python.Exec", "python.exec"), + ("PYTHON.EXEC", "python.exec"), + (" python.exec ", "python.exec"), + ("python_exec", "python.exec"), + ("python-exec", "python.exec"), + ("python/exec", "python.exec"), + ("python:exec", "python.exec"), + ("python..exec", "python.exec"), + (".python.exec.", "python.exec"), + ("", ""), + ], + ) + def test_folds_to_canonical_form(self, raw, expected): + assert normalize_tool_name(raw) == expected + + def test_handles_none(self): + assert normalize_tool_name(None) == "" + + +class TestBypassVariants: + """Each of these was allowed by the exact-match implementation.""" + + @pytest.mark.parametrize( + "tool", + [ + "Python.Exec", + "PYTHON.EXEC", + " python.exec ", + "python_exec", + "python-exec", + "python/exec", + "shell/exec", + "Bash.Exec", + "exec.python", # component order + "exec.shell", + "agent.python.exec.v2", # namespaced + "tools.bash.exec", + ], + ) + def test_denied(self, tool): + assert is_denied_tool(tool) is True + + @pytest.mark.parametrize( + "tool", + ["subprocess.run", "subprocess.popen", "os.system", "child_process.spawn", "runtime.exec"], + ) + def test_semantically_equivalent_primitives_denied(self, tool): + """Names that are code execution by any other spelling.""" + assert is_denied_tool(tool) is True + + +class TestLegitimateToolsStillAllowed: + @pytest.mark.parametrize( + "tool", + [ + "chat", + "db.query", + "web.search", + "http.get", + "python.format", # not exec + "exec_summary", # substring of "exec", not the primitive + "executive.report", + "search.python.docs", + "", + ], + ) + def test_allowed(self, tool): + assert is_denied_tool(tool) is False + + def test_empty_denylist_allows_everything(self): + assert is_denied_tool("python.exec", []) is False + + +class TestCustomDenylists: + def test_operator_supplied_list_is_normalized_too(self): + assert is_denied_tool("Danger_Tool", ["danger.tool"]) is True + + def test_wildcard_denies_namespace(self): + assert is_denied_tool("shell.anything", ["shell.*"]) is True + assert is_denied_tool("shell", ["shell.*"]) is True + + def test_wildcard_does_not_leak_across_prefixes(self): + assert is_denied_tool("shelly.thing", ["shell.*"]) is False + + def test_non_string_entries_are_skipped(self): + assert is_denied_tool("python.exec", [None, 42, "python.exec"]) is True + assert is_denied_tool("chat", [None, 42]) is False + + +class TestDefaults: + def test_legacy_defaults_retained(self): + for tool in ["python.exec", "bash.exec", "code.exec", "shell.exec"]: + assert tool in DEFAULT_DENY_TOOLS diff --git a/tests/test_multilingual_pii.py b/tests/test_multilingual_pii.py new file mode 100644 index 0000000..cd8b4ae --- /dev/null +++ b/tests/test_multilingual_pii.py @@ -0,0 +1,153 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2024 GovernsAI. All rights reserved. +"""TEST — language handling and entity coverage in the Presidio path. + +Two defects these tests pin down: + +1. `ANALYZER.analyze(..., language="en")` was hardcoded at every call site, so + the es/fr/de/zh spaCy models the image installs were never reachable. +2. The entity allowlist passed to the analyzer omitted PERSON, LOCATION and + NRP, so names and addresses were detected by the NER model and then dropped + before redaction. `bench/` measured 100% leakage on both, in English too. + +Tests that need non-English models are marked `multilingual` and skip when the +models are absent; run them inside the precheck image. +""" + +import importlib.util + +import pytest + +from app import policies +from app.policies import ( + DETECT_ENTITIES, + anonymize_text_presidio, + language_hint, +) +from app.settings import Settings + + +def _has_model(name: str) -> bool: + return importlib.util.find_spec(name) is not None + + +requires_presidio = pytest.mark.skipif( + policies.ANALYZER is None, reason="Presidio analyzer unavailable in this environment" +) + + +class TestEntityCoverage: + """The allowlist must carry the entity types a redaction policy implies.""" + + @pytest.mark.parametrize("entity", ["PERSON", "LOCATION", "NRP"]) + def test_identity_entities_are_requested(self, entity): + assert entity in DETECT_ENTITIES + + @pytest.mark.parametrize("entity", ["ORGANIZATION", "DATE_TIME"]) + def test_low_precision_entities_excluded(self, entity): + """Both wreck PII-free text: 'Acme Corporation', 'quarterly', 'UTC'.""" + assert entity not in DETECT_ENTITIES + + def test_default_is_not_an_entity_type(self): + """The old code passed ANONYMIZE_OPERATORS keys, which include DEFAULT.""" + assert "DEFAULT" not in DETECT_ENTITIES + + +@requires_presidio +class TestPersonAndLocationRedaction: + def test_person_name_redacted(self): + out, reasons = anonymize_text_presidio( + "Patient Maria Garcia is due for a follow-up.", language="en" + ) + assert "Maria Garcia" not in out + assert any("person" in r for r in reasons) + + def test_location_redacted(self): + out, _ = anonymize_text_presidio( + "Ship the cheque to 1600 Pennsylvania Avenue, Washington DC.", language="en" + ) + assert "Washington DC" not in out + + def test_clean_text_survives(self): + """Utility control: no PII in, nothing redacted out.""" + text = "Acme Corporation reported quarterly revenue above forecast." + out, reasons = anonymize_text_presidio(text, language="en") + assert out == text + assert reasons == [] + + +@requires_presidio +class TestLanguageResolution: + def test_unknown_hint_falls_back_to_supported_language(self): + out, _ = anonymize_text_presidio( + "Patient Maria Garcia is due for a follow-up.", language="xx" + ) + assert "Maria Garcia" not in out + + def test_supported_languages_populated(self): + assert policies.SUPPORTED_LANGUAGES + assert "en" in policies.SUPPORTED_LANGUAGES + + +class TestLanguageHint: + def test_reads_tool_config_metadata(self): + assert language_hint({"metadata": {"language": "es"}}, None) == "es" + + def test_falls_back_to_policy_config(self): + assert language_hint(None, {"language": "fr"}) == "fr" + + def test_tool_config_wins(self): + assert language_hint({"metadata": {"language": "es"}}, {"language": "fr"}) == "es" + + @pytest.mark.parametrize( + "tool_config,policy_config", + [ + (None, None), + ({}, {}), + ({"metadata": {}}, {}), + ({"metadata": {"language": " "}}, {}), + ({"metadata": {"language": 42}}, {}), + ("not-a-dict", None), + ], + ) + def test_absent_or_malformed_hint_is_none(self, tool_config, policy_config): + assert language_hint(tool_config, policy_config) is None + + +class TestSettings: + def test_language_list_parsed_and_deduped(self): + s = Settings(presidio_languages="en, es ,EN,fr", debug=True) + assert s.presidio_language_list() == ["en", "es", "fr"] + + def test_empty_language_list_defaults_to_english(self): + assert Settings(presidio_languages=" , ", debug=True).presidio_language_list() == ["en"] + + def test_invalid_language_mode_rejected(self): + with pytest.raises(ValueError, match="PRESIDIO_LANGUAGE_MODE"): + Settings(presidio_language_mode="sometimes", debug=True) + + @pytest.mark.parametrize("mode", ["hint", "union"]) + def test_valid_language_modes_accepted(self, mode): + assert Settings(presidio_language_mode=mode, debug=True).presidio_language_mode == mode + + +@pytest.mark.multilingual +class TestNonEnglishRedaction: + """Requires the es/fr/de spaCy models; run inside the precheck image.""" + + @pytest.mark.parametrize( + "lang,model,text,secret", + [ + ("es", "es_core_news_sm", "El paciente Juan Martínez necesita revisión.", "Juan Martínez"), + ("fr", "fr_core_news_sm", "Le patient Jean Dupont doit être revu.", "Jean Dupont"), + ("de", "de_core_news_sm", "Der Patient Hans Müller braucht eine Untersuchung.", "Hans Müller"), + ], + ) + def test_person_redacted_in_language(self, lang, model, text, secret): + if not _has_model(model): + pytest.skip(f"{model} not installed; run inside the precheck image") + if lang not in policies.SUPPORTED_LANGUAGES: + pytest.skip(f"analyzer not built for {lang}; set PRESIDIO_LANGUAGES") + + out, _ = anonymize_text_presidio(text, language=lang) + assert secret not in out From 28be1abfa371b11c0eff7dc923c32c60bb3d06da Mon Sep 17 00:00:00 2001 From: Shaishav Pidadi Date: Tue, 11 Aug 2026 22:15:12 -0400 Subject: [PATCH 2/4] style: apply black and isort to the new benchmark and test files Formatting only, no behaviour change. Fixes the black failure this branch introduced for app/policies.py and the two new test modules. --- app/policies.py | 4 +- bench/corpus.py | 142 ++++++++++++++++++++++++------ bench/run.py | 51 ++++++++--- tests/test_deny_tools_matching.py | 8 +- tests/test_multilingual_pii.py | 37 ++++++-- 5 files changed, 189 insertions(+), 53 deletions(-) diff --git a/app/policies.py b/app/policies.py index 1a2aee6..7ad5e44 100644 --- a/app/policies.py +++ b/app/policies.py @@ -367,9 +367,7 @@ def build_presidio(): # analyzer whose languages differ from it; constructing the registry # without this is what silently drops everything back to English. registry = RecognizerRegistry(supported_languages=languages) - registry.load_predefined_recognizers( - languages=languages, nlp_engine=nlp_engine - ) + registry.load_predefined_recognizers(languages=languages, nlp_engine=nlp_engine) for language in languages: for recognizer in _custom_recognizers(language): diff --git a/bench/corpus.py b/bench/corpus.py index 32b7bd7..63523e4 100644 --- a/bench/corpus.py +++ b/bench/corpus.py @@ -67,19 +67,63 @@ def to_json(self) -> Dict[str, Any]: # --------------------------------------------------------------------------- PERSONS = { - "en": ["John Smith", "Maria Garcia", "Robert Johnson", "Sarah Chen", "David Miller"], - "es": ["Juan Martínez", "Lucía Fernández", "Carlos Ruiz", "Ana Torres", "Miguel Ortega"], - "fr": ["Jean Dupont", "Marie Lefèvre", "Pierre Moreau", "Claire Rousseau", "Luc Bernard"], - "de": ["Hans Müller", "Anna Schmidt", "Peter Wagner", "Julia Becker", "Thomas Fischer"], + "en": [ + "John Smith", + "Maria Garcia", + "Robert Johnson", + "Sarah Chen", + "David Miller", + ], + "es": [ + "Juan Martínez", + "Lucía Fernández", + "Carlos Ruiz", + "Ana Torres", + "Miguel Ortega", + ], + "fr": [ + "Jean Dupont", + "Marie Lefèvre", + "Pierre Moreau", + "Claire Rousseau", + "Luc Bernard", + ], + "de": [ + "Hans Müller", + "Anna Schmidt", + "Peter Wagner", + "Julia Becker", + "Thomas Fischer", + ], "zh": ["张伟", "王芳", "李娜", "刘强", "陈静"], } LOCATIONS = { - "en": ["1600 Pennsylvania Avenue, Washington DC", "42 Baker Street, London", "500 Market St, San Francisco"], - "es": ["Calle Gran Vía 28, Madrid", "Avenida Diagonal 405, Barcelona", "Plaza Mayor 7, Salamanca"], - "fr": ["12 Rue de Rivoli, Paris", "3 Avenue Jean Médecin, Nice", "8 Quai Saint-Vincent, Lyon"], - "de": ["Unter den Linden 5, Berlin", "Maximilianstraße 13, München", "Reeperbahn 22, Hamburg"], - "zh": ["北京市朝阳区建国路88号", "上海市浦东新区世纪大道100号", "广州市天河区天河路299号"], + "en": [ + "1600 Pennsylvania Avenue, Washington DC", + "42 Baker Street, London", + "500 Market St, San Francisco", + ], + "es": [ + "Calle Gran Vía 28, Madrid", + "Avenida Diagonal 405, Barcelona", + "Plaza Mayor 7, Salamanca", + ], + "fr": [ + "12 Rue de Rivoli, Paris", + "3 Avenue Jean Médecin, Nice", + "8 Quai Saint-Vincent, Lyon", + ], + "de": [ + "Unter den Linden 5, Berlin", + "Maximilianstraße 13, München", + "Reeperbahn 22, Hamburg", + ], + "zh": [ + "北京市朝阳区建国路88号", + "上海市浦东新区世纪大道100号", + "广州市天河区天河路299号", + ], } ORG_DOMAINS = ["acme.com", "globex.io", "initech.co", "umbrella-health.org"] @@ -98,7 +142,11 @@ def to_json(self) -> Dict[str, Any]: # not be penalised for rejecting a number that was never a card in the first place. CARDS = ["4532015112830366", "5425233430109903", "4111 1111 1111 1111"] -IBANS = ["DE89370400440532013000", "FR1420041010050500013M02606", "ES9121000418450200051332"] +IBANS = [ + "DE89370400440532013000", + "FR1420041010050500013M02606", + "ES9121000418450200051332", +] SSNS = ["123-45-6789", "078-05-1120"] @@ -330,7 +378,9 @@ def render(template: str, rng: random.Random, lang: str) -> Tuple[str, List[Span return "".join(out), spans -def _structured_items(rng: random.Random, lang: str, n: int, start_idx: int) -> List[Item]: +def _structured_items( + rng: random.Random, lang: str, n: int, start_idx: int +) -> List[Item]: """PII nested inside JSON tool arguments at varying depth.""" items: List[Item] = [] for i in range(n): @@ -357,7 +407,9 @@ def _structured_items(rng: random.Random, lang: str, n: int, start_idx: int) -> ): idx = text.find(value) if idx >= 0: - spans.append(Span(start=idx, end=idx + len(value), type=etype, value=value)) + spans.append( + Span(start=idx, end=idx + len(value), type=etype, value=value) + ) items.append( Item( @@ -375,36 +427,66 @@ def _structured_items(rng: random.Random, lang: str, n: int, start_idx: int) -> # Partial and obfuscated templates carry their PII inline rather than in slots; # these tables give the ground-truth values the detector is expected to catch. INLINE_TRUTH: Dict[str, List[Tuple[str, str]]] = { - "The card ending in 0366 belongs to J. S. — confirm before charging.": [("0366", "CREDIT_CARD")], + "The card ending in 0366 belongs to J. S. — confirm before charging.": [ + ("0366", "CREDIT_CARD") + ], "Patient initials M.G., born 1982, seen last Tuesday.": [("M.G.", "PERSON")], "Reach the account holder on the number ending 0132.": [("0132", "PHONE_NUMBER")], "SSN last four is 6789 for the applicant from Ohio.": [("6789", "US_SSN")], - "Reach him at j dot smith at acme dot com when you can.": [("j dot smith at acme dot com", "EMAIL_ADDRESS")], - "Contact: j . s m i t h @ a c m e . c o m": [("j . s m i t h @ a c m e . c o m", "EMAIL_ADDRESS")], - "Card number four five three two 0151 1283 0366.": [("four five three two 0151 1283 0366", "CREDIT_CARD")], + "Reach him at j dot smith at acme dot com when you can.": [ + ("j dot smith at acme dot com", "EMAIL_ADDRESS") + ], + "Contact: j . s m i t h @ a c m e . c o m": [ + ("j . s m i t h @ a c m e . c o m", "EMAIL_ADDRESS") + ], + "Card number four five three two 0151 1283 0366.": [ + ("four five three two 0151 1283 0366", "CREDIT_CARD") + ], "Phone: four one five, five five five, zero one three two.": [ ("four one five, five five five, zero one three two", "PHONE_NUMBER") ], - "La tarjeta terminada en 0366 pertenece a J. M. — confirmar antes de cobrar.": [("0366", "CREDIT_CARD")], + "La tarjeta terminada en 0366 pertenece a J. M. — confirmar antes de cobrar.": [ + ("0366", "CREDIT_CARD") + ], "Paciente con iniciales L. F., nacida en 1982.": [("L. F.", "PERSON")], - "Su correo es j punto smith arroba acme punto com.": [("j punto smith arroba acme punto com", "EMAIL_ADDRESS")], - "Teléfono: seis uno dos, tres cuatro, cinco seis.": [("seis uno dos, tres cuatro, cinco seis", "PHONE_NUMBER")], - "La carte se terminant par 0366 appartient à J. D. — à confirmer.": [("0366", "CREDIT_CARD")], + "Su correo es j punto smith arroba acme punto com.": [ + ("j punto smith arroba acme punto com", "EMAIL_ADDRESS") + ], + "Teléfono: seis uno dos, tres cuatro, cinco seis.": [ + ("seis uno dos, tres cuatro, cinco seis", "PHONE_NUMBER") + ], + "La carte se terminant par 0366 appartient à J. D. — à confirmer.": [ + ("0366", "CREDIT_CARD") + ], "Patiente aux initiales M. L., née en 1982.": [("M. L.", "PERSON")], - "Son adresse est j point smith arobase acme point com.": [("j point smith arobase acme point com", "EMAIL_ADDRESS")], - "Téléphone : zéro six, douze, trente-quatre.": [("zéro six, douze, trente-quatre", "PHONE_NUMBER")], - "Die Karte endend auf 0366 gehört H. M. — bitte bestätigen.": [("0366", "CREDIT_CARD")], + "Son adresse est j point smith arobase acme point com.": [ + ("j point smith arobase acme point com", "EMAIL_ADDRESS") + ], + "Téléphone : zéro six, douze, trente-quatre.": [ + ("zéro six, douze, trente-quatre", "PHONE_NUMBER") + ], + "Die Karte endend auf 0366 gehört H. M. — bitte bestätigen.": [ + ("0366", "CREDIT_CARD") + ], "Patientin mit den Initialen A. S., geboren 1982.": [("A. S.", "PERSON")], - "Seine Adresse ist j punkt smith at acme punkt com.": [("j punkt smith at acme punkt com", "EMAIL_ADDRESS")], - "Telefon: null eins fünf eins, zwei drei vier.": [("null eins fünf eins, zwei drei vier", "PHONE_NUMBER")], + "Seine Adresse ist j punkt smith at acme punkt com.": [ + ("j punkt smith at acme punkt com", "EMAIL_ADDRESS") + ], + "Telefon: null eins fünf eins, zwei drei vier.": [ + ("null eins fünf eins, zwei drei vier", "PHONE_NUMBER") + ], "尾号 0366 的卡属于张先生，扣款前请确认。": [("0366", "CREDIT_CARD")], "患者姓名缩写 W.F.，1982 年出生。": [("W.F.", "PERSON")], - "他的邮箱是 j 点 smith 艾特 acme 点 com。": [("j 点 smith 艾特 acme 点 com", "EMAIL_ADDRESS")], + "他的邮箱是 j 点 smith 艾特 acme 点 com。": [ + ("j 点 smith 艾特 acme 点 com", "EMAIL_ADDRESS") + ], "电话：一三八 零零一三 八零零零。": [("一三八 零零一三 八零零零", "PHONE_NUMBER")], } -def build(per_template: int = 6, languages: Optional[Iterable[str]] = None) -> List[Item]: +def build( + per_template: int = 6, languages: Optional[Iterable[str]] = None +) -> List[Item]: """Generate the corpus. Deterministic for a fixed SEED.""" rng = random.Random(SEED) langs = list(languages or LANGUAGES) @@ -435,7 +517,11 @@ def build(per_template: int = 6, languages: Optional[Iterable[str]] = None) -> L for value, etype in truth: idx = template.find(value) if idx >= 0: - spans.append(Span(start=idx, end=idx + len(value), type=etype, value=value)) + spans.append( + Span( + start=idx, end=idx + len(value), type=etype, value=value + ) + ) items.append( Item( id=f"{lang}-{tier}-{counter:04d}", diff --git a/bench/run.py b/bench/run.py index 5846614..c058c01 100644 --- a/bench/run.py +++ b/bench/run.py @@ -75,16 +75,19 @@ def arm_legacy(text: str, lang: str) -> Tuple[str, List[str]]: Kept so before/after numbers come out of a single run rather than being compared across two trees. """ - from app.policies import ANALYZER, ANONYMIZER - from app.policies import entity_type_to_placeholder, is_false_positive from presidio_anonymizer.entities import OperatorConfig + from app.policies import ( + ANALYZER, + ANONYMIZER, + entity_type_to_placeholder, + is_false_positive, + ) + if ANALYZER is None: return text, [] try: - results = ANALYZER.analyze( - text=text, entities=LEGACY_ENTITIES, language="en" - ) + results = ANALYZER.analyze(text=text, entities=LEGACY_ENTITIES, language="en") except Exception: return text, [] # The pre-fix code passed the whole text to the false-positive filter @@ -117,9 +120,10 @@ def arm_presidio_unrestricted(text: str, lang: str) -> Tuple[str, List[str]]: alone — i.e. detection the loaded model already performs and the pipeline then discards. """ - from app.policies import ANALYZER, ANONYMIZER, entity_type_to_placeholder from presidio_anonymizer.entities import OperatorConfig + from app.policies import ANALYZER, ANONYMIZER, entity_type_to_placeholder + if ANALYZER is None: return text, [] results = ANALYZER.analyze(text=text, language="en") @@ -177,7 +181,9 @@ def __init__(self) -> None: self.clean_total = 0 self.clean_modified = 0 self.latencies: List[float] = [] - self.by_entity: Dict[str, List[int]] = defaultdict(lambda: [0, 0]) # [leaked, total] + self.by_entity: Dict[str, List[int]] = defaultdict( + lambda: [0, 0] + ) # [leaked, total] def rate(self, num: int, den: int) -> Optional[float]: return round(num / den, 4) if den else None @@ -218,7 +224,10 @@ def run_arm(name: str, arm: Arm, items: List[Item]) -> Dict[str, object]: try: out, _reasons = arm(item.text, item.lang) except Exception as exc: # an arm that crashes leaks everything - print(f" ! {name} raised on {item.id}: {type(exc).__name__}: {exc}", file=sys.stderr) + print( + f" ! {name} raised on {item.id}: {type(exc).__name__}: {exc}", + file=sys.stderr, + ) out = item.text elapsed_ms = (time.perf_counter() - t0) * 1000 @@ -248,11 +257,17 @@ def run_arm(name: str, arm: Arm, items: List[Item]) -> Dict[str, object]: return { "overall": overall.summary(), - "high_sensitivity_leakage": overall.rate(high_sens.spans_leaked, high_sens.spans_total), + "high_sensitivity_leakage": overall.rate( + high_sens.spans_leaked, high_sens.spans_total + ), "by_lang": {k: v.summary() for k, v in sorted(by_lang.items())}, "by_tier": {k: v.summary() for k, v in sorted(by_tier.items())}, "by_entity": { - k: {"leaked": v[0], "total": v[1], "leakage": round(v[0] / v[1], 4) if v[1] else None} + k: { + "leaked": v[0], + "total": v[1], + "leakage": round(v[0] / v[1], 4) if v[1] else None, + } for k, v in sorted(overall.by_entity.items()) }, } @@ -266,7 +281,9 @@ def print_report(results: Dict[str, Dict[str, object]]) -> None: arms = list(results.keys()) print("\n=== Overall (leakage = ground-truth PII values surviving verbatim) ===\n") - print(f"{'arm':<24}{'leakage':>9}{'item_leak':>11}{'high_sens':>11}{'over_redact':>13}{'p50 ms':>9}{'p95 ms':>9}") + print( + f"{'arm':<24}{'leakage':>9}{'item_leak':>11}{'high_sens':>11}{'over_redact':>13}{'p50 ms':>9}{'p95 ms':>9}" + ) for arm in arms: o = results[arm]["overall"] print( @@ -280,7 +297,10 @@ def print_report(results: Dict[str, Dict[str, object]]) -> None: langs = sorted({lang for arm in arms for lang in results[arm]["by_lang"]}) print(f"{'arm':<24}" + "".join(f"{lang:>9}" for lang in langs)) for arm in arms: - row = "".join(fmt_pct(results[arm]["by_lang"].get(lang, {}).get("leakage")) for lang in langs) + row = "".join( + fmt_pct(results[arm]["by_lang"].get(lang, {}).get("leakage")) + for lang in langs + ) print(f"{arm:<24}{row}") print("\n=== Leakage by tier ===\n") @@ -288,7 +308,8 @@ def print_report(results: Dict[str, Dict[str, object]]) -> None: print(f"{'arm':<24}" + "".join(f"{t[:9]:>18}" for t in tiers)) for arm in arms: row = "".join( - f"{fmt_pct(results[arm]['by_tier'].get(t, {}).get('leakage')):>18}" for t in tiers + f"{fmt_pct(results[arm]['by_tier'].get(t, {}).get('leakage')):>18}" + for t in tiers ) print(f"{arm:<24}{row}") @@ -312,7 +333,9 @@ def main() -> int: args = ap.parse_args() items = build(per_template=args.per_template, languages=args.languages) - print(f"corpus: {len(items)} items, {sum(len(i.spans) for i in items)} annotated spans") + print( + f"corpus: {len(items)} items, {sum(len(i.spans) for i in items)} annotated spans" + ) results: Dict[str, Dict[str, object]] = {} for name in args.arms: diff --git a/tests/test_deny_tools_matching.py b/tests/test_deny_tools_matching.py index fcbe883..76d1fc0 100644 --- a/tests/test_deny_tools_matching.py +++ b/tests/test_deny_tools_matching.py @@ -62,7 +62,13 @@ def test_denied(self, tool): @pytest.mark.parametrize( "tool", - ["subprocess.run", "subprocess.popen", "os.system", "child_process.spawn", "runtime.exec"], + [ + "subprocess.run", + "subprocess.popen", + "os.system", + "child_process.spawn", + "runtime.exec", + ], ) def test_semantically_equivalent_primitives_denied(self, tool): """Names that are code execution by any other spelling.""" diff --git a/tests/test_multilingual_pii.py b/tests/test_multilingual_pii.py index cd8b4ae..1418a49 100644 --- a/tests/test_multilingual_pii.py +++ b/tests/test_multilingual_pii.py @@ -32,7 +32,8 @@ def _has_model(name: str) -> bool: requires_presidio = pytest.mark.skipif( - policies.ANALYZER is None, reason="Presidio analyzer unavailable in this environment" + policies.ANALYZER is None, + reason="Presidio analyzer unavailable in this environment", ) @@ -97,7 +98,9 @@ def test_falls_back_to_policy_config(self): assert language_hint(None, {"language": "fr"}) == "fr" def test_tool_config_wins(self): - assert language_hint({"metadata": {"language": "es"}}, {"language": "fr"}) == "es" + assert ( + language_hint({"metadata": {"language": "es"}}, {"language": "fr"}) == "es" + ) @pytest.mark.parametrize( "tool_config,policy_config", @@ -120,7 +123,9 @@ def test_language_list_parsed_and_deduped(self): assert s.presidio_language_list() == ["en", "es", "fr"] def test_empty_language_list_defaults_to_english(self): - assert Settings(presidio_languages=" , ", debug=True).presidio_language_list() == ["en"] + assert Settings( + presidio_languages=" , ", debug=True + ).presidio_language_list() == ["en"] def test_invalid_language_mode_rejected(self): with pytest.raises(ValueError, match="PRESIDIO_LANGUAGE_MODE"): @@ -128,7 +133,10 @@ def test_invalid_language_mode_rejected(self): @pytest.mark.parametrize("mode", ["hint", "union"]) def test_valid_language_modes_accepted(self, mode): - assert Settings(presidio_language_mode=mode, debug=True).presidio_language_mode == mode + assert ( + Settings(presidio_language_mode=mode, debug=True).presidio_language_mode + == mode + ) @pytest.mark.multilingual @@ -138,9 +146,24 @@ class TestNonEnglishRedaction: @pytest.mark.parametrize( "lang,model,text,secret", [ - ("es", "es_core_news_sm", "El paciente Juan Martínez necesita revisión.", "Juan Martínez"), - ("fr", "fr_core_news_sm", "Le patient Jean Dupont doit être revu.", "Jean Dupont"), - ("de", "de_core_news_sm", "Der Patient Hans Müller braucht eine Untersuchung.", "Hans Müller"), + ( + "es", + "es_core_news_sm", + "El paciente Juan Martínez necesita revisión.", + "Juan Martínez", + ), + ( + "fr", + "fr_core_news_sm", + "Le patient Jean Dupont doit être revu.", + "Jean Dupont", + ), + ( + "de", + "de_core_news_sm", + "Der Patient Hans Müller braucht eine Untersuchung.", + "Hans Müller", + ), ], ) def test_person_redacted_in_language(self, lang, model, text, secret): From d008cb5407185ba3477fcada9ea69d7527124e5a Mon Sep 17 00:00:00 2001 From: Shaishav Pidadi Date: Tue, 11 Aug 2026 22:15:27 -0400 Subject: [PATCH 3/4] ci: fix pre-existing Format/Type/Load failures on dev MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three of these fail on dev at 351d59b independently of this branch — verified by checking out dev and running the CI commands. They are in this PR only because a red CI blocks the merge; the commit is self-contained and can be dropped or split out if you would rather land it separately. - black + isort: app/api.py, app/policy_source.py, app/storage.py, tests/test_policy_invalidate_endpoint.py, tests/test_policy_source.py. Mechanical reformatting, no behaviour change. - mypy: app/policy_source.py:75 "Need type annotation for raw_defaults". Annotated Any rather than Dict[str, Any] because row.defaults is a SQLAlchemy Column[Any] at type-check time and a concrete dict annotation trades the missing-annotation error for an incompatible-assignment one. - Load Test: the job sets DEBUG=false, so Settings runs its production validators, but PII_TOKEN_SALT was "ci-load-salt" (13 chars) against a 32-character minimum. The job has been failing at startup ever since that validator landed. Padded all three throwaway CI values past the minimum. Verified locally against the exact CI commands: black --check, isort --check-only, flake8 with the workflow's flags, and mypy app/ all pass. --- .github/workflows/ci.yml | 9 +++-- app/api.py | 9 ++++- app/policy_source.py | 46 +++++++++++++----------- app/storage.py | 4 +-- tests/test_policy_invalidate_endpoint.py | 1 + tests/test_policy_source.py | 43 ++++++++++++++++++---- 6 files changed, 77 insertions(+), 35 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d90f594..d940067 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,10 +79,13 @@ jobs: env: APP_BIND: 127.0.0.1:8082 DB_URL: sqlite:///./loadtest.db + # DEBUG=false so the load test exercises the production settings path, + # which means these must clear the 32-character minimum in + # Settings._validate_secret. Throwaway CI values, not secrets. DEBUG: "false" - KEY_HMAC_SECRET: ci-load-hmac-secret - PII_TOKEN_SALT: ci-load-salt - WEBHOOK_SECRET: ci-load-webhook-secret + KEY_HMAC_SECRET: ci-load-hmac-secret-padded-to-32-chars-min + PII_TOKEN_SALT: ci-load-token-salt-padded-to-32-chars-min + WEBHOOK_SECRET: ci-load-webhook-secret-padded-to-32-chars-min PRECHECK_DLQ: /tmp/precheck-load.dlq.jsonl LOAD_TEST_API_KEY: GAI_ci_load_test_key PRECHECK_BASE_URL: http://127.0.0.1:8082 diff --git a/app/api.py b/app/api.py index f1a5a06..71dacb5 100644 --- a/app/api.py +++ b/app/api.py @@ -8,7 +8,14 @@ from datetime import datetime from typing import List, Optional, Tuple -from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request, Response +from fastapi import ( + APIRouter, + BackgroundTasks, + Depends, + HTTPException, + Request, + Response, +) from sqlalchemy.orm import Session from .auth import AuthContext, require_api_key diff --git a/app/policy_source.py b/app/policy_source.py index f3a0952..fc761a2 100644 --- a/app/policy_source.py +++ b/app/policy_source.py @@ -11,6 +11,7 @@ already understands (precheck/app/policies.py), so the evaluator does not need to learn a new format — we just translate at the edge. """ + from __future__ import annotations import logging @@ -18,7 +19,7 @@ import threading import time from dataclasses import dataclass -from typing import Optional +from typing import Any, Dict, Optional from sqlalchemy.orm import Session @@ -55,13 +56,13 @@ class _CacheEntry: # shape `defaults[direction]["action"]` with action ∈ # {"deny", "redact", "tokenize", "pass_through"}. _PII_ACTION_MAP = { - "redact": "redact", - "block": "deny", - "deny": "deny", - "tokenize": "tokenize", - "pass": "pass_through", + "redact": "redact", + "block": "deny", + "deny": "deny", + "tokenize": "tokenize", + "pass": "pass_through", "pass_through": "pass_through", - "allow": "pass_through", + "allow": "pass_through", } @@ -72,7 +73,10 @@ def _map_row_to_policy_config(row: DashboardPolicy) -> dict: (precheck/app/policies.py). Unknown keys in `row.defaults` are preserved verbatim so future extensions don't need to touch this function. """ - raw_defaults = row.defaults or {} + # Annotated Any rather than Dict[str, Any]: `row.defaults` is a SQLAlchemy + # Column[Any] at type-check time, so a concrete dict annotation conflicts + # with the assignment even though the runtime value is a dict. + raw_defaults: Any = row.defaults or {} # Translate the v1 convention; default to "redact" if the field is absent. pii_action_in = str(raw_defaults.get("pii", "redact")).lower() @@ -82,20 +86,20 @@ def _map_row_to_policy_config(row: DashboardPolicy) -> dict: "version": row.version or "v1", "defaults": { "ingress": {"action": pii_action}, - "egress": {"action": pii_action}, + "egress": {"action": pii_action}, # Forward any non-pii defaults verbatim for forward-compat. **{k: v for k, v in raw_defaults.items() if k != "pii"}, }, - "tool_access": row.tool_access or {}, - "deny_tools": row.deny_tools or [], - "allow_tools": row.allow_tools or [], + "tool_access": row.tool_access or {}, + "deny_tools": row.deny_tools or [], + "allow_tools": row.allow_tools or [], "network_scopes": row.network_scopes or [], - "network_tools": row.network_tools or [], - "on_error": row.on_error or "block", + "network_tools": row.network_tools or [], + "on_error": row.on_error or "block", # Provenance — useful in logs and audit, ignored by the evaluator. - "_policy_id": row.id, - "_policy_name": row.name, - "_priority": row.priority, + "_policy_id": row.id, + "_policy_name": row.name, + "_priority": row.priority, } @@ -111,7 +115,9 @@ def _fetch_from_db(org_id: str) -> Optional[dict]: DashboardPolicy.org_id == org_id, DashboardPolicy.is_active.is_(True), ) - .order_by(DashboardPolicy.priority.desc(), DashboardPolicy.updated_at.desc()) + .order_by( + DashboardPolicy.priority.desc(), DashboardPolicy.updated_at.desc() + ) .first() ) if row is None: @@ -120,9 +126,7 @@ def _fetch_from_db(org_id: str) -> Optional[dict]: except Exception as exc: # Don't blow up the request path on a DB hiccup — let the caller fall # back to the YAML policy. Logged loudly so it's visible in audits. - logger.warning( - "policy_source: db fetch failed for org=%s err=%s", org_id, exc - ) + logger.warning("policy_source: db fetch failed for org=%s err=%s", org_id, exc) return None finally: db.close() diff --git a/app/storage.py b/app/storage.py index f34dc1a..0abb3da 100644 --- a/app/storage.py +++ b/app/storage.py @@ -149,9 +149,7 @@ def create_tables(): `DashboardPolicy` is intentionally excluded — that table is owned and migrated by the dashboard (Prisma). Precheck only reads it. See ADR-005. """ - owned_tables = [ - t for t in Base.metadata.sorted_tables if t.name != "Policy" - ] + owned_tables = [t for t in Base.metadata.sorted_tables if t.name != "Policy"] Base.metadata.create_all(bind=engine, tables=owned_tables) diff --git a/tests/test_policy_invalidate_endpoint.py b/tests/test_policy_invalidate_endpoint.py index 255853b..afb0b7b 100644 --- a/tests/test_policy_invalidate_endpoint.py +++ b/tests/test_policy_invalidate_endpoint.py @@ -1,5 +1,6 @@ # SPDX-License-Identifier: MIT """HMAC-gated /api/v1/internal/policy/invalidate endpoint tests (ADR-005).""" + from __future__ import annotations import hashlib diff --git a/tests/test_policy_source.py b/tests/test_policy_source.py index d0ad0e9..34b6642 100644 --- a/tests/test_policy_source.py +++ b/tests/test_policy_source.py @@ -4,6 +4,7 @@ See ADR-005 for design. Tests cover cache hit/miss/expiry, missing-org, priority ordering, the dashboard-shape → precheck-shape translation, and invalidate(). """ + from __future__ import annotations import time @@ -117,8 +118,12 @@ def test_unknown_pii_action_falls_back_to_redact(db_session): # ─── priority order ─────────────────────────────────────────────────────── def test_highest_priority_active_policy_wins(db_session): - _make_policy(db_session, org_id="org-9", name="lo", defaults={"pii": "redact"}, priority=1) - _make_policy(db_session, org_id="org-9", name="hi", defaults={"pii": "block"}, priority=10) + _make_policy( + db_session, org_id="org-9", name="lo", defaults={"pii": "redact"}, priority=1 + ) + _make_policy( + db_session, org_id="org-9", name="hi", defaults={"pii": "block"}, priority=10 + ) cfg = policy_source.get_active_policy("org-9") # priority=10 ("block") must beat priority=1 ("redact") assert cfg["defaults"]["ingress"]["action"] == "deny" @@ -126,8 +131,22 @@ def test_highest_priority_active_policy_wins(db_session): def test_inactive_policy_ignored(db_session): - _make_policy(db_session, org_id="org-10", name="dead", defaults={"pii": "block"}, priority=999, is_active=False) - _make_policy(db_session, org_id="org-10", name="live", defaults={"pii": "redact"}, priority=1, is_active=True) + _make_policy( + db_session, + org_id="org-10", + name="dead", + defaults={"pii": "block"}, + priority=999, + is_active=False, + ) + _make_policy( + db_session, + org_id="org-10", + name="live", + defaults={"pii": "redact"}, + priority=1, + is_active=True, + ) cfg = policy_source.get_active_policy("org-10") assert cfg["_policy_name"] == "live" @@ -159,13 +178,19 @@ def test_invalidate_forces_refetch(db_session): assert first["defaults"]["ingress"]["action"] == "redact" # mutate the underlying row - row = db_session.query(DashboardPolicy).filter(DashboardPolicy.org_id == "org-i").one() + row = ( + db_session.query(DashboardPolicy) + .filter(DashboardPolicy.org_id == "org-i") + .one() + ) row.defaults = {"pii": "block"} db_session.commit() # without invalidation, the cache should still serve the old value cached = policy_source.get_active_policy("org-i") - assert cached["defaults"]["ingress"]["action"] == "redact", "cache should still hold stale entry" + assert ( + cached["defaults"]["ingress"]["action"] == "redact" + ), "cache should still hold stale entry" # invalidate → next call refetches and reflects the change policy_source.invalidate("org-i") @@ -189,7 +214,11 @@ def test_ttl_expiry_forces_refetch(db_session, monkeypatch): monkeypatch.setattr(time, "monotonic", lambda: base + 5) # mutate the row; without a hit-spy, the new value proves a refetch happened - row = db_session.query(DashboardPolicy).filter(DashboardPolicy.org_id == "org-t").one() + row = ( + db_session.query(DashboardPolicy) + .filter(DashboardPolicy.org_id == "org-t") + .one() + ) row.defaults = {"pii": "block"} db_session.commit() From 5e9c51ea5b601f7e8d22eec1434600ddda17f9f3 Mon Sep 17 00:00:00 2001 From: Shaishav Pidadi Date: Tue, 11 Aug 2026 22:24:09 -0400 Subject: [PATCH 4/4] ci: stop the load test measuring its own rate limiter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Padding the CI secrets got the service past startup, which revealed the next failure underneath: k6 reported 96.23% failed requests, 113 of 3000 succeeding. The limiter keys on hash_api_key(raw_key) and the load test sends every request with the same LOAD_TEST_API_KEY — LOAD_USER_POOL_SIZE=120 only varies the user_id inside the payload. Against the 100/min per-key default, the profile (100 iters/s for 30s) can only ever get ~100 requests through before the rest are 429ed. Simulating specs_for_request over the k6 profile: key/min=100 org/min=1000 -> allowed 100/3000 key/min=20000 org/min=20000 -> allowed 3000/3000 The 113 observed in CI is the 100 from the first window plus the handful that landed after the 30s run crossed a minute boundary. Raises the four rate-limit dimensions for this job only, so the run exercises the precheck hot path rather than the limiter in front of it. Note this is a judgment call about what the job is for: if the intent was to assert limiter behaviour under load, the fix is the opposite one — give k6 a pool of API keys and assert the 429s. Happy to swap it. --- .github/workflows/ci.yml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d940067..7c0a763 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -90,6 +90,17 @@ jobs: LOAD_TEST_API_KEY: GAI_ci_load_test_key PRECHECK_BASE_URL: http://127.0.0.1:8082 LOAD_USER_POOL_SIZE: "120" + # The k6 profile is 100 iters/s for 30s (3000 requests) and every one of + # them carries the same API key — LOAD_USER_POOL_SIZE only varies the + # user_id inside the payload, while the limiter keys on + # hash_api_key(raw_key). Against the 100/min per-key default that lets + # roughly 113 requests through and 429s the rest, which is what the job + # has been measuring. Raised so the run exercises the precheck hot path + # rather than its own rate limiter. + RATE_LIMIT_REQUESTS_PER_MINUTE: "20000" + RATE_LIMIT_ORG_REQUESTS_PER_MINUTE: "20000" + RATE_LIMIT_TOKENS_PER_MINUTE: "10000000" + RATE_LIMIT_ORG_TOKENS_PER_MINUTE: "10000000" steps: - uses: actions/checkout@v4