From 1543d24908848f84d47d83eefd7335e373c09c22 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Wed, 22 Jul 2026 00:50:09 +0300 Subject: [PATCH] Enforce Cloud Armor metadata rule; keep calculate in preview MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit metadata (30/60s) enforced 2026-07-21: worst legit per-IP rate observed in the preview window was 5/min, 6x under the threshold. calculate (75/60s) stays in preview — observed legit/partner clients run 39-88/min there, so enforcing would 429 real use including the partner's API fallback. Also documents that Armor throttle rules log only CONFORM in preview, so enforcement readiness must be judged from raw LB-log rate analysis. Co-Authored-By: Claude Fable 5 --- changelog.d/armor-enforce-metadata.changed.md | 1 + ...T214911Z-pol-api-lb-metadata-enforced.yaml | 50 +++++++++++++++++++ docs/migration/lb-cloud-armor-runbook.md | 8 ++- 3 files changed, 58 insertions(+), 1 deletion(-) create mode 100644 changelog.d/armor-enforce-metadata.changed.md create mode 100644 docs/migration/armor/20260721T214911Z-pol-api-lb-metadata-enforced.yaml diff --git a/changelog.d/armor-enforce-metadata.changed.md b/changelog.d/armor-enforce-metadata.changed.md new file mode 100644 index 000000000..17218a138 --- /dev/null +++ b/changelog.d/armor-enforce-metadata.changed.md @@ -0,0 +1 @@ +Enforce the Cloud Armor metadata rate-limit rule; keep the calculate rule in preview. diff --git a/docs/migration/armor/20260721T214911Z-pol-api-lb-metadata-enforced.yaml b/docs/migration/armor/20260721T214911Z-pol-api-lb-metadata-enforced.yaml new file mode 100644 index 000000000..33b8db4c9 --- /dev/null +++ b/docs/migration/armor/20260721T214911Z-pol-api-lb-metadata-enforced.yaml @@ -0,0 +1,50 @@ +creationTimestamp: '2026-07-21T07:16:39.177-07:00' +description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md +fingerprint: dT_5QmeI6h8= +id: '200433714284642728' +kind: compute#securityPolicy +labelFingerprint: 42WmSpB8rSM= +name: pol-api-lb +rules: +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/metadata$') + preview: false + priority: 1000 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 30 + intervalSec: 60 +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$') + preview: true + priority: 1100 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 75 + intervalSec: 60 +- action: allow + description: default rule + kind: compute#securityPolicyRule + match: + config: + srcIpRanges: + - '*' + versionedExpr: SRC_IPS_V1 + preview: false + priority: 2147483647 +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb +type: CLOUD_ARMOR diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index 35c412bea..f8f9014df 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -12,7 +12,13 @@ record of what is currently deployed is the newest exported snapshot in |---|---| | Policy | `pol-api-lb`, attached to both backend services of the public API LB | | Rules | Per-IP throttles on the metadata and calculate path families; thresholds in the snapshot | -| Mode | Created in preview 2026-07-21; enforcement pending the gates below | +| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **stays in preview at 75/60s** — observed legit/partner clients run 39–88/min there, so enforcing would 429 real use (incl. the partner API fallback) | + +Note on the preview gate: Cloud Armor **throttle** rules in preview only ever +log `CONFORM` (never `EXCEEDED`), so the `previewSecurityPolicy` outcome field +cannot be used to count would-be-throttled requests. Judge a throttle rule's +enforcement readiness from **raw per-IP request-rate analysis of LB logs**, not +from preview outcome counts. Origin: overnight bot waves saturated backends / churned autoscaling (2026-07-13 → 2026-07-21 incidents; details in the cutover execution plan).