WIP perf(graphile): harden tenant density without SQL rewrite

[yyyyaaa]

Caution

This is a research draft, not a production-ready change. The measured data-plane design is promising, but production remains blocked on an atomic/versioned tenant security contract, WebSocket revocation, production policy proof, upstream Graphile review, and final workload qualification.

Summary

This draft supersedes the old Graphile scale stack #1330–#1335 and rebuilds its useful work on origin/main at 2331ba016, while rejecting the part that made tenant isolation depend on runtime SQL rewriting and shared tenant blueprints.

The objective is complete resident customer tenants per GiB of memory. A 1 GiB heap is one normalization point in the planned capacity matrix, not a product constraint.

Zhi's SQL-rewrite concern is sound: RLS filters rows after PostgreSQL resolves an object, so it cannot prove that a rewritten query reached the correct schema, function, sequence, index, metadata object, or privileged code path. Dan's plugin, BM25, and GUC concerns are also valid review requirements. This candidate addresses them by removing runtime rewrite/pooling entirely:

• there is no runtime SQL schema rewriting;
• there is no blueprint sharing between tenants;
• each exact physical tenant/API build contract gets its own PostGraphile instance and runtime pool identity;
• schema-scoped introspection only narrows build-time catalog input and is never an authorization or routing mechanism;
• BM25 remains enabled and binds physical schema-qualified index names; and
• stock introspection remains the default, while scoped-required fails closed and never silently falls back.

Why this approach is better than the old stack

Concern	Old #1330–#1335 direction	Current candidate	Consequence
Tenant routing	Share a canonical blueprint, then rewrite schema-qualified SQL	Compile and execute against the exact physical database and ordered schemas	A missed identifier or raw plugin query cannot silently bypass a rewrite layer that does not exist.
Memory reduction	Reuse runtime execution artifacts across tenants	Reduce the catalog retained during each dedicated instance build	The large measured saving comes from removing irrelevant introspection input, without weakening the physical execution boundary.
RLS	Could be mistaken for proof that rewritten SQL routed correctly	Defense in depth behind exact pool/build identity and least-privilege roles	RLS still limits rows, but routing, metadata, functions, sequences, and indexes do not depend on it.
Plugins/raw SQL	Every path would need to participate correctly in rewriting	Built-ins compile against real physical schemas; caller plugins are denied by default in production	Plugin behavior is explicit trusted code rather than an implicit rewrite-coverage promise.
BM25	Pooling/rewrite compatibility created exclusions	Enabled, with physical schema-qualified regclass-like values	Search remains part of the tenant surface instead of being skipped to make a benchmark pass.
GUC and prepared state	Session-local state could outlive assumptions or revert after a transaction	Complete per-request GUC initialization, explicit request transactions, checkout DISCARD ALL, prepared-cache clearing, and client destruction on reset failure	SET LOCAL restoration is treated as expected PostgreSQL behavior, while every reused checkout starts from a sanitized state.
API separation	Similar APIs could be grouped by a fingerprint	The build HMAC includes the exact database/API IDs, pool identity, schemas, roles, resolved settings/plugins, bindings, surfaces, and introspection mode	Host/service names remain routing labels and can never cause two different security contracts to share an instance.

This gives up the old stack's most aggressive same-blueprint sharing claim. That is deliberate: density is only useful when tenant isolation remains at least as strong as today's exact physical boundary.

Progress

• Refresh the research candidate onto fetched origin/main snapshot 2331ba016.
• Reapply valid quoting, plugin isolation, i18n, LLM/RAG, search, storage, bulk, PostGIS, ltree, function-binding, and realtime fixes.
• Add exact build/pool identities, optional least-privilege runtime credentials, complete request GUCs, checkout sanitation, and strict runtime-role admission.
• Harden the cache governor with serialized builds, awaited disposal, bounded draining, pressure refusal, stable 503 codes, and observability.
• Replace SQL-text introspection substitution with a parameterized, schema-scoped, dependency-closed candidate that fails closed.
• Refresh the cperf and hostile tenant fixtures, and record the current benchmark/security evidence below.
• Audit the current approach against Zhi's and Dan's SQL rewrite, plugin, BM25, prepared-state, and GUC concerns.
• Add TenantSecurityContractV1 with one atomic, monotonic revision across routing and all security metadata.
• Revalidate every WebSocket operation against the current revision and terminate stale active subscriptions.
• Prove the production RLS/role/dependency manifest from disposable copies of the real schemas.
• Make authRequired an authoritative routed API contract.
• Replace the local Graphile source patches with an upstream-reviewed introspection API.
• Refresh once more before review/merge: main advanced to d66ae8e18 after this branch was pushed, and GitHub currently reports merge conflicts. Per this draft's scope, parity is deferred rather than hidden.
• Rerun full current-main parity, external-provider/storage gates, the repeated 1/2/4 GiB complete-customer matrix, 15-minute workloads, and the two-hour churn soak.

Commit-by-commit review guide

The branch is split into ten dependency-ordered commits. Each exact commit passes CI=1 pnpm install --frozen-lockfile --ignore-scripts, and the final tree is byte-identical to the previously validated three-commit draft.

1. ab50c1b42 — plugin SQL quoting and metadata/build-cache isolation across bulk, i18n, LLM/RAG, search, ltree, PostGIS, and function bindings.
2. 2899a8724 — exact PostgreSQL pool identities, checkout sanitation, notification roles, and request query context.
3. 86a80e85a — tenant-isolated storage module discovery, physical bucket coordinates, credentials, signing, and RLS-aware access.
4. a76d05882 — request-scoped tenant context and fail-closed security/configuration loaders.
5. 6aad36dea — Graphile instance cache governance plus realtime resource and generation lifecycle management.
6. 817fdc4ad — exact Graphile build contracts and fail-closed schema-scoped introspection, including the isolated upstream patch candidate.
7. 1ed6d5868 — server integration and tenant request admission: runtime-role checks, internal headers, CAPTCHA, WebSockets, routing, and observability.
8. 3f40abd92 — attested cperf harness, workload scoring, process isolation, and report generation.
9. 93da3a662 — hostile complete-tenant and physical-database density fixtures.
10. 666e2a038 — security audit, performance evidence, original-stack disposition, and upstream/qualification notes.

What this draft includes

• Opaque pool and GraphileBuildContractV1 identities over the complete tenant/API execution contract, never hostname alone.
• Runtime checkout sanitation with DISCARD ALL, node-postgres/Graphile prepared-statement bookkeeping reset, and client destruction on failure.
• Strict rejection of superuser, BYPASSRLS, CREATEROLE, ownership, schema CREATE, unsafe inheritance/SET ROLE, unsafe SECURITY DEFINER, and cross-schema dependency capabilities.
• Production-default denial of caller Graphile presets/plugins, because arbitrary plugins are unrestricted Node/database trusted code.
• Heap-aware cache admission/eviction and resident-handler preservation under pressure.
• Parameterized schema-scoped introspection with required-schema assertions and catalog dependency closure.
• Fail-closed storage/config cache scoping, internal routing headers, runtime credential resolution, CAPTCHA operation admission, and WebSocket mutation admission.
• A cperf harness plus complete-tenant and physical-database hostile fixtures that treat missing surfaces, canaries, telemetry, or repetitions as failures.

Detailed notes live in:

• research/graphile-density/REPORT.md
• research/graphile-density/SECURITY-AUDIT.md
• research/graphile-density/PLUGIN-SQL-AUDIT.md
• research/graphile-density/UPSTREAM-REVIEW.md

Confirmed performance evidence

The clean benchmark used a production-shaped catalog with 62,298 pg_class rows, 346,369 attributes, 8,496 procedures, 23,709 types, and 4,036 namespaces. Each arm ran in three fresh Node processes with a 4 GiB V8 heap and warmed 100 distinct Grafast operations plus three exact replay passes.

Metric	Stock introspection	Scoped dependency introspection	Change
Median retained heap after forced GC	471,256,024 bytes / 449.42 MiB	6,869,560 bytes / 6.55 MiB	68.60x less
Median final Node RSS delta	1,160,708,096 bytes / 1,106.94 MiB	50,085,888 bytes / 47.77 MiB	23.17x less
Median cold build readiness	3,495.95 ms	130.34 ms	26.82x faster

Both arms emitted the same 17,976-byte GraphQL SDL with SHA-256 5fb82f96153815b23820a9ccf10322a20c864e49605ef5781cd33422b3b31020, and the recorded operations had zero errors, mismatches, or cross-tenant tokens.

The raw clean-run summaries remain local/gitignored at:

• research/graphile-density/artifacts/production-shaped-20260801-v6-clean/stock-release-destroy-clean-h4096-r3/summary.json
• research/graphile-density/artifacts/production-shaped-20260801-v6-clean/scoped-dependency-release-destroy-clean-h4096-r3/summary.json

These numbers measure one dedicated Graphile surface, not complete customers per GiB. They predate the final security-hardening pass, so they do not establish final-code throughput, p99, cache capacity, or production readiness. Security checks must not be removed to recover benchmark performance.

Confirmed isolation evidence and its limit

Before the final main refresh, a disposable PostgreSQL 18 A/B/C fixture used three distinct LOGIN NOINHERIT runtime roles, forced-RLS tenant tables, exact physical pools, scoped-required introspection, BM25/vector/PostGIS/ltree, and realtime-resident instances. It passed 56/56 hostile checks in 3.9 seconds with crossTenantTokens: 0.

The checks covered generated and plugin SQL, cross-schema identifiers, metadata, functions, sequences, search extensions, owner/BYPASS rejection, poisoned GUCs, rollback/savepoints, same-backend prepared-statement reuse, schema drift, cache invalidation, concurrent cold builds, realtime instances, and alternating connection reuse. Separate live PostgreSQL sanitizer and runtime-role suites passed 2/2 each.

The generated hostile-run artifact remains local/gitignored at research/graphile-density/complete-tenant-fixture/generated/hostile-validation.json. This evidence validates the fixture and exact physical data-plane mechanism, but it does not prove every deployed schema, policy, role grant, dependency object, provider integration, or control-plane transition. The hostile run must be repeated after the rebase and after the versioned security contract lands.

Why this is not production-ready

Production is currently a no-go. The unresolved work is part of the security contract, not cleanup.

1. Routing and security metadata do not share one atomic revision. Routing, RLS, auth, feature, CORS, public-key, and WebAuthn data are loaded independently. A domain handover or revocation can leave an in-flight HTTP request using a captured old authorization snapshot. Exact identities confine it to the captured physical tenant, but stale authorization is still incorrect.

2. Long-lived transports do not yet revalidate against authoritative current state. A WebSocket operation compares against the connection's captured route/generation. A handover or revocation can remain effective only after the socket closes or its generation retires. The new revision must drive admission, cache retirement, active subscription termination, and every operation check.

3. The production database policy manifest is unproven. The fixture proves the mechanism, while production still needs verification of every runtime/request role, exposed/dependency object, shared-row RLS policy, and intended FORCE ROW LEVEL SECURITY rule.

4. Authentication requirements are not yet authoritative per API. strictAuth=false deliberately supports public APIs and permits a missing RLS module. APIs that require authentication need a routed authRequired contract so missing configuration fails closed.

5. Scoped introspection depends on local source patches. It must remain off by default until Graphile maintainers review the isolated API and the patches are replaced with supported upstream behavior.

6. Operational trust boundaries still need deployment proof. The internal routing credential must be stripped at public ingress and accepted only over authenticated encrypted service hops. X-Meta-Schema is a disabled-by-default cross-tenant administrative capability and requires a separate private ingress if enabled. Runtime credential resolution and explicitly admitted plugins remain trusted code.

7. Release qualification is incomplete. Full parity with current main, external LLM/MinIO paths, multipart byte roundtrips, repeated complete-customer density runs, mixed workload latency/error gates, and the two-hour maximum-density soak remain open. The 6.55 MiB result is not yet a safe governor capacity setting.

The old PR stack is deprecated

This draft is the only continuation path. #1330–#1335 are retained as historical provenance and should not be merged independently.

Original PR	Disposition
#1330	Selectively reimplemented: valid quoting, plugin isolation, i18n, LLM/RAG, and search fixes were reapplied.
#1331	Hardened: cache governance remains, while runtime identity, credentials, GUCs, sanitation, and role safety are treated as an earlier trust boundary.
#1332	Replaced: fail-open SQL-text substitution was replaced by a parameterized, required-schema, fail-closed introspection candidate.
#1333 / #1334	Rejected as production designs: runtime SQL rewriting and shared tenant blueprints are absent from this candidate.
#1335	Refreshed: process isolation, canaries, workload scoring, and artifact concepts remain; pooling-specific claims are historical only.

The old approximately 14.7 MiB retained-heap, 417 ms cold-build, 87x reduction, and same-blueprint density claims came from the rejected rewrite/pooling system. They are historical context, not current evidence. The current dedicated-instance evidence is 6.55 MiB and 130.34 ms for one production-shaped surface; complete-customer density is still pending.

Validation status

Post-rebase focused checks completed for this draft:

• GraphQL server: 399 passed, 2 skipped, plus CJS/ESM build.
• Realtime plugin: 54/54 passed, plus build.
• LLM agent-discovery isolation: 8/8 passed, plus build.
• Request context: 65/65 passed, plus build.
• Presigned storage cache-isolation: 23/23 passed, plus package build.
• Lockfile regenerated from the rebased manifests; conflict markers and git diff --check are clean.

The full root build and broader package/security suites passed on the pre-rebase tip, but they are deliberately left unchecked above until rerun for current-main parity. The MinIO integration suite could not connect to a local MinIO service; live provider/storage qualification remains an explicit gate rather than a skipped success.

Scope

This PR changes only constructive. It contains no constructive-db changes, deployment, production enablement, SQL rewrite, or tenant blueprint pooling. Stock introspection remains the default, and no upstream maintainer has been contacted.

Suggested review order: security audit and routing/runtime boundary first, Graphile source patches second, plugin trust/raw SQL paths third, then cache governor and benchmark harness.

[yyyyaaa]

@pyramation @Zetazzz Conceptually, the algorithm changes are:

old PRs: reduce memory by making structurally similar tenants share one canonical Graphile instance:

normalize schema → fingerprint → shared blueprint
request → generate canonical SQL → rewrite schema names → execute

That scales with the number of unique blueprints, but isolation depends on every generated, plugin, raw-SQL, and prepared-statement path being rewritten correctly.

The new algorithm keeps an exact instance per tenant/API contract and instead makes each instance much smaller:

exact tenant contract → scoped catalog query → small dedicated Graphile build
request → already-physical SQL → sanitized pool + complete GUCs → execute

So the old approach reduced the number of instances; the new one reduces the memory cost of each instance.

[yyyyaaa]

This draft has been superseded by the smaller, bottom-up draft stack created for review:

1. Establish the tenant GraphQL runtime boundary #1654 — establish the tenant GraphQL runtime boundary
2. Isolate Graphile plugin and storage data paths #1655 — isolate Graphile plugin and storage data paths
3. Govern Graphile cache and realtime lifecycles #1656 — govern Graphile cache and realtime lifecycles
4. Build exact Graphile schemas from scoped introspection #1657 — build exact Graphile schemas from scoped introspection
5. Fail closed at the tenant request boundary #1658 — fail closed at the tenant request boundary
6. Add tenant-density validation and production evidence #1659 — add tenant-density validation and production evidence

The replacement stack preserves the current approach and its evidence while making each security/performance concern reviewable against its direct parent. All six replacement PRs are drafts and remain a production no-go until the blockers documented in their descriptions are closed.

The old #1330–#1335 SQL-rewrite/blueprint-pooling direction is deprecated. Valid plugin fixes, cache-governor ideas, and harness concepts were reimplemented or hardened, but runtime SQL rewriting and cross-tenant blueprint sharing are not part of the replacement. The dedicated-instance candidate gets its measured memory reduction from parameterized, dependency-scoped introspection and released build-only state while retaining exact physical tenant/API pool and build identities.

Closing #1652 as superseded; its branch is intentionally retained for provenance and recovery.