Skip to content

perf(transform): reuse compiled schema regexes and skip schemas absent from the content #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pyramation merged 1 commit into from
Aug 5, 2026
Merged

perf(transform): reuse compiled schema regexes and skip schemas absent from the content #341

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 54 additions & 10 deletions packages/transform/src/transform.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,8 @@ export function transformSchemaRefsInString(
// References embedded in opaque strings carry no object identity, so only
// whole-schema (schema-level default) routes can be applied here.
for (const [oldSchema, newSchema] of schemaLevelMap(schemaMapping)) {
const pattern = new RegExp(`(?<![\\w-])("?)${escapeRegexp(oldSchema)}\\1(?=\\.)`, 'g');
if (!out.includes(oldSchema)) continue;
const pattern = cachedRegExp(`(?<![\\w-])("?)${cachedEscapeRegexp(oldSchema)}\\1(?=\\.)`, 'g');
const before = out;
out = out.replace(pattern, `$1${newSchema}$1`);
if (out !== before) {
Expand Down Expand Up @@ -745,16 +746,22 @@ export function validateNoUntransformedSchemas(
return;
}

// Every pattern below requires a literal occurrence of the schema name, so a
// single case-insensitive substring scan rules out most schemas up front.
const lowerContent = content.toLowerCase();

for (const [oldSchema, newSchema] of moved) {
const escapedSchema = escapeRegexp(oldSchema);
if (!lowerContent.includes(oldSchema.toLowerCase())) continue;

const escapedSchema = cachedEscapeRegexp(oldSchema);

// Pattern 1: quoted or unquoted schema name followed by dot (schema-qualified)
const dotPattern = new RegExp(`(?:"${escapedSchema}"|\\b${escapedSchema})(?=\\.)`, 'g');
const dotPattern = cachedRegExp(`(?:"${escapedSchema}"|\\b${escapedSchema})(?=\\.)`, 'g');

// Pattern 2: standalone schema name in known SQL contexts
// Note: We don't use trailing \b because it fails after closing quotes
// (both '"' and whitespace are non-word characters, so no boundary exists).
const standalonePattern = new RegExp(
const standalonePattern = cachedRegExp(
`(?:ON\\s+SCHEMA\\s+|IN\\s+SCHEMA\\s+|CREATE\\s+SCHEMA\\s+|DROP\\s+SCHEMA\\s+(?:IF\\s+EXISTS\\s+)?|SET\\s+SCHEMA\\s+)` +
`(?:"${escapedSchema}"|\\b${escapedSchema}\\b)`,
'gi'
Expand All @@ -768,7 +775,7 @@ export function validateNoUntransformedSchemas(
const lines = content.split('\n');
const locations: string[] = [];

const combinedPattern = new RegExp(
const combinedPattern = cachedRegExp(
`(?:"${escapedSchema}"|\\b${escapedSchema})(?=\\.)|` +
`(?:ON\\s+SCHEMA\\s+|IN\\s+SCHEMA\\s+|CREATE\\s+SCHEMA\\s+|DROP\\s+SCHEMA\\s+(?:IF\\s+EXISTS\\s+)?|SET\\s+SCHEMA\\s+)` +

@pyramation pyramation Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we not using ASTs here, and why are we using regular expressions?

@devin-ai-integration devin-ai-integration Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair question — this PR doesn't introduce the regex approach, it only stops recompiling the same patterns, but the "why" is worth pinning down per call site:

validateNoUntransformedSchemas specifically cannot use the AST, because it's the safety net for the AST pass having missed a node. It runs on the deparsed output and asks "did any old schema name survive?" — checking the AST would only re-ask the question the AST pass already answered wrong, so it would pass exactly when it most needs to fail. Its own error message says as much ("indicates a missing visitor handler in create_sql_visitor").

The other three are text the SQL AST genuinely doesn't reach:

  • transformComments — pgpm -- Deploy: / -- requires: header paths; not SQL at all.
  • transformVerifyCalls — verify_function('schema.fn'); the schema name lives inside a string literal, so to the parser it's opaque text, not a qualified name.
  • transformJsonStringValues — same, schema names inside JSON string values.

So the regex isn't standing in for AST work here; it covers the non-AST residue plus a post-hoc assertion. Where it would be wrong is regex-rewriting actual SQL identifiers — that stays createSqlVisitor's job.

Happy to go further separately if you want: transformVerifyCalls could parse the literal's contents (known-shape qualified name) instead of pattern-matching, and the leftover check could be narrowed to string/comment tokens via the lexer rather than scanning the whole file.

`(?:"${escapedSchema}"|\\b${escapedSchema}\\b)`,
Expand Down Expand Up @@ -1072,6 +1079,38 @@ export function escapeRegexp(str: string): string {
return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}

/**
* Compiling a pattern costs far more than running it, and the string-level
* passes below rebuild the same handful of per-schema patterns for every file
* they touch — on a large corpus that is hundreds of thousands of identical
* compiles. Patterns are keyed by source + flags and reused; each returned
* RegExp is stateful (`g`/`y` carry `lastIndex`), so callers must only use it
* with `String#replace`/`String#match`, which reset it, or reset it
* themselves, exactly as the throwaway instances required.
*/
const regExpCache = new Map<string, RegExp>();

function cachedRegExp(source: string, flags: string): RegExp {
const key = `${flags}\u0000${source}`;
let re = regExpCache.get(key);
if (!re) {
re = new RegExp(source, flags);
regExpCache.set(key, re);
}
return re;
}

const escapeRegexpCache = new Map<string, string>();

function cachedEscapeRegexp(str: string): string {
let escaped = escapeRegexpCache.get(str);
if (escaped === undefined) {
escaped = escapeRegexp(str);
escapeRegexpCache.set(str, escaped);
}
return escaped;
}

/**
* Extract pgpm header comments from the beginning of SQL content.
*/
Expand Down Expand Up @@ -1148,7 +1187,7 @@ export function transformComments(

result.schemasFound.add(schema);

const pathPattern = new RegExp(`(schemas/)${escapeRegexp(schema)}(/|$)`, 'g');
const pathPattern = cachedRegExp(`(schemas/)${cachedEscapeRegexp(schema)}(/|$)`, 'g');
const before = newPath;
newPath = newPath.replace(pathPattern, `$1${newName}$2`);

Expand All @@ -1173,14 +1212,17 @@ export function transformVerifyCalls(
const schemas = Array.from(schemaMapping.keys()).sort((a, b) => b.length - a.length);

let newContent = content;
// The pattern needs the schema name spelled out, case-insensitively.
let lowerContent = content.toLowerCase();

for (const schema of schemas) {
const newName = schemaMapping.get(schema);
if (!newName) continue;
if (!lowerContent.includes(schema.toLowerCase())) continue;

const escapedSchema = escapeRegexp(schema);
const escapedSchema = cachedEscapeRegexp(schema);

const verifyPattern = new RegExp(
const verifyPattern = cachedRegExp(
`(verify_(?:function|table|trigger|type|domain|view|index|constraint|schema|policy|table_grant|function_grant|sequence_grant|type_grant)\\s*\\(\\s*')${escapedSchema}(\\.|'\\s*\\))`,
'gi'
);
Expand All @@ -1189,6 +1231,7 @@ export function transformVerifyCalls(
newContent = newContent.replace(verifyPattern, `$1${newName}$2`);

if (newContent !== before) {
lowerContent = newContent.toLowerCase();
result.schemasFound.add(schema);
result.schemasTransformed.set(schema, newName);
}
Expand All @@ -1212,10 +1255,11 @@ export function transformJsonStringValues(
for (const schema of schemas) {
const newName = schemaMapping.get(schema);
if (!newName) continue;
if (!newContent.includes(schema)) continue;

const escapedSchema = escapeRegexp(schema);
const escapedSchema = cachedEscapeRegexp(schema);

const jsonValuePattern = new RegExp(
const jsonValuePattern = cachedRegExp(
`(:")${escapedSchema}(")`,
'g'
);
Expand Down
Loading