Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .changeset/cli-near-miss-statement-trim.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
'stash': patch
---

Trim the leading comment block from near-miss statements reported by the Drizzle
migration rewriter (`stash eql migration --drizzle`, `stash eql install`).

The broad near-miss scan is anchored on the previous `;`, so a
`SET DATA TYPE … USING …` it could not safely repair was quoted back to the user
with every preceding comment and blank line glued to its front — in a file
opening with a comment block, that meant the whole header. The reported
statement is now the offending statement alone. Detection is unchanged; only the
text shown to the user is affected.

Keeps this rewriter in sync with its sibling in `@cipherstash/wizard`.
36 changes: 36 additions & 0 deletions .changeset/wizard-eql-v3-migration-rewrite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
'@cipherstash/wizard': minor
---

Teach the wizard's post-agent Drizzle step to repair EQL **v3** migrations, not
just legacy EQL v2.

The wizard now scaffolds EQL v3 columns, so `drizzle-kit generate` emits
`ALTER TABLE … ALTER COLUMN … SET DATA TYPE eql_v3_<name>` — which Postgres
rejects (there is no cast from `text`/`numeric` to an EQL domain). The migration
rewriter previously matched only the single `eql_v2_encrypted` type, so those v3
statements slipped through unrepaired and failed at migrate time.

The rewriter is ported to match the whole EQL v3 concrete-domain family
(`eql_v3_text_search`, `eql_v3_integer_ord`, …) alongside legacy
`eql_v2_encrypted`, across every mangled form drizzle-kit emits (including the
`"undefined".` prefix from 0.31.0+ and schema-qualified `pgSchema()` tables). It
now also flags near-miss `SET DATA TYPE … USING …` statements it cannot safely
repair instead of leaving broken SQL, and each rewritten file carries a clearer
warning that the ADD+DROP+RENAME is data-destroying and safe only on an empty
table — a populated table must use the staged `stash encrypt` flow. This
re-converges the rewriter with the sibling copy in the `stash` CLI.

The post-agent step now sweeps **every** candidate migration directory
(`drizzle/`, `migrations/`, `src/db/migrations/`) rather than stopping at the
first one that exists. Previously an empty or already-rewritten `drizzle/`
sitting next to a project's real `migrations/` caused those migrations to be
skipped entirely, so they still failed at migrate time. A directory that can't
be read is reported and the remaining candidates are still swept. Reported
near-miss statements are also trimmed of any preceding comment block, so the
statement quoted back to the user is the offending statement alone.

Database introspection also recognises v3 encrypted columns: `isEqlEncrypted`
now reports both `eql_v2_encrypted` and the `eql_v3_*` family as already
encrypted, so the agent won't scaffold over existing encrypted data of either
generation.
6 changes: 6 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,12 @@ jobs:
- name: Typecheck (bench — guards the stack/stack-drizzle importers)
run: pnpm exec turbo run build --filter @cipherstash/bench

# The wizard is built by tsup, which transpiles without typechecking, so
# nothing here caught the three `auth.AutoStrategy` resolution errors that
# sat in `main` until #771. Gate it so they cannot come back silently.
- name: Typecheck (wizard)
run: pnpm --filter @cipherstash/wizard run typecheck

- name: Lint — no hardcoded package-manager runners
run: pnpm run lint:runners

Expand Down
60 changes: 60 additions & 0 deletions packages/cli/src/__tests__/rewrite-migrations.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,66 @@ describe('rewriteEncryptedAlterColumns', () => {
expect(fs.readFileSync(filePath, 'utf-8')).toBe(original)
})

// A near-miss is quoted back to the user verbatim, so it must read as the
// offending statement alone. NEAR_MISS_RE opens with a lazy `[^;]*?`, which
// can only be bounded by the previous `;` — so without an explicit trim the
// reported "statement" drags in every comment and blank line since then.
it('reports a near-miss without the file-leading comment block', async () => {
const statement =
'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;'
const filePath = path.join(tmpDir, '0022_preamble.sql')
fs.writeFileSync(
filePath,
[
'-- Custom SQL migration file, put your code below! --',
'-- Hand-converts the email column in place.',
'',
statement,
'',
].join('\n'),
)

const { skipped } = await rewriteEncryptedAlterColumns(tmpDir)

expect(skipped).toHaveLength(1)
expect(skipped[0].statement).toBe(statement)
})

it('reports a near-miss without a preceding statement-breakpoint marker', async () => {
const statement =
'ALTER TABLE "users" ALTER COLUMN "meta" SET DATA TYPE eql_v3_json USING (meta)::jsonb;'
const filePath = path.join(tmpDir, '0023_breakpoint-preamble.sql')
fs.writeFileSync(
filePath,
[
'CREATE TABLE "users" ("id" integer PRIMARY KEY);',
'--> statement-breakpoint',
statement,
'',
].join('\n'),
)

const { skipped } = await rewriteEncryptedAlterColumns(tmpDir)

expect(skipped).toHaveLength(1)
expect(skipped[0].statement).toBe(statement)
})

it('keeps a multi-line near-miss statement intact after the preamble trim', async () => {
const statement = [
'ALTER TABLE "users"',
' ALTER COLUMN "email"',
' SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;',
].join('\n')
const filePath = path.join(tmpDir, '0024_multiline.sql')
fs.writeFileSync(filePath, `-- leading note\n\n${statement}\n`)

const { skipped } = await rewriteEncryptedAlterColumns(tmpDir)

expect(skipped).toHaveLength(1)
expect(skipped[0].statement).toBe(statement)
})

it('reports no skipped statements for a clean file', async () => {
const filePath = path.join(tmpDir, '0020_clean.sql')
fs.writeFileSync(filePath, 'CREATE TABLE "t" ("id" integer);\n')
Expand Down
26 changes: 25 additions & 1 deletion packages/cli/src/commands/db/rewrite-migrations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,27 @@ const ALTER_COLUMN_TO_ENCRYPTED_RE = new RegExp(
const NEAR_MISS_RE =
/[^;]*?\bSET\s+DATA\s+TYPE\b[^;]*?\beql_v[23][a-z0-9_]*[^;]*?;/gi

/**
* Blank lines and `--` line comments (including drizzle-kit's
* `--> statement-breakpoint`) at the head of a {@link NEAR_MISS_RE} match.
*
* That regex opens with a lazy `[^;]*?`, whose only left boundary is the
* previous `;` — or the start of the file when there is no preceding statement.
* So the raw match drags in every comment and blank line since then, and a
* near-miss in a file that opens with a comment block gets reported to the user
* with that whole block glued to its front. Strip the preamble so the statement
* we quote back reads as the offending statement alone.
*
* Only line comments are handled — `/* … *\/` block comments are not something
* drizzle-kit emits, and a stray one costs cosmetics, not correctness.
*/
const STATEMENT_PREAMBLE_RE = /^(?:[^\S\n]*(?:--[^\n]*)?\n)+/

/** Drop the leading blank/comment lines a `[^;]*?`-anchored match dragged in. */
function trimStatementPreamble(statement: string): string {
return statement.replace(STATEMENT_PREAMBLE_RE, '').trim()
}

/** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */
export interface SkippedAlter {
/** Absolute path of the migration file the statement lives in. */
Expand Down Expand Up @@ -189,7 +210,10 @@ export async function rewriteEncryptedAlterColumns(
// matcher. Flag it — non-fatally — rather than leave the user shipping SQL
// that fails at migrate time.
for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) {
skipped.push({ file: filePath, statement: nearMiss[0].trim() })
skipped.push({
file: filePath,
statement: trimStatementPreamble(nearMiss[0]),
})
}
}

Expand Down
1 change: 1 addition & 0 deletions packages/wizard/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"postbuild": "chmod +x ./dist/bin/wizard.js",
"dev": "tsup --watch",
"test": "vitest run",
"typecheck": "tsc --project tsconfig.json --noEmit",
"lint": "biome check ."
},
"dependencies": {
Expand Down
Loading