From e7539f3359a04ded3591e281c8d470e2367bbc84 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Thu, 30 Jul 2026 16:15:08 -0500 Subject: [PATCH 1/2] Fix DATA wildcard skipping historical full-install version scripts (#48) `base.mk`'s `DATA` wildcard (`sql/*--*--*.sql`) required two `--` separators, so it only matched upgrade scripts (`ext--a--b.sql`). Historical full-install scripts (`ext--a.sql`, one `--`) never matched, even though these are meant to be committed to git for update testing. `make install` silently never placed them in the extension directory, so `CREATE EXTENSION ext VERSION 'x.y.z'` failed for any version older than current, despite the file being present and tracked. Widened to `sql/*--*.sql`, which matches both forms. Wrapped in `$(sort ...)` to dedupe against `EXTENSION__CURRENT_VERSION__FILES` (also matched by the wider wildcard) -- without dedup, `install` is invoked with the current-version file listed twice and refuses to overwrite the copy it just created, failing `make install` outright. Related changes in pgxntool-test: - Add a `make print-DATA` regression test asserting the template's existing historical version file (`sql/pgxntool-test--0.1.0.sql`) is listed in `DATA` Co-Authored-By: Claude --- HISTORY.asc | 10 ++++++++++ base.mk | 9 ++++++++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/HISTORY.asc b/HISTORY.asc index 231809f..dc029aa 100644 --- a/HISTORY.asc +++ b/HISTORY.asc @@ -1,3 +1,13 @@ +STABLE +------ +== Fix `DATA` wildcard skipping historical full-install version scripts +`base.mk`'s `DATA` wildcard (`sql/*--*--*.sql`) only matched upgrade scripts +(two `--` separators), silently skipping historical full-install scripts +like `sql/ext--0.9.6.sql` (one `--`) even though these are meant to be +committed to git. `make install` never placed them in the extension +directory, so `CREATE EXTENSION ext VERSION 'x.y.z'` failed for any version +older than current. Widened to `sql/*--*.sql`, which matches both. + 2.2.0 ----- == Add `check-stale-expected` to catch orphaned test/expected files diff --git a/base.mk b/base.mk index e77cff2..711eb47 100644 --- a/base.mk +++ b/base.mk @@ -46,7 +46,14 @@ control.mk: $(PGXNTOOL_CONTROL_FILES) Makefile $(PGXNTOOL_DIR)/base.mk $(PGXNTOO -include control.mk -DATA = $(EXTENSION__CURRENT_VERSION__FILES) $(wildcard sql/*--*--*.sql) +# $(wildcard sql/*--*.sql) covers both upgrade scripts (ext--a--b.sql) and +# historical full-install scripts (ext--a.sql); EXTENSION__CURRENT_VERSION__FILES +# is also matched by it but is listed explicitly since it's a generated file +# that must exist as a prerequisite before the wildcard can see it. $(sort) +# dedupes the resulting overlap -- without it, `install` is invoked with the +# current-version file listed twice and refuses to overwrite the copy it +# just created, failing `make install` outright. +DATA = $(sort $(EXTENSION__CURRENT_VERSION__FILES) $(wildcard sql/*--*.sql)) DOC_DIRS += doc # NOTE: if this is empty it gets forcibly defined to NUL before including PGXS DOCS += $(foreach dir,$(DOC_DIRS),$(wildcard $(dir)/*)) From 96b8d20da67eb67b7f89a7fdc419ef84cbe9390a Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Fri, 31 Jul 2026 15:16:56 -0500 Subject: [PATCH 2/2] README: document generated-SQL wildcard-timing gotcha (issue #44) While confirming this branch's #48 fix doesn't drop upgrade scripts, a related but distinct gap came up: DATA's $(wildcard sql/*--*.sql) is a correct pattern (#48 was about the pattern, now fixed), but for extensions that generate their versioned SQL from another source (e.g. .sql.in) and gitignore the output, a clean `make install`/`make test` can still miss files generated earlier in the same invocation -- GNU Make caches a directory's wildcard results for the life of one invocation and doesn't invalidate that cache just because a recipe created new files. Tracked separately as issue #44; this documents the gotcha and workarounds (separate make invocations, or listing generated files explicitly via stable .sql.in-derived names) without attempting the larger design change (a generated-output directory convention) #44 floats. Co-Authored-By: Claude --- README.asc | 15 +++++++++++++++ README.html | 31 ++++++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/README.asc b/README.asc index a799c7c..ea6127a 100644 --- a/README.asc +++ b/README.asc @@ -457,6 +457,21 @@ If you need to support multiple versions of your extension: The version file for the current version (specified in the `.control` file's `default_version`) will be automatically regenerated when you run `make`, but other version files you create manually will be preserved. +=== Gotchas + +==== Generated SQL Files Can Be Missed on a Clean Build + +`DATA` (the PGXS variable controlling what `make install` actually installs) is seeded by pgxntool with `$(wildcard sql/*--*.sql)`, so PGXS installs whatever version-specific and upgrade files exist under `sql/` when `DATA` is evaluated. + +If your extension *generates* its versioned SQL from another source (e.g. a `.sql.in` template expanded by a custom rule) and gitignores the generated `.sql`, a clean `make install` (or `make test`, `make all install` in one invocation) can silently omit files that were only just generated during that same run. This isn't a pattern-matching problem — the glob is correct — it's a GNU Make quirk: `wildcard` caches a directory's contents internally for the life of one `make` invocation, and that cache isn't invalidated just because a recipe created new files earlier in the same run. So even though `DATA` is a recursively-expanded variable (re-evaluated each time it's referenced), re-referencing it later in the same invocation can still see the stale, pre-generation listing. + +Workarounds: + +- Run generation and install as separate `make` invocations (e.g. `make && make install`) so the second invocation's parse-time wildcard scan sees files the first invocation created. +- List the generated files in `DATA` explicitly, keyed off stable names derived from your `.sql.in` sources (known at parse time) rather than relying on `wildcard` to discover the generated `.sql` output. + +Tracked as https://github.com/Postgres-Extensions/pgxntool/issues/44[issue #44] — pgxntool doesn't yet have first-class support for generated versioned SQL; the ideas under discussion there (e.g. a dedicated generated-output directory pgxntool builds before installing) are a bigger design change than a one-line fix. + == Document Handling PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior. diff --git a/README.html b/README.html index 36d3cf5..0842db2 100644 --- a/README.html +++ b/README.html @@ -475,6 +475,7 @@

PGXNtool

  • 5.5. Alternative: Ignoring All Version Files
  • 5.6. Distribution Inclusion
  • 5.7. Multiple Versions
    • +
  • 5.8. Gotchas
  • 6. Document Handling @@ -1481,6 +1482,34 @@

    The version file for the current version (specified in the .control file’s default_version) will be automatically regenerated when you run make, but other version files you create manually will be preserved.

    +
    +

    5.8. Gotchas

    +
    +

    5.8.1. Generated SQL Files Can Be Missed on a Clean Build

    +
    +

    DATA (the PGXS variable controlling what make install actually installs) is seeded by pgxntool with $(wildcard sql/--.sql), so PGXS installs whatever version-specific and upgrade files exist under sql/ when DATA is evaluated.

    +
    +
    +

    If your extension generates its versioned SQL from another source (e.g. a .sql.in template expanded by a custom rule) and gitignores the generated .sql, a clean make install (or make test, make all install in one invocation) can silently omit files that were only just generated during that same run. This isn’t a pattern-matching problem — the glob is correct — it’s a GNU Make quirk: wildcard caches a directory’s contents internally for the life of one make invocation, and that cache isn’t invalidated just because a recipe created new files earlier in the same run. So even though DATA is a recursively-expanded variable (re-evaluated each time it’s referenced), re-referencing it later in the same invocation can still see the stale, pre-generation listing.

    +
    +
    +

    Workarounds:

    +
    +
    +
      +
    • +

      Run generation and install as separate make invocations (e.g. make && make install) so the second invocation’s parse-time wildcard scan sees files the first invocation created.

      +
      • +
    • +

      List the generated files in DATA explicitly, keyed off stable names derived from your .sql.in sources (known at parse time) rather than relying on wildcard to discover the generated .sql output.

      +
      • +
    +
    +
    +

    Tracked as issue #44 — pgxntool doesn’t yet have first-class support for generated versioned SQL; the ideas under discussion there (e.g. a dedicated generated-output directory pgxntool builds before installing) are a bigger design change than a one-line fix.

    +
    +
    +
    @@ -1954,7 +1983,7 @@