make pgtle PGTLE_VERSION=1.5.0+
make pgtle PGXNTOOL_PGTLE_VERSION=1.5.0
diff --git a/pgxntool/CLAUDE.md b/pgxntool/CLAUDE.md index 5db8c95..65e3bf7 100644 --- a/pgxntool/CLAUDE.md +++ b/pgxntool/CLAUDE.md @@ -91,18 +91,24 @@ include pgxntool/base.mk ### Phase 1: Meta Generation (`build_meta.sh`) - Processes `META.in.json` (template with placeholders/empty values) -- Strips out X_comment fields and empty values +- Strips lines with empty string values (this incidentally removes some, but not all, `X_comment` placeholder lines) - Produces clean `META.json` -### Phase 2: Variable Extraction (`meta.mk.sh`) -- Parses `META.json` using `JSON.sh` (a bash-based JSON parser) +### Phase 2: Distribution Variable Extraction (`meta.mk.sh`) +- Parses `META.json` (PGXN distribution metadata) - Generates `meta.mk` with Make variables: - `PGXN` - distribution name - `PGXNVERSION` - version number +- `base.mk` includes `meta.mk` via `-include` + +### Phase 3: Extension Variable Extraction (`control.mk.sh`) +- Parses each `.control` file (what PostgreSQL actually uses), not `META.json` +- Generates `control.mk` with Make variables and rules: - `EXTENSIONS` - list of extensions provided - - `EXTENSION_*_VERSION` - per-extension versions + - `EXTENSION_*_VERSION` - per-extension versions, from each `.control` file's `default_version` - `EXTENSION__CURRENT_VERSION__FILES` - the current/most-recent auto-generated versioned SQL file for each extension (not all version files) -- `base.mk` includes `meta.mk` via `-include` + - Rules for generating versioned SQL files +- `base.mk` includes `control.mk` via `-include` ### The Magic of base.mk @@ -157,10 +163,11 @@ make pgxntool-sync # Update to latest pgxntool via git subtree pull ### Critical Testing Rules **NEVER use `make installcheck` directly**. Always use `make test` instead. The `make test` target ensures: -- Clean builds before testing -- Proper test isolation -- Correct test dependency installation -- Proper cleanup and result comparison +- Correct test dependency installation (`testdeps`, and `test-build` when enabled) +- Extension is installed before tests run (`install`) +- Test comparison via `installcheck`, with diffs shown on failure + +Note: `make test` intentionally does *not* depend on `clean` — depending on `clean` caused problems with incremental/watch-based builds (see `base.mk`). If your tests need a clean build to pass, that's a sign of a missing dependency elsewhere, not something to fix by adding `clean` back. **Database Connection Requirement**: PostgreSQL must be running before executing `make test`. If you get connection errors (e.g., "could not connect to server"), stop and ask the user to start PostgreSQL. @@ -204,7 +211,7 @@ When tests fail, examine the diff output carefully. The actual test output in `t ### Distribution Packaging - `make dist` creates `../PGXN-VERSION.zip` -- Always creates git tag matching version +- Creates a git tag matching version, unless that tag already exists and points at HEAD (in which case tagging is skipped) - Uses `git archive` to package - Validates repo is clean before tagging @@ -219,7 +226,7 @@ When tests fail, examine the diff output carefully. The actual test output in `t pgxntool can generate pg_tle (Trusted Language Extensions) registration SQL for deploying extensions in AWS RDS/Aurora without filesystem access. -**Usage:** `make pgtle` or `make pgtle PGTLE_VERSION=1.5.0+` +**Usage:** `make pgtle` or `make pgtle PGXNTOOL_PGTLE_VERSION=1.5.0+` **Output:** `pg_tle/{version_range}/{extension}.sql` @@ -273,11 +280,11 @@ Generated files depend on: ## Scripts -- **setup.sh** - Initializes pgxntool in a new extension project (copies templates, creates directories) -- **build_meta.sh** - Strips empty fields from META.in.json to create META.json -- **meta.mk.sh** - Parses META.json via JSON.sh and generates meta.mk with Make variables -- **JSON.sh** - Third-party bash JSON parser (MIT licensed) -- **safesed** - Utility for safe sed operations +Each script at the top level of this repository (pgxntool/) begins with a shebang followed by a short +header comment describing its purpose within the first ~10 lines — read that header +rather than relying on a hand-maintained list here, which would just go stale the same +way this section itself once did. Exception: trivial third-party or one-line utilities +may lack such a header. ## Related Repositories diff --git a/pgxntool/HISTORY.asc b/pgxntool/HISTORY.asc index 231809f..bdd5121 100644 --- a/pgxntool/HISTORY.asc +++ b/pgxntool/HISTORY.asc @@ -1,3 +1,30 @@ +2.3.0 +----- +== Rename `PGTLE_VERSION` to `PGXNTOOL_PGTLE_VERSION` +`make pgtle`'s version-limiting variable was named `PGTLE_VERSION`, which +make auto-imports from an identically-named environment variable. That name +is also a natural choice for a CI job's "which pg_tle to test against" env +var -- when one was set, `make pgtle`/`make run-pgtle` silently misbehaved +instead of erroring. Renamed to `PGXNTOOL_PGTLE_VERSION` to avoid the +collision. If you invoke `make pgtle PGTLE_VERSION=...` directly, update it +to `PGXNTOOL_PGTLE_VERSION`. + +== Fix `DATA` wildcard skipping historical full-install version scripts +`base.mk`'s `DATA` wildcard (`sql/*--*--*.sql`) only matched upgrade scripts +(two `--` separators), silently skipping historical full-install scripts +like `sql/ext--0.9.6.sql` (one `--`) even though these are meant to be +committed to git. `make install` never placed them in the extension +directory, so `CREATE EXTENSION ext VERSION 'x.y.z'` failed for any version +older than current. Widened to `sql/*--*.sql`, which matches both. + +== Fix `make test` always exiting 0, even when every test fails +`.IGNORE: installcheck` made `make` treat `installcheck` as always successful +regardless of `pg_regress`'s exit status, and `test`'s recipe only `cat`ed +`regression.diffs` for visibility without checking it. `make test` now exits +non-zero when `regression.diffs` is non-empty, after printing it. + +Issues fixed in this release: #35, #49, #78, #79, #48 + 2.2.0 ----- == Add `check-stale-expected` to catch orphaned test/expected files diff --git a/pgxntool/README.asc b/pgxntool/README.asc index a799c7c..45c4b59 100644 --- a/pgxntool/README.asc +++ b/pgxntool/README.asc @@ -51,6 +51,8 @@ Whether `test-build` runs is controlled by the `PGXNTOOL_ENABLE_TEST_BUILD` vari NOTE: `test` intentionally does *not* depend on `clean` — that caused problems with incremental/watch-based builds. If your tests need a clean build to pass, that's a sign of a missing dependency elsewhere rather than something to fix by adding `clean` back. +NOTE: `test` exits non-zero (after printing `regression.diffs`) if any test fails. Previously it always exited 0 regardless of test results, silently masking failures from CI and other automation that relies on the exit code. + NOTE: While you can still run `make installcheck` or any other valid PGXS make target directly, it's recommended to use `make test` when using pgxntool. The `test` target ensures proper test isolation and correct dependency installation. === test-build @@ -270,7 +272,7 @@ WARNING: If your project has a `.gitattributes` file, `make dist`/`make dist-onl `make forcedist` is a shortcut for `forcetag dist`: it force-recreates the tag (see `forcetag` above) before rebuilding the distribution .zip. === pgxntool-sync -This rule will pull down the latest released version of PGXNtool via `git subtree pull` and then reconcile the files `setup.sh` copied into your project (`.gitignore`, `test/deps.sql`) with a 3-way merge. +This rule will pull down the latest released version of PGXNtool via `git subtree pull` and then reconcile the files `setup.sh` copied into your project (`.gitignore`, `test/deps.sql`) with a 3-way merge (it also verifies the `test/pgxntool` symlink, recreating it if missing). NOTE: Your repository must be clean (no modified files) in order to run this. Running this command will produce a git commit of the merge. @@ -308,10 +310,10 @@ Generates pg_tle (Trusted Language Extensions) registration SQL files for deploy `make pgtle` generates SQL files in `pg_tle/` subdirectories organized by pg_tle version ranges. For version range details, see `pgtle_versions.md`. -Set `PGTLE_VERSION` on the command line to limit generation to the single version range that value falls into, instead of every known range: +Set `PGXNTOOL_PGTLE_VERSION` on the command line to limit generation to the single version range that value falls into, instead of every known range: ---- -make pgtle PGTLE_VERSION=1.5.0 +make pgtle PGXNTOOL_PGTLE_VERSION=1.5.0 ---- === check-pgtle @@ -457,6 +459,21 @@ If you need to support multiple versions of your extension: The version file for the current version (specified in the `.control` file's `default_version`) will be automatically regenerated when you run `make`, but other version files you create manually will be preserved. +=== Gotchas + +==== Generated SQL Files Can Be Missed on a Clean Build + +`DATA` (the PGXS variable controlling what `make install` actually installs) is seeded by pgxntool with `$(wildcard sql/*--*.sql)`, so PGXS installs whatever version-specific and upgrade files exist under `sql/` when `DATA` is evaluated. + +If your extension *generates* its versioned SQL from another source (e.g. a `.sql.in` template expanded by a custom rule) and gitignores the generated `.sql`, a clean `make install` (or `make test`, `make all install` in one invocation) can silently omit files that were only just generated during that same run. This isn't a pattern-matching problem — the glob is correct — it's a GNU Make quirk: `wildcard` caches a directory's contents internally for the life of one `make` invocation, and that cache isn't invalidated just because a recipe created new files earlier in the same run. So even though `DATA` is a recursively-expanded variable (re-evaluated each time it's referenced), re-referencing it later in the same invocation can still see the stale, pre-generation listing. + +Workarounds: + +- Run generation and install as separate `make` invocations (e.g. `make && make install`) so the second invocation's parse-time wildcard scan sees files the first invocation created. +- List the generated files in `DATA` explicitly, keyed off stable names derived from your `.sql.in` sources (known at parse time) rather than relying on `wildcard` to discover the generated `.sql` output. + +Tracked as https://github.com/Postgres-Extensions/pgxntool/issues/44[issue #44] — pgxntool doesn't yet have first-class support for generated versioned SQL; the ideas under discussion there (e.g. a dedicated generated-output directory pgxntool builds before installing) are a bigger design change than a one-line fix. + == Document Handling PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior. @@ -665,9 +682,9 @@ make dist PGXN_REMOTE=upstream Default: auto-detected, the first of `asciidoctor` or `asciidoc` found on `PATH`. Path to the Asciidoc processor used to build `.html` files from `$(ASCIIDOC_EXTS)` source files. Override if the processor you want isn't first on `PATH`, or isn't on `PATH` at all. See <<_document_handling>>. -=== PGTLE_VERSION +=== PGXNTOOL_PGTLE_VERSION -Default: unset (generates every known pg_tle version range). Set on the command line to limit `make pgtle` to the single version range this value falls into. See <<_pgtle>>. +Default: unset (generates every known pg_tle version range). Set on the command line to limit `make pgtle` to the single version range this value falls into. Not named `PGTLE_VERSION`: make auto-imports same-named environment variables, and that name collided silently with CI jobs that set a `PGTLE_VERSION` env var for an unrelated purpose (which pg_tle to test against). See <<_pgtle>>. === PG_CONFIG diff --git a/pgxntool/README.html b/pgxntool/README.html index 36d3cf5..ba71c2a 100644 --- a/pgxntool/README.html +++ b/pgxntool/README.html @@ -475,6 +475,7 @@
Set PGTLE_VERSION on the command line to limit generation to the single version range that value falls into, instead of every known range:
Set PGXNTOOL_PGTLE_VERSION on the command line to limit generation to the single version range that value falls into, instead of every known range:
make pgtle PGTLE_VERSION=1.5.0+
make pgtle PGXNTOOL_PGTLE_VERSION=1.5.0
.control file’s default_version) will be automatically regenerated when you run make, but other version files you create manually will be preserved.
+DATA (the PGXS variable controlling what make install actually installs) is seeded by pgxntool with $(wildcard sql/--.sql), so PGXS installs whatever version-specific and upgrade files exist under sql/ when DATA is evaluated.
If your extension generates its versioned SQL from another source (e.g. a .sql.in template expanded by a custom rule) and gitignores the generated .sql, a clean make install (or make test, make all install in one invocation) can silently omit files that were only just generated during that same run. This isn’t a pattern-matching problem — the glob is correct — it’s a GNU Make quirk: wildcard caches a directory’s contents internally for the life of one make invocation, and that cache isn’t invalidated just because a recipe created new files earlier in the same run. So even though DATA is a recursively-expanded variable (re-evaluated each time it’s referenced), re-referencing it later in the same invocation can still see the stale, pre-generation listing.
Workarounds:
+Run generation and install as separate make invocations (e.g. make && make install) so the second invocation’s parse-time wildcard scan sees files the first invocation created.
List the generated files in DATA explicitly, keyed off stable names derived from your .sql.in sources (known at parse time) rather than relying on wildcard to discover the generated .sql output.
Tracked as issue #44 — pgxntool doesn’t yet have first-class support for generated versioned SQL; the ideas under discussion there (e.g. a dedicated generated-output directory pgxntool builds before installing) are a bigger design change than a one-line fix.
+Default: unset (generates every known pg_tle version range). Set on the command line to limit make pgtle to the single version range this value falls into. See pgtle.
Default: unset (generates every known pg_tle version range). Set on the command line to limit make pgtle to the single version range this value falls into. Not named PGTLE_VERSION: make auto-imports same-named environment variables, and that name collided silently with CI jobs that set a PGTLE_VERSION env var for an unrelated purpose (which pg_tle to test against). See pgtle.