diff --git a/pgxntool/CLAUDE.md b/pgxntool/CLAUDE.md index 5db8c95..65e3bf7 100644 --- a/pgxntool/CLAUDE.md +++ b/pgxntool/CLAUDE.md @@ -91,18 +91,24 @@ include pgxntool/base.mk ### Phase 1: Meta Generation (`build_meta.sh`) - Processes `META.in.json` (template with placeholders/empty values) -- Strips out X_comment fields and empty values +- Strips lines with empty string values (this incidentally removes some, but not all, `X_comment` placeholder lines) - Produces clean `META.json` -### Phase 2: Variable Extraction (`meta.mk.sh`) -- Parses `META.json` using `JSON.sh` (a bash-based JSON parser) +### Phase 2: Distribution Variable Extraction (`meta.mk.sh`) +- Parses `META.json` (PGXN distribution metadata) - Generates `meta.mk` with Make variables: - `PGXN` - distribution name - `PGXNVERSION` - version number +- `base.mk` includes `meta.mk` via `-include` + +### Phase 3: Extension Variable Extraction (`control.mk.sh`) +- Parses each `.control` file (what PostgreSQL actually uses), not `META.json` +- Generates `control.mk` with Make variables and rules: - `EXTENSIONS` - list of extensions provided - - `EXTENSION_*_VERSION` - per-extension versions + - `EXTENSION_*_VERSION` - per-extension versions, from each `.control` file's `default_version` - `EXTENSION__CURRENT_VERSION__FILES` - the current/most-recent auto-generated versioned SQL file for each extension (not all version files) -- `base.mk` includes `meta.mk` via `-include` + - Rules for generating versioned SQL files +- `base.mk` includes `control.mk` via `-include` ### The Magic of base.mk @@ -157,10 +163,11 @@ make pgxntool-sync # Update to latest pgxntool via git subtree pull ### Critical Testing Rules **NEVER use `make installcheck` directly**. Always use `make test` instead. The `make test` target ensures: -- Clean builds before testing -- Proper test isolation -- Correct test dependency installation -- Proper cleanup and result comparison +- Correct test dependency installation (`testdeps`, and `test-build` when enabled) +- Extension is installed before tests run (`install`) +- Test comparison via `installcheck`, with diffs shown on failure + +Note: `make test` intentionally does *not* depend on `clean` — depending on `clean` caused problems with incremental/watch-based builds (see `base.mk`). If your tests need a clean build to pass, that's a sign of a missing dependency elsewhere, not something to fix by adding `clean` back. **Database Connection Requirement**: PostgreSQL must be running before executing `make test`. If you get connection errors (e.g., "could not connect to server"), stop and ask the user to start PostgreSQL. @@ -204,7 +211,7 @@ When tests fail, examine the diff output carefully. The actual test output in `t ### Distribution Packaging - `make dist` creates `../PGXN-VERSION.zip` -- Always creates git tag matching version +- Creates a git tag matching version, unless that tag already exists and points at HEAD (in which case tagging is skipped) - Uses `git archive` to package - Validates repo is clean before tagging @@ -219,7 +226,7 @@ When tests fail, examine the diff output carefully. The actual test output in `t pgxntool can generate pg_tle (Trusted Language Extensions) registration SQL for deploying extensions in AWS RDS/Aurora without filesystem access. -**Usage:** `make pgtle` or `make pgtle PGTLE_VERSION=1.5.0+` +**Usage:** `make pgtle` or `make pgtle PGXNTOOL_PGTLE_VERSION=1.5.0+` **Output:** `pg_tle/{version_range}/{extension}.sql` @@ -273,11 +280,11 @@ Generated files depend on: ## Scripts -- **setup.sh** - Initializes pgxntool in a new extension project (copies templates, creates directories) -- **build_meta.sh** - Strips empty fields from META.in.json to create META.json -- **meta.mk.sh** - Parses META.json via JSON.sh and generates meta.mk with Make variables -- **JSON.sh** - Third-party bash JSON parser (MIT licensed) -- **safesed** - Utility for safe sed operations +Each script at the top level of this repository (pgxntool/) begins with a shebang followed by a short +header comment describing its purpose within the first ~10 lines — read that header +rather than relying on a hand-maintained list here, which would just go stale the same +way this section itself once did. Exception: trivial third-party or one-line utilities +may lack such a header. ## Related Repositories diff --git a/pgxntool/HISTORY.asc b/pgxntool/HISTORY.asc index 231809f..bdd5121 100644 --- a/pgxntool/HISTORY.asc +++ b/pgxntool/HISTORY.asc @@ -1,3 +1,30 @@ +2.3.0 +----- +== Rename `PGTLE_VERSION` to `PGXNTOOL_PGTLE_VERSION` +`make pgtle`'s version-limiting variable was named `PGTLE_VERSION`, which +make auto-imports from an identically-named environment variable. That name +is also a natural choice for a CI job's "which pg_tle to test against" env +var -- when one was set, `make pgtle`/`make run-pgtle` silently misbehaved +instead of erroring. Renamed to `PGXNTOOL_PGTLE_VERSION` to avoid the +collision. If you invoke `make pgtle PGTLE_VERSION=...` directly, update it +to `PGXNTOOL_PGTLE_VERSION`. + +== Fix `DATA` wildcard skipping historical full-install version scripts +`base.mk`'s `DATA` wildcard (`sql/*--*--*.sql`) only matched upgrade scripts +(two `--` separators), silently skipping historical full-install scripts +like `sql/ext--0.9.6.sql` (one `--`) even though these are meant to be +committed to git. `make install` never placed them in the extension +directory, so `CREATE EXTENSION ext VERSION 'x.y.z'` failed for any version +older than current. Widened to `sql/*--*.sql`, which matches both. + +== Fix `make test` always exiting 0, even when every test fails +`.IGNORE: installcheck` made `make` treat `installcheck` as always successful +regardless of `pg_regress`'s exit status, and `test`'s recipe only `cat`ed +`regression.diffs` for visibility without checking it. `make test` now exits +non-zero when `regression.diffs` is non-empty, after printing it. + +Issues fixed in this release: #35, #49, #78, #79, #48 + 2.2.0 ----- == Add `check-stale-expected` to catch orphaned test/expected files diff --git a/pgxntool/README.asc b/pgxntool/README.asc index a799c7c..45c4b59 100644 --- a/pgxntool/README.asc +++ b/pgxntool/README.asc @@ -51,6 +51,8 @@ Whether `test-build` runs is controlled by the `PGXNTOOL_ENABLE_TEST_BUILD` vari NOTE: `test` intentionally does *not* depend on `clean` — that caused problems with incremental/watch-based builds. If your tests need a clean build to pass, that's a sign of a missing dependency elsewhere rather than something to fix by adding `clean` back. +NOTE: `test` exits non-zero (after printing `regression.diffs`) if any test fails. Previously it always exited 0 regardless of test results, silently masking failures from CI and other automation that relies on the exit code. + NOTE: While you can still run `make installcheck` or any other valid PGXS make target directly, it's recommended to use `make test` when using pgxntool. The `test` target ensures proper test isolation and correct dependency installation. === test-build @@ -270,7 +272,7 @@ WARNING: If your project has a `.gitattributes` file, `make dist`/`make dist-onl `make forcedist` is a shortcut for `forcetag dist`: it force-recreates the tag (see `forcetag` above) before rebuilding the distribution .zip. === pgxntool-sync -This rule will pull down the latest released version of PGXNtool via `git subtree pull` and then reconcile the files `setup.sh` copied into your project (`.gitignore`, `test/deps.sql`) with a 3-way merge. +This rule will pull down the latest released version of PGXNtool via `git subtree pull` and then reconcile the files `setup.sh` copied into your project (`.gitignore`, `test/deps.sql`) with a 3-way merge (it also verifies the `test/pgxntool` symlink, recreating it if missing). NOTE: Your repository must be clean (no modified files) in order to run this. Running this command will produce a git commit of the merge. @@ -308,10 +310,10 @@ Generates pg_tle (Trusted Language Extensions) registration SQL files for deploy `make pgtle` generates SQL files in `pg_tle/` subdirectories organized by pg_tle version ranges. For version range details, see `pgtle_versions.md`. -Set `PGTLE_VERSION` on the command line to limit generation to the single version range that value falls into, instead of every known range: +Set `PGXNTOOL_PGTLE_VERSION` on the command line to limit generation to the single version range that value falls into, instead of every known range: ---- -make pgtle PGTLE_VERSION=1.5.0 +make pgtle PGXNTOOL_PGTLE_VERSION=1.5.0 ---- === check-pgtle @@ -457,6 +459,21 @@ If you need to support multiple versions of your extension: The version file for the current version (specified in the `.control` file's `default_version`) will be automatically regenerated when you run `make`, but other version files you create manually will be preserved. +=== Gotchas + +==== Generated SQL Files Can Be Missed on a Clean Build + +`DATA` (the PGXS variable controlling what `make install` actually installs) is seeded by pgxntool with `$(wildcard sql/*--*.sql)`, so PGXS installs whatever version-specific and upgrade files exist under `sql/` when `DATA` is evaluated. + +If your extension *generates* its versioned SQL from another source (e.g. a `.sql.in` template expanded by a custom rule) and gitignores the generated `.sql`, a clean `make install` (or `make test`, `make all install` in one invocation) can silently omit files that were only just generated during that same run. This isn't a pattern-matching problem — the glob is correct — it's a GNU Make quirk: `wildcard` caches a directory's contents internally for the life of one `make` invocation, and that cache isn't invalidated just because a recipe created new files earlier in the same run. So even though `DATA` is a recursively-expanded variable (re-evaluated each time it's referenced), re-referencing it later in the same invocation can still see the stale, pre-generation listing. + +Workarounds: + +- Run generation and install as separate `make` invocations (e.g. `make && make install`) so the second invocation's parse-time wildcard scan sees files the first invocation created. +- List the generated files in `DATA` explicitly, keyed off stable names derived from your `.sql.in` sources (known at parse time) rather than relying on `wildcard` to discover the generated `.sql` output. + +Tracked as https://github.com/Postgres-Extensions/pgxntool/issues/44[issue #44] — pgxntool doesn't yet have first-class support for generated versioned SQL; the ideas under discussion there (e.g. a dedicated generated-output directory pgxntool builds before installing) are a bigger design change than a one-line fix. + == Document Handling PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior. @@ -665,9 +682,9 @@ make dist PGXN_REMOTE=upstream Default: auto-detected, the first of `asciidoctor` or `asciidoc` found on `PATH`. Path to the Asciidoc processor used to build `.html` files from `$(ASCIIDOC_EXTS)` source files. Override if the processor you want isn't first on `PATH`, or isn't on `PATH` at all. See <<_document_handling>>. -=== PGTLE_VERSION +=== PGXNTOOL_PGTLE_VERSION -Default: unset (generates every known pg_tle version range). Set on the command line to limit `make pgtle` to the single version range this value falls into. See <<_pgtle>>. +Default: unset (generates every known pg_tle version range). Set on the command line to limit `make pgtle` to the single version range this value falls into. Not named `PGTLE_VERSION`: make auto-imports same-named environment variables, and that name collided silently with CI jobs that set a `PGTLE_VERSION` env var for an unrelated purpose (which pg_tle to test against). See <<_pgtle>>. === PG_CONFIG diff --git a/pgxntool/README.html b/pgxntool/README.html index 36d3cf5..ba71c2a 100644 --- a/pgxntool/README.html +++ b/pgxntool/README.html @@ -475,6 +475,7 @@

PGXNtool

  • 5.5. Alternative: Ignoring All Version Files
  • 5.6. Distribution Inclusion
  • 5.7. Multiple Versions
  • +
  • 5.8. Gotchas
  • 6. Document Handling @@ -499,7 +500,7 @@

    PGXNtool