Skip to content

Extract PR #46's manual structural-diff technique into reusable bin/structural_diff tooling #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jnasbyupgrade wants to merge 3 commits into
base: master
Choose a base branch
from
+457 −42
Open

Extract PR #46's manual structural-diff technique into reusable bin/structural_diff tooling #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3b0164f
Add bin/structural_diff, a reusable structural comparison tool, and w…
jnasbyupgrade Jul 30, 2026
8ee184e
Fix a real type-ACL divergence the new structural_diff tool surfaced,…
jnasbyupgrade Jul 30, 2026
3b4ca9a
Stop touching the already-tagged 0.2.3 release; converge the type-ACL…
jnasbyupgrade Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 22 additions & 11 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,28 +606,39 @@ jobs:
# the current version) and are otherwise never exercised. Both origins are
# checked because ALTER EXTENSION UPDATE takes the shortest path (see the
# pg-upgrade-test matrix): 0.2.0 goes via the 0.2.0--0.2.2 script, 0.2.1 via
# 0.2.1--0.2.2. update-check asserts each lands on 0.2.2. No suite runs (the
# current version needs PG12+).
# 0.2.1--0.2.2. update-check-version asserts each lands on 0.2.2 -- NOT
# plain update-check: landing on 0.2.2 from these origins is a KNOWN
# divergence from a fresh 0.2.2 install (trigger__parse and the
# pg_class_v omit_column bug, repaired in 0.2.2->0.2.3; a type-ACL gap,
# repaired only in 0.2.3->0.3.0), and both 0.2.0->0.2.2 / 0.2.1->0.2.2
# are already-published scripts that cannot be edited to fix it
# directly, so asserting fresh-parity here would fail forever by
# design. No suite runs (the current version needs PG12+).
#
# The rebuild_020/rebuild_021 checks exercise the REAL broken path end to
# end from BOTH pre-0.2.2 origins: a 0.2.0 (resp. 0.2.1) install on PG10
# leaves relhasoids in _cat_tools.pg_class_v (the buggy 0.2.0--0.2.2 /
# 0.2.1--0.2.2 omit_column no-op), and updating to 0.2.3 routes through
# 0.2.2--0.2.3 (shortest path <origin>--0.2.2 then 0.2.2--0.2.3), firing
# the conditional rebuild that strips relhasoids. 0.2.3 is the furthest
# PG10 can reach (0.2.3--0.3.0 needs PG12+). On PG12+ relhasoids never
# existed, so only PG10 exercises the rebuild. The psql assertion fails
# loudly if the rebuild did not fire (relhasoids still present) --
# complementing the stronger 10→18 pg_upgrade bridge legs. `$$` is escaped
# as `\$\$` so the shell passes literal dollar quotes through to psql.
# the conditional rebuild that strips relhasoids and the trigger__parse
# repair. It stays update-check-version, not plain update-check: 0.2.3
# is ALSO already-published (tagged), so the type-ACL gap above is not
# (and cannot be) fixed until 0.2.3->0.3.0 either -- this landing point
# still diverges from a fresh 0.2.3 install on that ACL alone. 0.2.3 is
# the furthest PG10 can reach (0.2.3--0.3.0 needs PG12+). On PG12+
# relhasoids never existed, so only PG10 exercises the rebuild. The
# psql assertion fails loudly if the rebuild did not fire (relhasoids
# still present) -- complementing the stronger 10→18 pg_upgrade bridge
# legs. `$$` is escaped as `\$\$` so the shell passes literal dollar
# quotes through to psql.
if: matrix.pg == '10'
run: |
bin/test_existing update-check cat_tools_from_020 0.2.0 0.2.2
bin/test_existing update-check cat_tools_from_021 0.2.1 0.2.2
bin/test_existing update-check-version cat_tools_from_020 0.2.0 0.2.2
bin/test_existing update-check-version cat_tools_from_021 0.2.1 0.2.2
for origin in 020:0.2.0 021:0.2.1; do
db="cat_tools_rebuild_${origin%%:*}"
from="${origin##*:}"
bin/test_existing update-check "$db" "$from" 0.2.3
bin/test_existing update-check-version "$db" "$from" 0.2.3
psql -d "$db" -v ON_ERROR_STOP=1 -c "DO \$\$ BEGIN IF EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid='_cat_tools.pg_class_v'::regclass AND attname='relhasoids' AND NOT attisdropped AND attnum>0) THEN RAISE EXCEPTION 'pg_class_v still exposes relhasoids after update through 0.2.2->0.2.3 -- rebuild did not fire'; END IF; END \$\$"
done
- name: Update 0.2.2 → current and run the suite (existing mode, PG12+)
Expand Down
87 changes: 87 additions & 0 deletions bin/structural_diff
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
#!/usr/bin/env bash
#
# Structurally compare, object-by-object, every member of the cat_tools
# extension in two databases: function bodies (pg_get_functiondef), view
# definitions (pg_get_viewdef), type labels/columns (enums, the one plain
# table, the one standalone composite type), comments, and ACLs. A nonempty
# diff is a bug -- the whole point of an extension UPDATE script is that it
# reaches the SAME objects a fresh install of the target version would.
#
# This generalizes a manual comparison run by hand while fixing
# https://github.com/Postgres-Extensions/cat_tools/pull/46: that PR found a
# real fresh-vs-update divergence this way (cat_tools.trigger__parse, as
# produced by the pre-0.2.2 update scripts, hardcoded `EXECUTE PROCEDURE` and
# skipped an empty-args guard that the fresh-install body had, breaking every
# trigger parse on PG11+). Committing the comparison as a script makes it a
# standing, automated check instead of something that has to be remembered
# and run by hand.
#
# USAGE: bin/structural_diff <subcommand> [args]
#
# dump DB [EXTNAME]
# Print the signature of every EXTNAME member object in DB (EXTNAME
# defaults to cat_tools). Useful on its own for eyeballing one
# database's structure, and it's what `compare` diffs under the hood.
#
# compare DB1 DB2 [EXTNAME]
# Dump both databases and diff them. Prints a unified diff and exits
# non-zero if they differ; exits 0 (and prints an OK line) if
# identical.
#
# See bin/structural_diff.sql for the query that defines "signature" (and how
# it decides which object kinds get a real structural definition vs. falling
# back to just identity/comment/ACL).
set -euo pipefail

SCRIPT_DIR=$(cd "$(dirname "$(readlink -f "$0")")" && pwd)

# ---------------------------------------------------------------------------
# Subcommand implementations
# ---------------------------------------------------------------------------

dump() {
local db=$1 extname=${2:-cat_tools}
psql -d "$db" -v extname="'$extname'" -f "$SCRIPT_DIR/structural_diff.sql"
}

compare() {
local db1=$1 db2=$2 extname=${3:-cat_tools}
# Run in a subshell so the EXIT trap (temp-file cleanup) is scoped to this
# comparison only. A trap set with plain `trap ... RETURN` is NOT scoped to
# the function that set it -- it re-fires on every later function return in
# the same shell, including main()'s, by which point f1/f2 no longer exist.
(
f1=$(mktemp)
f2=$(mktemp)
trap 'rm -f "$f1" "$f2"' EXIT
dump "$db1" "$extname" > "$f1"
dump "$db2" "$extname" > "$f2"
if diff -u --label "$db1" --label "$db2" "$f1" "$f2"; then
echo "OK: '$db1' and '$db2' are structurally identical for extension '$extname'"
else
echo "FAIL: structural diff between '$db1' and '$db2' for extension '$extname' (see diff above) -- an update path reached objects that differ from a fresh install" >&2
exit 1
fi
)
}

usage() {
echo "usage: bin/structural_diff <subcommand> [args]" >&2
echo " dump DB [EXTNAME]" >&2
echo " compare DB1 DB2 [EXTNAME]" >&2
exit 2
}

# Explicit subcommand dispatch on $1, matching bin/test_existing and
# bin/assert_fs_clean's convention.
main() {
local cmd=${1:-}
shift || true
case "$cmd" in
dump) dump "$@" ;;
compare) compare "$@" ;;
*) usage ;;
esac
}

main "$@"
170 changes: 170 additions & 0 deletions bin/structural_diff.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
/*
* Structural signature dump for every object that belongs to an extension
* (pg_depend deptype = 'e'), used by bin/structural_diff to compare a
* database reached via an extension UPDATE against a FRESH install of the
* same target version. Any nonempty diff between two runs of this query is a
* bug: the two paths are supposed to produce byte-identical objects.
*
* Run via: psql -d DBNAME -v extname="'cat_tools'" -f bin/structural_diff.sql
*
* Origin: this generalizes a manual comparison technique (diffing
* pg_get_functiondef / pg_get_viewdef / type labels / comments / ACLs /
* extension membership between a fresh install and an updated database) used
* to find the trigger__parse divergence fixed in
* https://github.com/Postgres-Extensions/cat_tools/pull/46.
*
* Emits one text block per member object, ordered by its pg_describe_object()
* identity so the SAME object sorts to the SAME position regardless of the
* OIDs assigned along each installation path. Each block covers:
* - a structural definition, using pg_get_functiondef/pg_get_viewdef for
* routines/views, an ordered column dump for the one plain table and the
* one standalone composite type this extension has, and an ordered label
* list for enums -- whichever a member's catalog/kind actually calls for.
* Row types implicitly created BY a member relation, and the array type
* shadowing any other member type, are skipped: their structure is fully
* captured by the relation/base-type entry already, so listing them again
* would just duplicate that comparison under a second identity.
* - its comment (pg_description), generically via obj_description().
* - its ACL, generically via whichever ACL column its catalog has (proacl /
* typacl / relacl / nspacl); sorted, since grant order is not meaningful.
*
* This is deliberately NOT specific to cat_tools's current object list: any
* object kind this extension does not (yet) use falls through to the ELSE
* branch below, which still includes it (via its pg_describe_object identity,
* comment and ACL) so a future new member is compared at least at that level
* rather than silently skipped, even though this file does not (yet) know how
* to render a structural definition for it.
*/
\set ON_ERROR_STOP on
\pset format unaligned
\pset tuples_only on
\pset fieldsep ''

WITH ext AS (
SELECT oid FROM pg_extension WHERE extname = :extname
), members AS (
SELECT d.classid, d.objid
FROM pg_depend d, ext
WHERE d.refclassid = 'pg_extension'::regclass
AND d.refobjid = ext.oid
AND d.deptype = 'e'
), skip_shadow AS (
/* Implicit row type of a member relation: same structure as the relation
* itself, so comparing it too would just duplicate that check. */
SELECT t.oid
FROM pg_type t
JOIN members rel ON rel.classid = 'pg_class'::regclass AND rel.objid = t.typrelid
WHERE t.typtype = 'c'
UNION
/* Array type shadowing another member type: same element type, no
* independent structure of its own. */
SELECT t.oid
FROM pg_type t
JOIN members base ON base.classid = 'pg_type'::regclass AND base.objid = t.typelem
WHERE t.typelem <> 0
), acl AS (
SELECT m.classid, m.objid,
(
SELECT array_to_string(array_agg(a::text ORDER BY a::text), ',')
FROM unnest(
CASE m.classid
WHEN 'pg_proc'::regclass THEN (SELECT proacl FROM pg_proc WHERE oid = m.objid)
WHEN 'pg_type'::regclass THEN (SELECT typacl FROM pg_type WHERE oid = m.objid)
WHEN 'pg_class'::regclass THEN (SELECT relacl FROM pg_class WHERE oid = m.objid)
WHEN 'pg_namespace'::regclass THEN (SELECT nspacl FROM pg_namespace WHERE oid = m.objid)
ELSE NULL
END
) a
) AS acl_text
FROM members m
), relation_cols AS (
/* Ordered column dump, shared by the plain-table case (pg_class relkind
* 'r') and the standalone-composite-type case (pg_type typtype 'c' whose
* typrelid is NOT a member relation, i.e. survived skip_shadow) -- both
* describe a set of (name, type, not-null, default) columns identically. */
SELECT m.classid, m.objid,
(
SELECT string_agg(
format(
'%s %s%s%s'
, a.attname
, format_type(a.atttypid, a.atttypmod)
, CASE WHEN a.attnotnull THEN ' NOT NULL' ELSE '' END
, COALESCE(' DEFAULT ' || pg_get_expr(ad.adbin, ad.adrelid), '')
)
, E'\n' ORDER BY a.attnum
)
FROM pg_attribute a
LEFT JOIN pg_attrdef ad ON ad.adrelid = a.attrelid AND ad.adnum = a.attnum
WHERE a.attrelid = CASE m.classid
WHEN 'pg_class'::regclass THEN m.objid
WHEN 'pg_type'::regclass THEN (SELECT typrelid FROM pg_type WHERE oid = m.objid)
END
AND a.attnum > 0
AND NOT a.attisdropped
)
/* Table constraints (PK/UNIQUE/CHECK/FK) have no equivalent on a
* standalone composite type, so this is NULL there and simply appends
* nothing. */
|| COALESCE(
E'\n' || (
SELECT string_agg(pg_get_constraintdef(c.oid), E'\n' ORDER BY c.conname)
FROM pg_constraint c
WHERE m.classid = 'pg_class'::regclass AND c.conrelid = m.objid
)
, ''
) AS cols
FROM members m
WHERE (m.classid = 'pg_class'::regclass AND (SELECT relkind FROM pg_class WHERE oid = m.objid) = 'r')
OR (m.classid = 'pg_type'::regclass AND (SELECT typtype FROM pg_type WHERE oid = m.objid) = 'c')
)
SELECT
'=== ' || pg_describe_object(m.classid, m.objid, 0) || E' ===\n'
|| 'DEFINITION:' || E'\n' || COALESCE(
CASE
WHEN m.classid = 'pg_proc'::regclass
THEN pg_get_functiondef(m.objid)
WHEN m.classid = 'pg_class'::regclass AND (SELECT relkind FROM pg_class WHERE oid = m.objid) IN ('v', 'm')
THEN pg_get_viewdef(m.objid, true)
WHEN m.classid = 'pg_class'::regclass AND (SELECT relkind FROM pg_class WHERE oid = m.objid) = 'r'
THEN (SELECT cols FROM relation_cols rc WHERE rc.classid = m.classid AND rc.objid = m.objid)
WHEN m.classid = 'pg_type'::regclass AND (SELECT typtype FROM pg_type WHERE oid = m.objid) = 'e'
THEN (SELECT string_agg(enumlabel, ',' ORDER BY enumsortorder) FROM pg_enum WHERE enumtypid = m.objid)
WHEN m.classid = 'pg_type'::regclass AND (SELECT typtype FROM pg_type WHERE oid = m.objid) = 'c'
THEN (SELECT cols FROM relation_cols rc WHERE rc.classid = m.classid AND rc.objid = m.objid)
WHEN m.classid = 'pg_type'::regclass AND (SELECT typtype FROM pg_type WHERE oid = m.objid) = 'd'
THEN (
SELECT format(
'base=%s notnull=%s default=%s check=%s'
, t.typbasetype::regtype, t.typnotnull, t.typdefault
, (SELECT string_agg(pg_get_constraintdef(c.oid), ' AND ' ORDER BY c.oid)
FROM pg_constraint c WHERE c.contypid = m.objid)
)
FROM pg_type t WHERE t.oid = m.objid
)
WHEN m.classid = 'pg_cast'::regclass
THEN (
SELECT format(
'CAST (%s AS %s) METHOD %s CONTEXT %s'
, ct.castsource::regtype, ct.casttarget::regtype
, CASE ct.castmethod
WHEN 'f' THEN 'FUNCTION ' || ct.castfunc::regprocedure::text
WHEN 'i' THEN 'INOUT'
WHEN 'b' THEN 'BINARY COERCION'
END
, ct.castcontext
)
FROM pg_cast ct WHERE ct.oid = m.objid
)
ELSE NULL
END
, '(no structural definition rendered for this object kind -- see identity/comment/ACL below)'
)
|| E'\n' || 'COMMENT: ' || COALESCE(obj_description(m.objid, m.classid::regclass::text), '(none)')
|| E'\n' || 'ACL: ' || COALESCE((SELECT acl_text FROM acl WHERE acl.classid = m.classid AND acl.objid = m.objid), '(none)')
|| E'\n'
AS block
FROM members m
LEFT JOIN skip_shadow s ON m.classid = 'pg_type'::regclass AND s.oid = m.objid
WHERE s.oid IS NULL
ORDER BY pg_describe_object(m.classid, m.objid, 0);
Loading
Loading