From e00928b8f9bdaacf66dee223ca9baee2eea65916 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 14 Aug 2026 16:16:28 -0500 Subject: [PATCH 1/3] docs(data-integrity): domain integrity extends to OAS + codec types Expand the Domain integrity section to describe how Object-Augmented Schemas and the codec system extend domain integrity beyond scalar SQL types: a codec defines a domain-specific datatype with arbitrary validation checks on encode and typed access methods on decode, and because the object write is bound to the same database transaction as its row, object-backed attributes obey the same transaction discipline (commit/rollback together) as tabular data. --- src/explanation/data-integrity.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/src/explanation/data-integrity.md b/src/explanation/data-integrity.md index 450a7bcc..8281f341 100644 --- a/src/explanation/data-integrity.md +++ b/src/explanation/data-integrity.md @@ -75,9 +75,24 @@ each column: `int32`, `int64`, `float32`, `float64`, `decimal(p, s)`, `varchar(N)`, `char(N)`, `date`, `datetime`, `uuid`, and `enum(...)` for closed categorical sets. Choosing `sex : enum('M', 'F', 'U')` instead of a free-text field means an invalid category can never be stored, and -`decimal(5, 2)` fixes both magnitude and precision. The type system is -extensible: custom codecs attach domain rules to structured and object-backed -attributes as well. +`decimal(5, 2)` fixes both magnitude and precision. + +The type system is **extensible**, and this is where DataJoint carries domain +integrity beyond the scalar SQL types. Through +[Object-Augmented Schemas](data-pipelines.md#object-augmented-schemas) and the +[codec system](../how-to/create-custom-codec.md), an attribute can hold a +domain-specific datatype — a calcium-imaging movie, a spike-sorting result, a +fitted model — whose contents live in object storage but are addressed and +governed by the schema. A [codec](../reference/specs/codec-api.md) *defines* that +datatype: it can run **arbitrary validation checks** when a value is encoded — +rejecting anything outside the domain's valid set, exactly as `enum` does for a +category — and it exposes **typed access methods** for reading the value back. +Crucially, the object write is bound to the **same database transaction** as the +row that references it, so an object-backed attribute obeys the same discipline +as any scalar column: the row and its object commit together or not at all, and +a failed check or write rolls the whole insert back. Domain integrity therefore +holds uniformly — from `int32` and `enum` to arbitrary structured and +object-backed types — under one consistency guarantee. ### 2. Completeness From 30eb54507e8384e74858f1bb1287ac03110d49b0 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 17 Aug 2026 13:21:25 -0500 Subject: [PATCH 2/3] docs(data-integrity): correct the OAS transaction wording Address the accuracy caveat on the codec/domain-integrity paragraph: object stores are not transactional participants, so the earlier 'row and object commit together or not at all' overstated atomicity. Reframe accurately: the codec's validation check runs at encode time (before any write), so an invalid value is rejected up front and its row is never committed; the stored object is schema-addressed and reclaimed by garbage collection when unreferenced, so a failed insert leaves at most an orphaned object, never a bad value in the table. --- src/explanation/data-integrity.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/src/explanation/data-integrity.md b/src/explanation/data-integrity.md index 8281f341..c097cc5e 100644 --- a/src/explanation/data-integrity.md +++ b/src/explanation/data-integrity.md @@ -87,12 +87,16 @@ governed by the schema. A [codec](../reference/specs/codec-api.md) *defines* tha datatype: it can run **arbitrary validation checks** when a value is encoded — rejecting anything outside the domain's valid set, exactly as `enum` does for a category — and it exposes **typed access methods** for reading the value back. -Crucially, the object write is bound to the **same database transaction** as the -row that references it, so an object-backed attribute obeys the same discipline -as any scalar column: the row and its object commit together or not at all, and -a failed check or write rolls the whole insert back. Domain integrity therefore -holds uniformly — from `int32` and `enum` to arbitrary structured and -object-backed types — under one consistency guarantee. +The check runs at encode time, *before* anything is written, so an invalid +value is rejected up front and the row that would reference it is never +committed — the same guarantee scalar columns give, extended to object-backed +types. The stored object is integrated with the row's lifecycle rather than +being an opaque side file: it is addressed by the schema and reclaimed by +[garbage collection](referential-integrity.md) when no row references it +(object stores are not transactional participants, so a failed insert leaves at +most an unreferenced object, never a bad value in the table). +Domain integrity therefore holds uniformly — from `int32` and `enum` to +arbitrary structured and object-backed types. ### 2. Completeness From 194d42d139c85a09fa3fd1a2c8d6b7ce24dc5e06 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Tue, 18 Aug 2026 16:58:40 -0500 Subject: [PATCH 3/3] docs(data-integrity): garbage-collection link points at the garbage-collection page --- src/explanation/data-integrity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/explanation/data-integrity.md b/src/explanation/data-integrity.md index c097cc5e..5b792209 100644 --- a/src/explanation/data-integrity.md +++ b/src/explanation/data-integrity.md @@ -92,7 +92,7 @@ value is rejected up front and the row that would reference it is never committed — the same guarantee scalar columns give, extended to object-backed types. The stored object is integrated with the row's lifecycle rather than being an opaque side file: it is addressed by the schema and reclaimed by -[garbage collection](referential-integrity.md) when no row references it +[garbage collection](../how-to/garbage-collection.md) when no row references it (object stores are not transactional participants, so a failed insert leaves at most an unreferenced object, never a bad value in the table). Domain integrity therefore holds uniformly — from `int32` and `enum` to