Automate enum sync for spec files

[MonkeyCanCode]

Summary

The current core-spec/spec.yaml has 2 enum sections which would require manual sync from core-spec/osi-schema.json. This PR addresses the requested TODO for automating this sync. Also, I removed the TODO annotation and replaced with # Auto-generated from osi-schema.json (xxxx).

Things not covered in this PR:

1. CI pipeline that will perform check and raise error if there is a diff detected

• this is due i am not sure how we want to handle dependency for this script (in this case PyYAML). Do we create a different pyproject for it? more detail in ml. As the discussion is still on-going, thus I follow the same pattern validation/validate.py is using by checking if a package is importable or not.

2. Documentation on where this should be run

• I am not able to find where will be a good spot to document this. I am more toward to make this as part of pre-commit and have it control via make/just instead.

Sample outputs:

# helper
(venv) ➜  ossie git:(automate_enum_sync) python3 scripts/generate-spec-types.py
usage: generate-spec-types.py [-h] [--check] [--apply]

options:
  -h, --help  show this help message and exit
  --check     Check spec.yaml is in sync
  --apply     Update spec.yaml in-place

# check (happy path - no drift)
(venv) ➜  ossie git:(automate_enum_sync) python3 scripts/generate-spec-types.py --check
(venv) ➜  ossie git:(automate_enum_sync) echo $?
0

# drift (caused by manual edit)
➜  ossie git:(automate_enum_sync) git diff | tee
diff --git a/core-spec/spec.yaml b/core-spec/spec.yaml
index 32fbb3e..599f2ae 100644
--- a/core-spec/spec.yaml
+++ b/core-spec/spec.yaml
@@ -36,7 +36,6 @@ dialects:
   - "TABLEAU"               # Tableau
   - "DATABRICKS"            # Databricks SQL
   - "MAQL"                  # GoodData MAQL (Multi-Dimensional Analytical Query Language)
-  - "BIGQUERY"              # Google BigQuery GoogleSQL

 # Supported logical data types for fields and metrics
 # TODO: Generate this list from the authoritative DataType enum in
@@ -50,7 +49,6 @@ datatypes:
   - "Date"                  # Calendar date without time of day
   - "Time"                  # Time of day without a date or timezone
   - "DateTime"              # Date and time without a timezone or offset
-  - "DateTimeTz"            # Instant identified using offset or timezone context
   - "Opaque"                # Known type outside the portable vocabulary

# check (un-happy path - drift)
(venv) ➜  ossie git:(automate_enum_sync)  python3 scripts/generate-spec-types.py --check
--- /Users/yong/Desktop/GitHome/ossie/core-spec/spec.yaml (current)
+++ /Users/yong/Desktop/GitHome/ossie/core-spec/spec.yaml (expected)
@@ -37,6 +37,7 @@
   - "TABLEAU"               # Tableau
   - "DATABRICKS"            # Databricks SQL
   - "MAQL"                  # GoodData MAQL (Multi-Dimensional Analytical Query Language)
+  - "BIGQUERY"              # Google BigQuery GoogleSQL

 # Supported logical data types for fields and metrics
 # Auto-generated from osi-schema.json ($defs.DataType.enum).
@@ -49,6 +50,7 @@
   - "Date"                  # Calendar date without time of day
   - "Time"                  # Time of day without a date or timezone
   - "DateTime"              # Date and time without a timezone or offset
+  - "DateTimeTz"            # Instant identified using offset or timezone context
   - "Opaque"                # Known type outside the portable vocabulary

 # Vendor name for custom extensions (free-form string)
/Users/yong/Desktop/GitHome/ossie/core-spec/spec.yaml is out of sync with /Users/yong/Desktop/GitHome/ossie/core-spec/osi-schema.json. Run --apply to fix.

# status code from un-happy path
(venv) ➜  ossie git:(automate_enum_sync) echo $?
1

# apply fix
(venv) ➜  ossie git:(automate_enum_sync) python3 scripts/generate-spec-types.py --apply
Wrote /Users/yong/Desktop/GitHome/ossie/core-spec/spec.yaml

# validation
(venv) ➜  ossie git:(automate_enum_sync) python3 scripts/generate-spec-types.py --check
(venv) ➜  ossie git:(automate_enum_sync) echo $?
0

Related Issues

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue
• Breaking changes to the spec are clearly called out in the summary

Ontology

• Ontology changes in ontology/ are consistent with spec changes
• New or modified terms are defined and documented

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes
• New converters include tests under the converter's test directory

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate
• CONTRIBUTING.md is updated if the contribution process changed

Examples

• examples/ are added or updated for any new spec constructs or converter support

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval

[jbonofre]

Thanks for tackling the enum-sync TODO! I ran the script locally: --check passes on the committed tree, --apply is idempotent, and drift is detected/reconciled correctly. Behavior is solid.

One overall design note plus a few inline comments below. Nothing here is a blocker.

Main point: the per-value descriptions are hardcoded in the script rather than sourced from the schema, so this keeps the values in sync automatically (which is what the TODO asked for) but moves the descriptions sync burden from spec.yaml into the script. See the inline comment for details.

[jbonofre]

The per-value descriptions are hardcoded here. osi-schema.json only carries a single description per enum (e.g. "Supported SQL and expression language dialects"), not per-value text — so today they genuinely can't be read from the schema.

Consequence: when a new value is added to the schema, --apply emits it as a bare entry with no comment, and someone has to remember to add its description to this dict. I confirmed this by appending NEWSQL to the schema:

  - "BIGQUERY"              # Google BigQuery GoogleSQL
  - "NEWSQL"

So the manual-sync burden isn't eliminated, just relocated. A cleaner long-term design would put per-value descriptions in the schema (single source of truth) and generate both. Not a blocker for this PR — just worth calling out.

[MonkeyCanCode]

For sure, those appears to be added manually assuming as I was not able to find where they were coming from. Now to your point with add those into the schema, we would need to change the schema here. I think we should start a ML on this before making change here as it is a breaking change due to lack of native per enum value description support. In this case, no change will be apply here and I will start a ML for adding this support. WDYT?

[MonkeyCanCode]

Thanks for the review @jbonofre . 4/5 feedbacks had been addressed. Please let me know if we want to proceed with a ML for the enhancement for the last one.