Skip to content

Allow non matching schemas on EXPORT PARTITION#2111

Open
k-morozov wants to merge 6 commits into
antalya-26.3from
feature/antalya/26.3/allow_non_matching_schema_export_partition
Open

Allow non matching schemas on EXPORT PARTITION#2111
k-morozov wants to merge 6 commits into
antalya-26.3from
feature/antalya/26.3/allow_non_matching_schema_export_partition

Conversation

@k-morozov

@k-morozov k-morozov commented Jul 23, 2026

Copy link
Copy Markdown

Closes: #1717

Problem. EXPORT PART/EXPORT PARTITION matched source and destination columns positionally, exactly like INSERT INTO dest SELECT * FROM src, and required the column count to match exactly. This is unnecessarily strict for a common
real scenario: a MergeTree table's schema grows over time (new trailing columns added), but older partitions still need to be exported to an external table (Iceberg/object storage) whose schema was fixed when it was created. Such exports were rejected outright with NUMBER_OF_COLUMNS_DOESNT_MATCH, with no way to just drop the newer columns and export the rest.

Solution. Added the export_merge_tree_part_schema_mismatch_mode setting (default strict, preserving the old behavior). The new ignore_extra_source_columns_by_position mode allows the source to have more columns than the destination: the extra trailing source columns (by declared/readable position, the same order INSERT SELECT uses) are dropped, both at synchronous validation time (verifyExportSchemaCastable, so a real mismatch still fails immediately at ALTER TABLE ... EXPORT time) and at actual data-export time (ExportPartTask::addExportConvertingActions, via a projection step before the existing positional CAST). The reverse direction — destination has more columns than source — is still always rejected in both modes.

Known follow-up. Ignored columns are still fully read and decompressed from disk before being discarded in the projection step, so no I/O is saved. Skipping the read up front isn't safe as a naive truncation: a kept
ALIAS/MATERIALIZED column can legally forward-reference an ignored one, so it would need a real dependency-graph walk instead. Left as a possible future optimization

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added the export_merge_tree_part_schema_mismatch_mode setting. With ignore_extra_source_columns_by_position (default: strict), EXPORT PART/EXPORT PARTITION allows a source table with extra trailing columns — they are simply
ignored instead of failing with NUMBER_OF_COLUMNS_DOESNT_MATCH.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov
k-morozov force-pushed the feature/antalya/26.3/allow_non_matching_schema_export_partition branch from 0d6a23a to 267e79f Compare July 24, 2026 07:07
@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown

Workflow [PR], commit [e74a863]

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov k-morozov changed the title [WIP] Allow non matching schemas on EXPORT PARTITION Allow non matching schemas on EXPORT PARTITION Jul 24, 2026
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov
k-morozov marked this pull request as ready for review July 24, 2026 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow non matching schemas on export partition

1 participant