Skip to content

refactor!: compact metadata and operation arguments - #8196

Merged
Xuanwo merged 7 commits into
mainfrom
xuanwo/compact-metadata
Aug 30, 2026
Merged

refactor!: compact metadata and operation arguments#8196
Xuanwo merged 7 commits into
mainfrom
xuanwo/compact-metadata

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Aug 29, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #8195.

Rationale for this change

RFC-8194 identified Metadata and finalized raw operation arguments as large, allocation-heavy owned values. This change implements the accepted compact immutable representation while keeping metadata independently owned and preserving complete if_not_changed intent until capability-aware dispatch.

What changes are included in this PR?

  • Add a private exact-size compact value block with u16 offsets, shared clones, deterministic user-metadata encoding, and a hard 64 KiB bound.
  • Move metadata construction and transformation to MetadataBuilder, store listed paths in the metadata block, and enforce explicit file content length at the Operator boundary.
  • Freeze public operation options into compact raw arguments without capability-dependent conversion, then select primitive conditional fields at dispatch.
  • Migrate core layers, services, affected bindings, and the object_store integration to the new APIs.
  • Keep the current single OpCopy design and the measured scalar placement. The prototype-only benchmark harness and rejected representations are not part of the long-term suite; layout, boundary, and behavior tests cover the retained invariants.

Performance

A one-off Divan harness compared this PR at 81afb6b1 with the exact pre-refactor structures from base main at 0f4ca09a in the same release binary. Results below are single-threaded medians from an Apple M4 Max (macOS 26.6.2, rustc 1.97.0), using 100 samples and at least one second per benchmark.

Layout

Type main This PR Change
Metadata 264 B 40 B -84.8%
OpRead 256 B 16 B -93.8%
OpWrite 256 B 32 B -87.5%

Retaining 10,000 values

The allocation column reports peak live allocations, not cumulative allocation churn during construction.

Scenario main peak heap / allocations This PR peak heap / allocations Heap change Construction median
List metadata (length, ETag, version) 3.480 MB / 20,001 1.520 MB / 10,003 -56.3% 286.2 µs -> 620.2 µs (2.17x slower)
List entries including paths 3.980 MB / 30,001 1.760 MB / 10,004 -55.8% 634.5 µs -> 1.565 ms (2.47x slower)
Stat metadata (seven strings) 4.540 MB / 70,001 2.640 MB / 10,008 -41.9% 858.1 µs -> 1.494 ms (1.74x slower)
Stat plus eight user-metadata pairs 14.070 MB / 240,001 4.641 MB / 10,025 -67.0% 3.271 ms -> 6.910 ms (2.11x slower)

Clone and clone-then-modify

Scenario Clone: main -> PR Improvement Clone then replace ETag: main -> PR Improvement
List metadata 38.77 ns -> 2.109 ns 18.4x 60.73 ns -> 52.11 ns 14.2%
Stat metadata 115.3 ns -> 1.947 ns 59.2x 136.3 ns -> 66.44 ns 51.3%
Stat plus user metadata 461.5 ns -> 1.804 ns 255.8x 479.9 ns -> 73.61 ns 84.7%

Finalizing and cloning raw arguments

Scenario main -> PR Change
Finalize OpRead 57.08 ns -> 128.8 ns 2.26x slower
Clone OpRead 139.0 ns -> 1.591 ns 87.4x faster; 8 allocations -> 0
Finalize OpWrite 177.6 ns -> 405.8 ns 2.28x slower
Clone OpWrite 485.4 ns -> 1.926 ns 252.0x faster; 25 allocations -> 0

The compact representation deliberately pays more during one-time construction/finalization. It substantially reduces retained memory and makes repeated clones nearly constant-cost through shared immutable storage. The temporary harness was removed after measurement; the accepted RFC retains the broader prototype methodology and design data.

Are there any user-facing changes?

Yes. This is a breaking Rust API change:

  • Metadata::new, Default, set_*, and with_* are replaced by Metadata::builder and Metadata::into_builder.
  • Metadata::user_metadata and OpWrite::user_metadata return a borrowed UserMetadata view with lookup and IntoIterator support.
  • Public with_* mutation methods are removed from finalized OpRead, OpStat, OpWrite, OpDelete, OpCopy, OpList, and OpRestore values; callers convert the corresponding public options instead.
  • File metadata returned through an Operator must include an explicit full-object content length.

The upgrade guide documents these migrations.

Validation

  • cargo nextest run --workspace --no-fail-fast --all-features (704 passed, 5 skipped)
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test -p opendal-core --doc --all-features (159 passed, 12 ignored)
  • cargo doc --lib --no-deps --all-features
  • ./scripts/workspace.py cargo fmt -- --check
  • Focused checks for the C, C++, Node.js, Python, and object_store crates

AI Usage Statement

OpenAI Codex materially assisted with implementation, repository-wide API migration, simplification review, validation, and the one-off benchmark comparison. The accepted RFC and current repository behavior were treated as authoritative. The benchmark used exact pre-refactor layouts from the base commit and current public types from this PR; the temporary harness is intentionally not checked in. The author reviewed the scope and requested this pull request.

@Xuanwo
Xuanwo marked this pull request as ready for review August 30, 2026 17:58
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" labels Aug 30, 2026
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 30, 2026
@Xuanwo
Xuanwo merged commit 7c8f580 into main Aug 30, 2026
808 of 809 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/compact-metadata branch August 30, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-changes lgtm This PR has been approved by a maintainer releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants