feat: add cross-query metadata and snapshot caching

[manuzhang]

Summary

• add a shared, bounded FileIO content cache for metadata JSON, manifest lists, and manifests
• persist parsed manifest-list entries on snapshots for reuse across scans
• wire Java-compatible cache properties through registry, SQL, and in-memory catalogs
• add cache behavior and concurrency regression coverage

Why

Repeated table loads and scans currently reread immutable Iceberg metadata files and reparse snapshot manifest lists. This adds cross-query reuse while preserving normal data-file I/O paths.

[Copilot]

Pull request overview

This PR introduces cross-query reuse for immutable Iceberg metadata I/O by adding a bounded metadata-content cache in FileIO and persisting parsed snapshot manifest-list entries on Snapshot instances to avoid repeated parsing across scans.

Changes:

• Add MetadataCache + MetadataCacheOptions and integrate it into FileIO via ReadFileCached() / NewCachedInputFile(), with Java-compatible property names and defaults.
• Persist snapshot manifest parsing results via a Snapshot-owned shared cache state and return std::span<const ManifestFile> from SnapshotCache.
• Wire cache enablement through catalogs/registry and add unit tests + build system registration.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/iceberg/update/expire_snapshots.cc	Adjust manifest collection to work with the new span<const ManifestFile> cache surface.
src/iceberg/test/metadata_cache_test.cc	Adds coverage for cache reuse, eviction, invalidation, and concurrency behavior.
src/iceberg/test/meson.build	Registers the new metadata cache test with Meson.
src/iceberg/test/CMakeLists.txt	Registers the new metadata cache test with CMake.
src/iceberg/table_metadata.cc	Switches metadata JSON reads to go through the cached read path.
src/iceberg/snapshot.h	Adds snapshot-shared cache state and tightens manifest spans to const.
src/iceberg/snapshot.cc	Implements snapshot-shared manifest-list parsing/cache coalescing.
src/iceberg/metadata_cache.h	Introduces the public cache/options API and property contract.
src/iceberg/metadata_cache.cc	Implements the bounded, expiring, coalescing metadata cache.
src/iceberg/meson.build	Adds metadata_cache.cc and installs metadata_cache.h.
src/iceberg/manifest/manifest_reader.cc	Enables metadata-content caching for manifests and manifest lists via reader options.
src/iceberg/file_reader.h	Adds ReaderOptions::cache_content to control metadata caching.
src/iceberg/file_io.h	Adds cache configuration and cached read/input APIs to FileIO.
src/iceberg/file_io.cc	Implements cached reads, cached input wrapping, and cache configuration plumbing.
src/iceberg/file_io_registry.cc	Configures metadata caching during registry-based FileIO creation.
src/iceberg/CMakeLists.txt	Adds metadata_cache.cc to the build.
src/iceberg/catalog/sql/sql_catalog.cc	Wires cache configuration through SQL catalog properties when enabled.
src/iceberg/catalog/memory/in_memory_catalog.cc	Wires cache configuration through in-memory catalog properties when enabled.
src/iceberg/avro/avro_reader.cc	Passes caching intent down to Arrow input opening.
src/iceberg/arrow/arrow_io.cc	Adds an opt-in cached-input path when cache is enabled + requested.
src/iceberg/arrow/arrow_io_internal.h	Extends Arrow input open helper signature to accept cache intent.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/iceberg/file_reader.h:104

• ReaderOptions now exposes cache_content, but not all reader implementations appear to honor it. For example, parquet_reader.cc still calls arrow::OpenArrowInputStream(options.io, options.path, options.length) without passing options.cache_content, so enabling cache_content would have no effect for Parquet reads. If the intent is that this flag controls whether the underlying FileIO uses NewCachedInputFile, it should be forwarded consistently by readers that open Arrow input streams.

  /// \brief FileIO instance to open the file.
  std::shared_ptr<class FileIO> io;
  /// \brief Cache this immutable metadata file's content when FileIO caching is enabled.
  bool cache_content = false;
  /// \brief The projection schema to read from the file. This field is required.
  std::shared_ptr<class Schema> projection;

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/iceberg/metadata_cache.cc:168

• MetadataCacheOptions::expiration_interval_ms is only validated for non-negativity. Extremely large values can overflow/behave unexpectedly when converted to std::chrono::milliseconds in IsExpired(), leading to incorrect eviction behavior. Consider rejecting values larger than std::chrono::milliseconds::max().count().

    ICEBERG_PRECHECK(options.expiration_interval_ms >= 0,
                     "Metadata cache expiration interval must not be negative: {}",
                     options.expiration_interval_ms);

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/iceberg/snapshot.h:422

• cache_data is eagerly allocated for every Snapshot instance and the manifest-list parse results appear to be retained for the lifetime of the Snapshot (no bounding/eviction). For tables with many snapshots and large manifest lists, this can cause unbounded memory retention across queries. Consider lazy-initializing the cache state on first use and/or adding a bounded/clearable policy similar to MetadataCache.

  /// Internal lazy state shared by Snapshot copies so manifest lists are parsed once.
  mutable std::shared_ptr<internal::SnapshotCacheData> cache_data =
      internal::MakeSnapshotCacheData();

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/iceberg/metadata_cache.cc:45

• ParseBoolean only accepts exact "true"/"false" and rejects common case variants (e.g., "TRUE"), which can break configuration coming from Java-style properties sources. Since this is meant to be Java-compatible, parse booleans case-insensitively.

Result<bool> ParseBoolean(std::string_view key, const std::string& value) {
  if (value == "true") {
    return true;
  }
  if (value == "false") {

src/iceberg/file_io.cc:213

• NewCachedInputFile checks the cache using the caller-provided file_location, but CachedInputFile later uses input_file_->location() as the cache key. If an InputFile implementation normalizes/aliases locations, this inconsistency can cause missed cache hits and duplicate entries; use the InputFile's location consistently.

  if (auto cached = cache->GetIfPresent(file_location)) {
    return std::make_unique<CachedInputFile>(std::move(input_file), std::move(cache),
                                             static_cast<int64_t>(cached->size()));
  }

src/iceberg/file_io.cc:281

• ClearMetadataCache currently clears only the byte-content MetadataCache, but it does not reset the per-FileIO SnapshotCacheData (parsed manifest-list entries). That means manifest-list parsing cache can grow without any clear/invalidation path even when callers explicitly clear metadata caches.

void FileIO::ClearMetadataCache() {
  auto cache = GetMetadataCache();
  if (cache != nullptr) {
    cache->Clear();
  }