feat: add lazy scan planning iterators

[manuzhang]

What changed

• add a fallible, pull-based Iterator<T> utility
• stream manifest entries and file scan task planning without materializing all results
• preserve scan metrics reporting for completed and partially consumed iterators
• update the demo and add coverage for iterator lifetime and table scan planning

Why

The existing scan planning APIs materialize every manifest entry and file scan task before returning. Large tables can therefore require memory proportional to the full scan plan. The new iterator APIs process one manifest batch at a time and let callers consume tasks on demand while keeping the existing eager APIs compatible.

User impact

Callers can use DataTableScan::PlanFilesIterator() to bound planning memory and stop planning early. Existing PlanFiles() behavior remains available. Streaming planning is pull-based and does not eagerly submit manifest work to the configured planning executor.

[Copilot]

Pull request overview

This PR introduces a fallible, pull-based Iterator<T> abstraction and uses it to add streaming (lazy) scan planning APIs so manifest reading and file task planning can be consumed incrementally instead of fully materialized, while preserving scan metrics reporting for fully and partially consumed plans.

Changes:

• Add a generic Iterator<T> interface (Next() + ToVector()) for fallible, lazily produced values.
• Implement streaming manifest-entry reading and streaming file-scan-task planning via new *Iterator() APIs on ManifestReader, ManifestGroup, and DataTableScan (keeping existing eager APIs intact).
• Update examples and add tests covering iterator lifetime/resource ownership and lazy planning behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/iceberg/util/meson.build	Installs the new iterator.h header in Meson builds.
src/iceberg/util/iterator.h	Adds the fallible pull-based Iterator<T> interface and a ToVector() helper.
src/iceberg/type_fwd.h	Forward-declares Iterator<T> for use in public APIs.
src/iceberg/test/table_scan_test.cc	Adds coverage for PlanFilesIterator() behavior and iterator lifetime beyond the scan object.
src/iceberg/test/manifest_reader_test.cc	Adds coverage that manifest entry iterators own reader resources and can outlive the reader.
src/iceberg/table_scan.h	Adds DataTableScan::PlanFilesIterator() public API.
src/iceberg/table_scan.cc	Implements lazy scan planning and metrics reporting for partially consumed iterators.
src/iceberg/manifest/manifest_reader.h	Adds EntriesIterator() / LiveEntriesIterator() streaming APIs (defaulting to eager adaptation).
src/iceberg/manifest/manifest_reader.cc	Implements streaming manifest entry iteration and refactors eager reads to build on iterators.
src/iceberg/manifest/manifest_reader_internal.h	Updates internal reader interface/state to support iterator-owned resources.
src/iceberg/manifest/manifest_group.h	Adds ManifestGroup::PlanFilesIterator() streaming planning API.
src/iceberg/manifest/manifest_group.cc	Implements pull-based task planning iterator over manifest batches and entries.
example/demo_example.cc	Demonstrates consuming scan tasks via PlanFilesIterator() using Next().

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/iceberg/util/iterator.h:62

• Iterator::ToVector() unconditionally moves the element into the output vector. This fails to compile for copy-only T (copy-constructible but not move-constructible). Using std::move_if_noexcept(*value) keeps move semantics for moveable types while falling back to copy when move is unavailable/undesirable.

      values.push_back(std::move(value).value());

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.