feat(client): result rows no longer surface Elasticsearch hit metadata

[fupelaqu]

Summary

SQL result rows no longer surface Elasticsearch hit metadata.

• _index, _score, _sort are removed for good — investigation across the whole ecosystem (core, testkit, REPL, softclient4es-jdbc, softclient4es-arrow, softclient4es-extensions, softclient4es-web) found zero consumers: every downstream surface either ignored them or stripped them defensively. The JDBC driver and the Arrow Flight sidecar actually leaked them to BI tools / advertised them as extra Arrow fields — removal narrows those schemas to the selected columns.

• _id becomes opt-in through a new HOCON setting, disabled by default:

  elastic {
    include-document-id = false   # env override: ELASTIC_INCLUDE_DOCUMENT_ID
  }

Design: decide at injection time, never strip per row

_id has exactly one internal consumer: ranking windows (ROW_NUMBER/RANK/DENSE_RANK, #101) match base rows to their per-partition ordinals by document id (SearchApi.enrichDocumentWithWindowValues). An earlier draft carried _id on every parsed row and stripped it at the public egresses — rejected because it costs a ListMap rebuild per row on the scroll hot path (the exact path arrow#139 is fighting).

Instead, the decision is made once per parse/page and _id is injected only when it will actually be kept:

• parseSimpleHits computes a single boolean — retainDocumentId || includeDocumentId || fields.contains("_id") — before the per-row loop. Default case: no injection, no strip, zero added per-row work anywhere.
• parseResponse (and the chain below it) gains a retainDocumentId: Boolean = false parameter. Only the window-enrichment base query sets it: SearchApi.singleSearchInternal(retainDocumentId = true) on the search side, and ScrollConfig.retainDocumentId on the scroll side (the client modules' page parsers pass config.retainDocumentId through — es6 rest/jest, es7, es8, es9).
• The only rows that ever pay a per-row _id strip are window-enriched ones, where the strip rides the row rebuild the enrichment already performs (enrichResponseWithWindowValues / scrollWithWindowEnrichment).
• Nested per-hit maps (inner hits, non-ranking top_hits objects) gate _id at extraction too; the ranking top_hits branch always keeps it (internal only — the ordinal replaces the window column value).

Changes

• ElasticConversion: extractHitMetadata → extractHitId (only _id, never _index/_score/_sort); includeDocumentId hook + keepsDocumentId/stripDocumentId helpers; retainDocumentId threaded through parseResponse/jsonToRows/parseSimpleHits; ElasticConversion.DocumentIdField constant.
• ElasticConfig (both 2.12 Configs and 2.13 ConfigReader variants): new includeDocumentId: Boolean = false; softnetwork-elastic.conf gains include-document-id + ELASTIC_INCLUDE_DOCUMENT_ID; ElasticClientApi wires it into the conversion layer for every client.
• ScrollConfig: internal retainDocumentId flag (set by ScrollApi, not by callers).
• Client modules (es6 rest/jest, es7 rest, es8/es9 java): scroll page parsers (extractAllResults/extractHitsOnly) pass config.retainDocumentId to parseResponse.
• AggregateApi metadata-key filter reduced to _id; IndicesApi INSERT … AS SELECT keeps a defensive - "_id" so an enabled flag never writes ids into _source.
• Testkit assertion helpers (normalizeRow) now strip only _id/_version, keeping expected-row assertions independent of the client configuration.
• Docs: documentation/client/common_principles.md (config reference + env var), documentation/client/indices.md.

Tests

• Unit (no Docker): ElasticConversionSpec +5 — default parse injects nothing (even with _index/_score/sort in the raw response); retainDocumentId = true carries _id; explicit SELECT _id and the enabled flag carry _id; window-strip semantics; inner-hit _id gating. 742 core + 500 sql + 120 bridge unit tests green; + compile (2.12 + 2.13), scalafmtCheck, headerCheck green.
• Integration — new HitMetadataSpec (testkit template + 5 client subclasses: es6 rest/jest, es7 rest, es8/es9 java), 3-shard index, 8 cases per client:
  • one-shot (LIMIT), scroll-routed (no LIMIT, SELECT without LIMIT returns only 10 rows on the non-scroll search path #209), async, and UNION ALL row queries surface exactly the selected columns;
  • windowed ROW_NUMBER keeps exact per-partition ordinals on both the one-shot and scroll paths — proves the retain channel feeds the ordinal lookup on every client;
  • explicit SELECT _id surfaces the id even with the flag disabled;
  • GROUP BY rows carry no metadata;
  • with include-document-id = true, _id (and only _id) appears on every path and equals the bulk-indexed id. Both spec clients are SPI-instantiated with the flag pinned (the factory caches per cluster URL; ambient ELASTIC_INCLUDE_DOCUMENT_ID can't flip assertions).
• Full ES8 JavaClient* integration suite: 299 passed, 0 failed. ES9/ES7/ES6(rest+jest): HitMetadata + Select/Scroll/WindowPartition completeness + WindowFunction guards green on real clusters.

Known limitations (adversarial review outcome — accepted, flag restores access)

• SELECT _id AS alias yields a null alias column (it was already null before this change; the id itself is reachable via un-aliased SELECT _id or the flag).
• Inner-hit _id projection (UNNEST-style SELECT o._id) requires the flag — nested rows are gated at extraction.
• A SELECT whose fields are all aggregations including a ranking window surfaces the raw top_hits per-hit maps, which carry _id (internal to the ordinal machinery; pre-change they carried all four metadata keys).
• UNION ALL keeps _id per the head request's projection — consistent with the engine's existing UNION semantics (all legs are normalized to the head's column shape).

Downstream notes (0.20.x release note material)

• Behavior change: any consumer that read _id/_index/_score/_sort from result rows implicitly must either select _id explicitly, or enable elastic.include-document-id. Typed searchAs[T] entities with a required _id field fall in this category.
• REPL acceptance suites (repl BOM repo) that pin plain-SELECT stdout columns need their expected output refreshed.
• Arrow Flight sidecar: advertised SELECT schemas narrow by the metadata fields (both QueryRows and QueryStream change together — effectiveSchema's subset check stays consistent).
• softclient4es-jdbc: ResultSetMetaData for ES-path queries now reports only the selected columns (strictly safer for positional access; JdbcIntegrationSpec's >= 3 column bound can be tightened to == 3 on the next core bump).

🤖 Generated with Claude Code