perf(client): parse each Elasticsearch page once on the scroll hits path (softclient4es-arrow#160)

[fupelaqu]

What

Parse each Elasticsearch page once on the scroll / PIT+search_after hits paths (es8 + es9 java clients). Previously every page was parsed three times: the typed client parsed the HTTP response (SearchResponse[JMap[String,Object]]), convertToJson re-serialized the whole response to a JSON String, and core's parseResponse re-parsed that string into a Jackson tree. JFR on the Flight SQL sidecar during a JOIN-leg extraction measured the string round-trip alone at ~19% of total CPU — per-character string writing (WriterBasedJsonGenerator._writeString2, double→ASCII) and re-parsing (ReaderBasedJsonParser._finishString/_parseName, ASCII→double), all scaling with the number and width of selected columns.

How

• The paging searches (pitSearchAfter, scrollClassic initial + continuation) now run with document type ObjectNode, so each _source is materialized exactly once as the Jackson tree the row parser consumes.
• New hitsToResponseNode builds the response-envelope node directly from the typed response: _id + the re-parented _source tree per hit — no serialization, no re-parse. Core's parseSimpleHits reads exactly _id, _source, inner_hits, fields per hit; hits carrying inner_hits or fields (UNNEST, script fields) fall back to a whole-hit convertToTree for full shape fidelity.
• New convertToTree serializes via a Jackson TokenBuffer (JacksonJsonpGenerator over the buffer, readTree(buffer.asParser())) — a token-level copy with no string materialization. Aggregation-bearing responses (at most one per query) go through it instead of StringWriter + readTree.
• Callers now use core's existing node-level parseSingleSearchResponse instead of parseResponse(String). One-shot search / msearch paths are unchanged.

Why (softclient4es-arrow#160)

Post-join aggregation (J2) cost +9.8 s over the bare join (J0) — ~3× Trino's marginal cost. Phase instrumentation showed the DuckDB join+aggregate+stream phase at ~850 ms for both J0 and J2; the entire marginal cost was the 10M-row leg's extraction slowing from 256.7k to 227.1k docs/s because the aggregate needs one extra keyword column, and per-column extraction CPU was dominated by the double parse.

Measured on the arrow#160 benchmark corpus (overlay image = published 0.2.5-SNAPSHOT sidecar + these jars, same machine/day, INFO logging):

	J0 (join only)	J2 (join + GROUP BY)	J2 − J0
before	40.01 s	44.99 s	+4.98 s
after	28.34 s	31.06 s	+2.72 s

10M-leg extraction rate: 256.7k → 364.5k docs/s (J0) and 227.1k → 332.0k docs/s (J2). Row-count oracles exact on J0/J1/J2 (1,000,000 / 125,361 / 100).

Behaviour note

The old string round-trip serialized with JacksonConfig (Include.NON_NULL), which silently dropped null-valued _source entries; the direct tree preserves them as explicit nulls. Rows for requested fields are unchanged (normalizeRow fills nulls either way); only SELECT * rows can now surface a null-valued column that previously vanished.

Tests

• es8: es8java/testOnly *JavaClient* — 299 passed, 0 failed (real ES 8.18.3, Docker), covering ScrollCompleteness, SelectCompleteness (incl. script-fields shape), LimitCompleteness, GroupByCompleteness, WindowPartitionCompleteness, HitMetadata, GatewayApi.
• es9: es9java/testOnly *JavaClient* — real ES 9.0.3, Docker.
• Cross-compiled 2.12 + 2.13; scalafmt + headerCheck clean.

Second commit — #228: single parse everywhere (es6/es7 + one-shot paths)

Commit 848ff277 extends the single-parse contract to the rest of the ecosystem. Closes #228.

• Executor contract: executeSingleSearch / executeMultiSearch (sync + async) now return Option[JsonNode] instead of Option[String]; core gains the node-level parseResponseTree dispatch and parseInnerHits moved from Gson to Jackson. Internal private[client] surface only — no downstream references exist.
• es6/es7 REST: one-shot search, msearch and all scroll/search_after/PIT paging now run over the low-level RestClient, Jackson-parsing the raw response entity bytes exactly once (the typed path parsed every page three times). Request bodies still come from the typed builders; endpoints are percent-encoded like RequestConverters; msearch sends UTF-8 bytes as bare application/json (ES 6.8 rejects application/x-ndjson; charset=UTF-8 with a 406 — caught by the suite).
• es6 Jest: parses the retained raw body (getJsonString) — the Gson re-serialization pass is gone.
• es8/es9: one-shot search/msearch use document type ObjectNode + new searchResponseToTree/msearchResponseToTree (same re-parenting technique as this PR's scroll fix; msearch failure items keep full fidelity so core still sees their error).

Hardening from the adversarial review (Blind Hunter + Edge Case Hunter, all fixes suite-validated):

• Permanent 4xx on paging fails fast instead of burning retries (ResponseException is an IOException, which retryWithBackoff retries; 408/429/5xx stay retriable — a resilience gain).
• A first scroll page answering 200-with-failed-shards now releases the scroll context it created.
• A hit page without sort values aborts paging loudly instead of silently restarting from page one (unbounded duplicates).
• Paging streams now FAIL on error instead of ending quietly: a mid-scroll failure used to surface a silently truncated result set as a successful result (same defect class as SELECT without LIMIT returns only 10 rows on the non-scroll search path #209/SELECT with an explicit LIMIT above index.max_result_window fails, while the same query with NO LIMIT succeeds #224). Applied to es6-rest/es6-jest/es7-rest and the same pre-existing swallow on es8/es9; a continuation shard failure is non-retriable since re-polling a scroll cursor skips rows. Core's scrollRows/async recovers translate the failure into a proper ElasticFailure.

Behaviour notes (release-note material): the null-survival note above now applies to es6/es7 and to one-shot paths as well (one-shot and scroll rows are now consistent for SELECT *); paging pages with partial shard failures fail loudly on es6/es7 (they were silently accepted); mid-stream paging errors surface as query failures instead of silently truncated successful results (all versions).

Tests (all on the final code, real ES via Docker): core 746 unit tests; es6rest 303; es6jest 290; es7rest 305; es8java 316; es9java 316; 2.12+2.13 cross-compile; scalafmt + headerCheck clean. New unit contracts: parseResponseTree cases in ElasticConversionSpec, searchResponseToTree/msearchResponseToTree in JavaClientConversionSpec (es8+es9).

Remaining follow-up (not in this PR): ArrowTypeMapping.fillBatch (~9% CPU) is the next-order term, in softclient4es-arrow.

Closes #228

🤖 Generated with Claude Code