feat(bigquery): add ArrowDeserializer helper utility

[jinseopkim0]

Stacked PR 2 of 3: Adds the ArrowDeserializer class which handles decoding serialized Arrow schemas and record batches into standard FieldValueList rows.

[gemini-code-assist]

Code Review

This pull request introduces the ArrowDeserializer class to convert Apache Arrow schemas and record batches into BigQuery schemas and FieldValueList rows. Feedback on the implementation highlights several critical issues: a potential native memory leak if vector creation fails, a bug in repeated record deserialization where subfields are omitted, missing defensive checks for schema and size mismatches, and an inefficient timestamp conversion that should cast to TimeStampVector and handle different time units.

[jinseopkim0]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces comprehensive documentation (including developer, user, and storage API guides) and adds support for Apache Arrow serialization and deserialization in the BigQuery client and JDBC driver. It also introduces a centralized type registry and temporal utilities for optimized type coercion. The review feedback highlights several critical improvements: optimizing memory allocation and preventing resource leaks in ArrowDeserializer by passing the BufferAllocator and using proper try-finally blocks; avoiding precision loss in timestamp formatting by using integer division; ensuring that the new Arrow configuration options are actually transmitted to the BigQuery backend by serializing them in QueryJobConfiguration.toPb(); and adhering to timezone handling rules by using legacy Calendar manipulation and LocalTime.ofNanoOfDay for java.sql.Time conversions.

[jinseopkim0]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces Apache Arrow serialization options and deserialization support to the BigQuery client library, alongside extensive documentation and optimized type registry utilities for the JDBC driver. The review feedback highlights critical issues where the new Arrow configuration fields in QueryJobConfiguration are not serialized or deserialized in toPb() and fromPb(), rendering them ineffective. Additionally, the reviewer recommends reusing a single BufferAllocator in ArrowDeserializer to avoid high allocation overhead, ensuring exception-safe LIFO cleanup of FieldVector instances to prevent memory leaks, and using a timezone-aware manual conversion when converting java.sql.Time to java.time.LocalTime to preserve millisecond precision.

[lqiu96]

IIUC, this may have some funky behavior with native memory. Is that why we need to catch Throwable here?

[jinseopkim0]

Yes, this is to prevent leaks.

[lqiu96]

Do we need to close ReadChannel afterwards? Or do we expect this to be the caller's responsibility?

[jinseopkim0]

Thanks for the questions. Closing ReadChannel isn't strictly necessary here because ByteArrayReadableSeekableByteChannel wraps a heap byte[] in memory. However, as a best practice, I've updated the code to wrap it in try-with-resources inside deserializeSchema so callers don't need to worry about channel lifecycle.

[lqiu96]

qq, I'm not entirely sure how this works. Is the size of ArrowRecordBatch and rowBatch always equal? What happens if there is a difference in say pageSize and ArrowRecordBatch?

[jinseopkim0]

Thanks for the questions.

Is the size of ArrowRecordBatch and rowBatch always equal?

No, they are not always equal.

• ArrowRecordBatch is the server's streaming chunk size (typically 1k–4k rows per gRPC response).
• rowBatch is the client's requested page container (pageSize, default 10k rows).

What happens if there is a difference in say pageSize and ArrowRecordBatch?

loadArrowRows iterates through incoming responses and accumulates rows into rowBatch. If a batch fills the remaining capacity of rowBatch (or hits maxResults), it stops mid-batch, marks hasMore = true, and returns the page. The next page fetch will resume reading from the updated offset (totalRowsReturned + rowBatch.size())

We have unit tests in ArrowDeserializerTest covering both multi-batch accumulation and mid-batch stoppage (testLoadArrowRows_multiBatchStream, testLoadArrowRows_respectsPageSize, and testLoadArrowRows_unconsumedBatchRowsSignalHasMore).

[lqiu96]

Any reason in particular for LegacySQLTypeName instead of StandardSQLTypeName?

[jinseopkim0]

Thanks for the question. In the BigQuery Java Veneer library, the Field class has historically used LegacySQLTypeName for all table and query result schema definitions (Field.getType() returns LegacySQLTypeName, and Field.newBuilder(...) requires it).

[lqiu96]

qq, what are the other types where the are child nodes if it's not a list?

[jinseopkim0]

Thanks for the question. In Apache Arrow, the non-list type with child nodes is ArrowType.Struct (representing a BigQuery RECORD / STRUCT). Per the Apache Arrow Columnar Format Specification (Struct Layout) and Arrow Java's Field.java#getChildren(), struct fields store their nested sub-fields inside arrowField.getChildren().