[KYUUBI #7655][SERVER] Support virtual threads in the binary frontend

[wangzhigang1999]

Why are the changes needed?

Virtual threads are mature in JDK 21, which Kyuubi already supports. As an I/O-bound gateway, Kyuubi is a good fit for this execution model.

This PR makes a small first step by adding optional virtual-thread support to the server-side Thrift Binary frontend. It is disabled by default and does not affect engine frontends or other executors.

The implementation preserves the worker concurrency limit and rejection behavior while retaining compatibility with older JDKs.

See #7655 for the motivation and benchmark results.

How was this patch tested?

• Targeted suites passed on JDK 11 and JDK 21.
• dev/reformat
• git diff --check
• Kyuubi Server fast package build.
• Five-minute A/B test with real Spark SQL.

Was this patch assisted by generative AI tooling?

Assisted-by: OpenAI Codex (GPT-5)

[wangzhigang1999]

@pan3793
I finished a larger benchmark on JDK 25, limited to the server-side TBinaryFrontendService change in this PR. The test included 32 runs, each with a 60-second warm-up and a 300-second measurement window.

In the 10,000-connection capacity test, virtual threads reduced the Kyuubi Server OS thread count from 10,053 to 56, NMT thread committed memory from 971.4 MiB to 5.1 MiB, pod peak memory from 1,653.8 MiB to 947.7 MiB, and RSS from 1,217.5 MiB to 888.4 MiB. Binary RPC performance remained close overall, and real Spark SQL showed no meaningful regression. All 16 virtual-thread JFR recordings reported zero jdk.VirtualThreadPinned events on JDK 25.

Since JEP 491 was delivered in JDK 24, these results do not cover synchronized pinning behavior on JDK 21. Do you think we should run a separate JDK 21 comparison before recommending this option on JDK 21 runtimes?

• English report
• Chinese report

[wangzhigang1999]

I ran four targeted JDK 21 tests against the server-side TBinaryFrontendService: idle sessions, operation lifecycle, query-log fetching, and expected-error logging. Each test used a 60-second warm-up and a 300-second measurement window. With JFR pin detection set to 1 ms, all four recordings reported zero VirtualThreadPinned events, with no failures or carrier starvation.

JFR did show monitor contention in synchronous Log4j output, especially during error logging, but it did not cause pinning in these tests. Kyuubi already uses Log4j 2.26.1, whose output path remains synchronized, so upgrading Log4j would not change this. Since the feature is opt-in, I suggest keeping this PR unchanged and documenting that slow synchronous logging sinks may still pose a risk on JDK 21.

These results apply only to the server frontend; engine-side virtual threads would require a separate and more cautious evaluation because they enter longer-running SQL, extension, and external I/O paths.

[pan3793]

I think users who want to use virtual threads should know the pin issue, the current state is good enough.

[wangzhigang1999]

Thanks, merged to master.