[KYUUBI #7511][SPARK] Start Spark Connect in the Spark SQL engine

[alexandrefimov]

Why are the changes needed?

Part of #7511, the first slice agreed there: Spark Connect running inside the Spark SQL engine, next to the Thrift frontend, so that Spark Connect clients reach a Kyuubi-managed engine — the same SparkContext, the same session extensions, and so the same authorization rules a JDBC client gets.

This PR starts the service and stops there. The endpoint is not advertised in discovery yet, the engine's idle and shutdown checks do not count Connect sessions yet, and the token still travels on the command line; those are the next PRs on this issue, and the new deployment page lists them as limitations so the feature is not mistaken for finished.

Connect is started by configuration alone: the driver plugin goes into spark.plugins and the bound endpoint is read back reflectively. A compile-time reference does not build in the engine module — io.grpc is relocated to org.apache.kyuubi.shade.io.grpc there and the Scala signatures of SparkConnectService carry io.grpc types — which also keeps the default spark-3.5 build free of Connect classes.

Which engines serve Connect is decided in SparkProcessBuilder, where the values are the server's own. kyuubi.engine.spark.connect.enabled is immutable, but the engine rebuilds its KyuubiConf from every spark.kyuubi.* entry it is handed, so the prefixed copy of that key is dropped from the engine's command line. An engine that would not run as the session user — any share level but USER, or kyuubi.engine.doAs.enabled=false — starts without Connect rather than failing, because the share level is the session's own choice. That matters because a plan submitted over Connect runs as the user the engine runs as and not as the client's user_id: on 4.0.3 with the Ranger extension, a client claiming user_id=bob against an engine launched for testalice is denied as testalice.

With Connect on, the engine does refuse to start when the runtime cannot authenticate its clients — Spark 3.5 serves Connect with no authentication at all, spark.connect.authenticate.token arrived in 4.0 — or when no token is configured. An endpoint that authenticates nobody is worse than no endpoint. The Spark 4.0 requirement is a capability probe (Connect.getAuthenticateToken is present) rather than a version comparison, following the "feature detection, not version strings" rule in AGENTS.md.

How was this patch tested?

SparkConnectLauncherSuite covers the configuration the engine writes (the plugin appended once and only once, an ephemeral port unless the deployment picked one, deployment values kept), each of the three refusals, and the reflection itself against stand-in classes shaped like Spark's. SparkProcessBuilderSuite covers the server-side decision, including a session trying to turn Connect on through spark.kyuubi.engine.spark.connect.enabled.

No suite starts the real service: the engine module does not depend on spark-connect, and adding the client stubs is a problem of its own, since Spark 4.x builds them against protobuf 4.x while Kyuubi carries 3.25.8. So this commit was also run end to end against Spark 4.0.3 — spark-4.0.3-bin-hadoop3, server built with -Pspark-4.0 -Pscala-2.13, spark.master=local[*], USER share level, beeline as the client:

• feature on with a token configured: the engine logs Spark Connect service is listening on sc://192.168.1.85:57214 — the port it reports is the one the service bound, read back through the reflection this PR adds — and JDBC queries keep working;
• feature off at the server, client connecting with ?spark.kyuubi.engine.spark.connect.enabled=true: the value does reach the session, SET returns true, while the engine's launch command carries no such config and Connect never starts;
• feature on, session asking for kyuubi.engine.share.level=CONNECTION: the server logs Not starting Spark Connect for this engine: it serves the session user only with kyuubi.engine.share.level=USER and kyuubi.engine.doAs.enabled=true, which are CONNECTION and true here, and that engine starts without Connect;
• feature on with no token: the engine refuses to start with no Spark Connect token is configured. Set spark.connect.authenticate.token or the SPARK_CONNECT_AUTHENTICATE_TOKEN environment variable so that clients are authenticated.

The Spark 3.5 case — Connect without authentication — is covered by the suite only; I did not build a 3.5 runtime for it.

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: Claude Opus 5

[alexandrefimov]

The one red check, Kyuubi and Spark Test (21, 3.11, 4.2, -Pscala-2.13, …), is HadoopCredentialsManagerSuite: schedule credentials renewal timing out: 0 did not equal 1 after 10 attempts over 1.1 seconds. That is the credentials renewal scheduler in kyuubi-server, while this PR touches the Spark engine module, KyuubiConf and SparkProcessBuilder — and the same job ran the new SparkConnectLauncherSuite green, 186 tests in that module. The neighbouring test in that suite already has a flaky-test issue, #2303.

Could someone re-run it? The (17, 3.11, 3.5, …mirror…) job was cancelled in the same run and needs the re-run too.