[KYUUBI #7664][DOC] Reformat and improve docs for KSHC - #7667
Conversation
| for such cases, consider setting up Cross-Realm Kerberos trusts, then you just need to talk with one KDC. | ||
|
|
||
| For HMS Thrift API used by Spark, it's known that Hive 2.3.9 client is compatible with HMS from 2.1 to 4.0, and | ||
| Hive 2.3.10 client is compatible with HMS from 1.1 to 4.0, such version combinations should cover the most cases. |
There was a problem hiding this comment.
the claim is no longer valid; HIVE-26537 (4.1.0, 4.0.1) actually broke the Thrift wire protocol by removing the Thrift API used by Spark (the get_table method).
There was a problem hiding this comment.
so let's be conservative, replace the 4.0 with 3.1
There was a problem hiding this comment.
Agreed, replaced 4.0 with 3.1 for both Hive client versions.
| Since v1.13.0, KSHC supports Dynamic Partition Pruning (DPP) for partitioned Hive tables, which | ||
| significantly reduces the amount of data scanned when joining against large partitioned tables. | ||
| DPP is implemented by wrapping Spark's built-in `ParquetScan` / `OrcScan`. As a result, native | ||
| engines such as Gluten and Comet cannot recognize these scans by class name, so KSHC-converted |
There was a problem hiding this comment.
we can promise nothing for those ecosystem projects because they use the spark private api, I would drop this statement
There was a problem hiding this comment.
Dropped the statement entirely.
|
|
||
| Currently, KSHC has the following limitations: | ||
|
|
||
| - KSHC does not support Hive functions / UDFs. |
There was a problem hiding this comment.
this might confuse and scare users, since it can be interpreted as:
"all my existing Hive UDFs, including those ADD JAR ... and CREATE TEMPORARY FUNCTION ..., will stop working if I want to use KSHC"
There was a problem hiding this comment.
Rephrased to clarify that only persistent views/UDFs (CREATE VIEW / CREATE FUNCTION / SHOW VIEWS / SHOW FUNCTIONS) fail through a KSHC catalog, while temporary views and temporary functions are unaffected. Also added spark_catalog as the workaround.
| ```{note} | ||
| Catalog-level configurations (`spark.sql.catalog.<catalog>.*`) are captured when the catalog is | ||
| initialized, so they must be present at the Spark application bootstrap (in `spark-defaults.conf` | ||
| or via `--conf`). Setting them through `SET` after the catalog has been created does not take |
There was a problem hiding this comment.
... so they must be present at the Spark application bootstrap (in
spark-defaults.confor via--conf).
I don't think the "must" is correct, the V2 session catalog is lazily initialized and session-isolated, so technically configs will take effect as long as it set before the catalog is initialized, of course, for ONE_FOR_ALL, configs only apply to the first initialized Hive client instance.
What matters here is that, for the kerberized cluster mode, if you want to distribute HMS tokens (non-keytab mode), you must declare all catalogs with metastore.uris at bootstrap so that spark-submit can fetch the HMS tokens ahead.
The above words are just an explanation, which do not fit the docs standard and need to be reorganized and polished
There was a problem hiding this comment.
Good catch, the V2 catalog is lazily initialized and session-isolated, so configs apply as long as they're set before first access. Only the Kerberized cluster mode (no keytab) genuinely needs metastore.uris declared at bootstrap.
2 participants
Why are the changes needed?
How was this patch tested?
Tested by building the documentation and comparing the pages against the live site:
Was this patch assisted by generative AI tooling?
Assisted-by: DeepSeek-V4-Pro