Skip to content

docs+install: 0.20.4 sweep, JOIN fan-out (#195), lifted alias restriction, and a JDK bootstrap on Windows - #232

Merged
fupelaqu merged 2 commits into
mainfrom
chore/docs-0.20.4-installer
Aug 13, 2026
Merged

docs+install: 0.20.4 sweep, JOIN fan-out (#195), lifted alias restriction, and a JDK bootstrap on Windows#232
fupelaqu merged 2 commits into
mainfrom
chore/docs-0.20.4-installer

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Two commits, off the merged 0.20.4 main: the Windows installer changes, then the elasticsql half of the post-release documentation sweep. The softclient4es-web half is a separate PR in that repo.

1. feat(install) — bootstrap a JDK on Windows, and make install.cmd self-sufficient

install.cmd downloads install.ps1 when it is not beside it, so one file is a complete install. A local install.ps1 still wins, so a downloaded pair — or a released bundle — is never silently mixed with main. Written as goto + a separate errorlevel test rather than if … call … || exit, because how cmd binds || inside an if body is ambiguous and this is not a platform where a subtlety can be settled by running it.

install.ps1 installs a JDK instead of aborting. Resolution order — and JAVA_HOME is the thing actually tested when it is set, not the PATH java, which is frequently a different and older JVM:

%JAVA_HOME%\bin\java.exe  ->  `java` on PATH  ->  neither, or too old

In the last case it downloads a portable Temurin 17 JDK (Adoptium zip, never an MSI, so no administrator rights) into <install>\jdk. 17 satisfies both floors (11 for ES 6/7/8, 17 for ES 9), so there is one download to reason about rather than two.

  • The archive unpacks as jdk-17.x.y+z\; it is staged and moved one level up so the final JAVA_HOME is the fixed path <install>\jdk — the launcher hard-codes %BASE_DIR%\jdk\bin and must not glob for a name that changes with every Temurin build.
  • Machine state is left alone. JAVA_HOME/PATH are set for the installer's own session only — deliberately not SetEnvironmentVariable(…,"User"), which would silently repoint every other tool on the box and dangle after uninstall.
  • Resolve-Java runs after the -ListVersions early exit (listing versions must never download a JDK) and before bundle selection, which reads the resolved major.

Both generated launchers resolve Java the same way<install>\jdk%JAVA_HOME% → PATH. This is the half that makes the bootstrap real: they invoked bare java before, so a bootstrapped JDK would have been invisible the moment the user opened a new terminal, and the install would look broken for exactly the users the feature is for. They prepend to PATH rather than call an absolute exe, so every java below stays unchanged and nothing has to quote a path containing spaces.

2. docs — versions, JOIN fan-out (#195), lifted alias restriction (arrow#137)

Versions, resolved against the published JFrog listings rather than commit titles — the published set really is engine 0.20.3, drivers/arrow-extensions 0.2.4, community-extensions 0.2.3, bundle line 0.20.1..0.20.3 — so every bump is exactly one release:

from to
engine / bundle examples 0.20.3 0.20.4
jdbc / adbc / flight-sql 0.2.4 0.2.5
arrow-extensions 0.2.4 0.2.5
community-extensions 0.2.3 0.2.4

Deliberately not bumped: dml_statements.md:450"moved off Hadoop entirely in 0.20.3" is a historical fact about which release changed the behaviour. Bumping it would make the sentence false.

The repl.md example listing gains 0.20.4 and its total is corrected 2 → 4: it claimed Total: 2 version(s) under three bullets, and a live -ListVersions run prints Total: 3 version(s) for the three published bundles.

arrow#137 — the restriction is gone. The "Two ORDER BY gotchas" callout documented behaviour that no longer exists. Rewritten against the shipped planner code (JoinPlanner.processPostJoinIds / isSelectAlias / referencesNoColumn) rather than from the issue text, so the boundaries are the real ones: SELECT aliases and ordinals resolve post-join in GROUP BY/HAVING/ORDER BY, case-insensitively — but an alias is still not legal in SELECT/ON/WHERE, and it must be bare, since ORDER BY d.cnt qualifies a name no table owns and dies inside DuckDB. Stated as "since 0.2.5", never "up to and including" — that phrasing is how this file class rots. The second gotcha (a column present on both sides) is untouched: nothing in this train changed it.

#195 — fan-out on a non-unique key. New section with the issue's verified 2 × 3 × 2 = 12 example. Three corrections made against the source rather than transcribing the issue:

  • The issue groups "AVG/SUM" as not visibly breaking. SUM is inflated exactly like COUNT. What uniform duplication preserves is AVG, MIN, MAX — which is precisely the trap: three columns of four corroborate a count that is 6× off. Also noted that AVG is wrong when the fan-out factor varies within a group.
  • The issue calls a truncated fan-out "a wrong answer that returns successfully". The engine reports it (SQLWarning 01004, x-result-truncated on Flight SQL) per this file's own truncation section, which the new text links to instead of contradicting.
  • Cap figures taken from the meters table in this file: Community 10,000 / Pro 1,000,000.

Windows install docs. repl.md documents the Java bootstrap (probe order, Temurin 17 covering both floors, no admin, session-scoped env, uninstall removes it) and the single-file install.cmd path; README's cmd.exe line becomes one command; the install tree gains jdk\.

Verification

  • install.ps1 parses clean; 16/16 checks in a harness that loads the real function bodies out of install.ps1's own AST — not a transcription — and drives Resolve-Java: version parsing (17.0.11→17, 1.8.0_292→8, missing→0); JAVA_HOME above the floor is used with no bootstrap; JAVA_HOME below the floor bootstraps exactly once and repoints session JAVA_HOME/PATH; a JAVA_HOME pointing nowhere is not silently trusted; a failed bootstrap returns false rather than passing silently.
  • -ListVersions still exits 0 against live JFrog and never reaches Java resolution; -Help exits 0; -EsVersion 5 still exits 1. install.cmd stays ASCII + CRLF.
  • Every SQL example added or touched was parse-probed through the real Parser.apply — 6/6 parse and round-trip. That covers COUNT(DISTINCT …), which the new remedy advice depends on and would otherwise have been a guess.
  • ⚠️ No Windows host here, so cmd.exe executing install.cmd and Expand-Archive on a real Temurin zip are not covered.

Version gate — verified on JFrog

Every coordinate this PR writes was HEAD-checked against the live repository, per the sweep rule that versions are verified before they are written:

coordinate state
softclient4es8-java-client 0.20.4 ✅ 200 — and content-checked (ScrollConfig.retainDocumentId, #226)
softclient4es-community-extensions 0.2.4 ✅ 200 — content-checked (ElasticsearchCapability.watcherAvailability, #50)
softclient4es-arrow-extensions 0.2.5 ✅ 200
softclient4es{6,7,8,9}-jdbc-driver 0.2.5 ✅ 200 — 270 / 278 / 301 / 300 MB
softclient4es-jdbc-driver_2.13 0.2.5 ✅ 200
softclient4es{6,7,8,9}-adbc-driver 0.2.5 ✅ 200
softclient4es{6,7,8,9}-arrow-flight-sql 0.2.5 ✅ 200
softclient4es{6,7,8,9}-cli-all_2.13 0.20.4 ✅ 200 — 278 / 287 / 310 / 309 MB, and content-verified

The es8 fat driver was also opened and inspected, since the entire reason jdbc pins arrow is that the fix ships inside that jar: it carries DuckDBSpill (arrow-join 0.2.5, #141), JoinDetector with the CrossIndexJoin classification under arrow/join/routing/, and core 0.20.4's ScrollConfig.retainDocumentId plus the scrollRows rename from #224.

Gate cleared — the REPL bundle is published and verified. The es8 bundle was opened: softclient4es-bundle-info.properties reports exactly this train (engine.version=0.20.4, community.extensions.version=0.2.4, arrow.extensions.version=0.2.5, java.floor=11); Cli.class appears exactly once; licenses/ + NOTICE are present; and the merged META-INF/services/app.softnetwork.elastic.client.ExtensionSpi lists all five extensions including JoinExtension — the one a bad first-merge silently drops, which would ship a bundle with no cross-index JOIN.

The documented -ListVersions example was also checked against a live run rather than left as prose: the installer now prints 0.20.1 / 0.20.2 / 0.20.3 / 0.20.4 and Total: 4 version(s), exactly what this PR writes — which also confirms the Total: 2 it replaced was wrong.

Closes #195

🤖 Generated with Claude Code

fupelaqu and others added 2 commits August 13, 2026 08:36
…sufficient

Two changes to the Windows install path, and one consequence of the second
that must not ship without it.

1. install.cmd downloads install.ps1 when it is not beside it.

Previously it errored and told the user to fetch two files. Now one file is
enough. A LOCAL install.ps1 still wins, so a downloaded pair — or a released
bundle — stays self-consistent instead of being silently mixed with main.

Written as `goto` plus a separate errorlevel test rather than
`if ... call ... || exit`: how cmd binds `||` inside an `if` body is ambiguous,
and this is not a platform where a subtlety can be settled by running it. The
download lives in a subroutine for the same reason the launcher's two FOR loops
are on separate lines — cmd expands every %VAR% in a parenthesised block in ONE
parse pass, so `set "PS1=..."` followed by a use of %PS1% inside that block
would read the OLD value.

2. install.ps1 installs a JDK instead of aborting.

Resolution order, and JAVA_HOME is the thing actually tested when it is set —
not the PATH `java`, which is frequently a different and older JVM:

    %JAVA_HOME%\bin\java.exe  ->  `java` on PATH  ->  neither, or too old

In the last case the installer downloads a portable Temurin 17 JDK (an Adoptium
zip, never an MSI, so no administrator rights) and unpacks it to <install>\jdk.
17 satisfies BOTH floors — 11 for ES 6/7/8, 17 for ES 9 — so there is one
download to reason about rather than two.

The archive unpacks as jdk-17.x.y+z\, a version-stamped directory. It is staged
and MOVED one level up so the final JAVA_HOME is the fixed path <install>\jdk:
the launcher hard-codes %BASE_DIR%\jdk\bin and must not glob for a name that
changes with every Temurin build.

Machine state is left alone. JAVA_HOME and PATH are set for the installer's own
SESSION only — deliberately not SetEnvironmentVariable(...,"User"), which would
silently repoint every other tool on the box. Nothing is needed in later
sessions because of (3).

Get-JavaMajorVersion is split: Get-JavaMajorFromExe takes an explicit exe path,
so JAVA_HOME and PATH are probed by the same code. Resolve-Java runs AFTER the
-ListVersions early exit (listing versions must never download a JDK) and BEFORE
bundle selection, which reads the resolved major. Check-Prerequisites now
asserts that outcome rather than re-deriving it — still exactly ONE parse of
`java -version` per run, so a localised or multi-line output cannot be read two
different ways.

3. The generated launchers resolve Java the SAME way.

This is the half that makes (2) real rather than a one-shot trick: both
launchers invoked bare `java`, so a bootstrapped JDK would have been invisible
the moment the user opened a new terminal — the install would look broken for
exactly the users the feature is for. bin\softclient4es.bat and
bin\softclient4es.ps1 now try <install>\jdk, then %JAVA_HOME%, then PATH.

They PREPEND to PATH rather than call an absolute exe: every `java` below stays
unchanged, and there is no quoting of a path that contains spaces (the default
target is under %USERPROFILE%). `setlocal` scopes the edit to the launcher.

The summary and VERSION now report which JVM was chosen, and the printed tree
shows jdk\ when one was bootstrapped.

Verified on macOS with pwsh 7 — there is no Windows host here, so cmd.exe
executing install.cmd and Expand-Archive on a real Temurin zip are NOT covered:

  * install.ps1 parses clean (Language.Parser.ParseFile, 0 errors)
  * 16/16 checks in a harness that loads the REAL function bodies out of
    install.ps1's own AST — not a transcription — and exercises Resolve-Java:
    version parsing (17.0.11 -> 17, 1.8.0_292 -> 8, missing exe -> 0);
    JAVA_HOME above the floor is used and nothing is bootstrapped; JAVA_HOME
    BELOW the floor bootstraps exactly once and repoints session JAVA_HOME/PATH;
    a JAVA_HOME pointing nowhere is not silently trusted; a failed bootstrap
    returns false rather than passing silently
  * -ListVersions still exits 0 against live JFrog and never reaches Java
    resolution; -Help exits 0; -EsVersion 5 still exits 1
  * install.cmd stays ASCII + CRLF, and its only goto target exists

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ias restriction (arrow#137)

The elasticsql half of the post-release sweep. The softclient4es-web half is a
separate PR in that repo.

VERSIONS. Resolved against the published JFrog listings, not against commit
titles — the currently published set really is engine 0.20.3, drivers /
arrow-extensions 0.2.4, community-extensions 0.2.3, bundle line 0.20.1..0.20.3,
so every bump here is exactly one release:

  engine / bundle examples   0.20.3 -> 0.20.4   README, repl.md, install.{sh,ps1}
  jdbc / adbc / flight-sql   0.2.4  -> 0.2.5    jdbc.md, adbc_driver.md,
                                                arrow_flight_sql.md,
                                                download_analytics.md, README
  arrow-extensions           0.2.4  -> 0.2.5    README
  community-extensions       0.2.3  -> 0.2.4    README

Deliberately NOT bumped — dml_statements.md:450 says local JSON / JSON_ARRAY
"were additionally moved off Hadoop entirely in 0.20.3". That is a historical
fact about which release changed the behaviour, not a current-version reference;
bumping it would make the sentence false.

The repl.md example listing gains 0.20.4 and its total is corrected 2 -> 4. It
claimed "Total: 2 version(s)" under three bullets; a live -ListVersions run
prints "Total: 3 version(s)" for the three published bundles, so the count was
simply wrong.

JOINS — arrow#137, fixed. The "Two ORDER BY gotchas" callout documented a
restriction that no longer exists. Rewritten against the shipped planner code
(JoinPlanner.processPostJoinIds / isSelectAlias / referencesNoColumn) rather
than from the issue text, so the boundaries are the real ones: SELECT aliases
and ordinals resolve post-join in GROUP BY, HAVING and ORDER BY, matching is
case-insensitive — but an alias is still NOT legal in SELECT, ON or WHERE, and
it must be bare, because `ORDER BY d.cnt` qualifies a name no table owns and
dies inside DuckDB. Version boundary stated as "since 0.2.5", never "up to and
including", which is how this file class rots.

The second gotcha — ORDER BY a column present on BOTH sides — is untouched.
Nothing in this train changed it, so nothing new is asserted about it.

JOINS — #195, fan-out on a non-unique key. New section next to that callout,
with the issue's verified 2x3x2 = 12 example. One correction to the issue's own
suggested wording, because it matters: it groups "AVG/SUM" together as not
visibly breaking. SUM is inflated exactly like COUNT. What is preserved by
uniform duplication is AVG, MIN and MAX — which is precisely the trap, since
three columns of four then corroborate a count that is 6x off. Also notes that
AVG *is* wrong when the fan-out factor varies within a group.

Two further corrections against the source rather than the issue:
  * the issue calls a truncated fan-out "a wrong answer that returns
    successfully". The engine reports it — SQLWarning 01004, and an
    x-result-truncated header on Flight SQL, per this file's own Row truncation
    section, which the new text links to instead of contradicting.
  * cap figures taken from the meters table in this file: Community 10,000 /
    Pro 1,000,000.

WINDOWS INSTALL. repl.md documents the new Java bootstrap (probe order, Temurin
17 covering both floors, no admin, session-scoped env, uninstall removes it) and
the single-file install.cmd path; README's cmd.exe line becomes one command; the
install tree gains jdk\.

Every SQL example added or touched here was parse-probed through the real
Parser.apply — 6/6 parse AND round-trip (the rendered AST re-parses). That
covers COUNT(DISTINCT ...), which the new remedy advice depends on and which
would have been a guess otherwise.

⚠️ NOT YET SATISFIABLE, and a merge gate: the sweep convention is to verify each
version on JFrog BEFORE writing it, because a version with no -all bundle falls
back silently to the plain artifact and the docs end up demonstrating the path
that has no cross-index JOIN. The 0.20.4 / 0.2.5 artifacts are not published at
authoring time. Before merging, HEAD-check: softclient4es{6,7,8,9}-cli-all_2.13
0.20.4, softclient4es{6,7,8,9}-{jdbc,adbc}-driver 0.2.5,
softclient4es{6,7,8,9}-arrow-flight-sql 0.2.5, softclient4es-arrow-extensions
0.2.5, softclient4es-community-extensions 0.2.4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document JOIN fan-out on non-unique keys — correct engine, wrong-looking numbers

1 participant