Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@ curl -fsSL https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/
irm https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.ps1 | iex
```

**Windows (cmd.exe, when `.ps1` files are blocked):** download `install.cmd` and `install.ps1` side by side. It takes the same flags as `install.ps1` and only launches it with `-ExecutionPolicy Bypass`, for that one process.
**Windows (cmd.exe, when `.ps1` files are blocked):** one file — it fetches `install.ps1` if it is not beside it, takes the same flags, and only launches it with `-ExecutionPolicy Bypass`, for that one process.
```bat
install.cmd
curl -O https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.cmd && install.cmd
```

### Connect and Query
Expand Down Expand Up @@ -216,10 +216,10 @@ Download the self-contained fat JAR for your Elasticsearch version:

| Elasticsearch Version | Artifact |
|-----------------------|----------------------------------------|
| ES 6.x | `softclient4es6-jdbc-driver-0.2.4.jar` |
| ES 7.x | `softclient4es7-jdbc-driver-0.2.4.jar` |
| ES 8.x | `softclient4es8-jdbc-driver-0.2.4.jar` |
| ES 9.x | `softclient4es9-jdbc-driver-0.2.4.jar` |
| ES 6.x | `softclient4es6-jdbc-driver-0.2.5.jar` |
| ES 7.x | `softclient4es7-jdbc-driver-0.2.5.jar` |
| ES 8.x | `softclient4es8-jdbc-driver-0.2.5.jar` |
| ES 9.x | `softclient4es9-jdbc-driver-0.2.5.jar` |

> **Java 11+ recommended** (17+ for ES 9.x): **cross-index JOINs require Java 11+** — the embedded JOIN engine is built on Apache Arrow 18.x, which ships Java-11 bytecode.

Expand All @@ -236,20 +236,20 @@ Driver class: app.softnetwork.elastic.jdbc.ElasticDriver
<dependency>
<groupId>app.softnetwork.elastic</groupId>
<artifactId>softclient4es8-jdbc-driver</artifactId>
<version>0.2.4</version>
<version>0.2.5</version>
</dependency>
```

**Gradle:**

```groovy
implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.4'
implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.5'
```

**sbt:**

```scala
libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.4"
libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.5"
```

The JDBC driver JARs are Scala-version-independent (no `_2.12` or `_2.13` suffix) and include all required dependencies.
Expand Down Expand Up @@ -337,13 +337,13 @@ For programmatic access, add SoftClient4ES to your project.
resolvers += "Softnetwork" at "https://softnetwork.jfrog.io/artifactory/releases/"

// Choose your Elasticsearch version
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es8-java-client" % "0.20.3"
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es8-java-client" % "0.20.4"
// Add the community extensions for materialized views (optional)
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-community-extensions" % "0.2.3"
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-community-extensions" % "0.2.4"
// Add the arrow extensions for cross-index JOIN (required for JOINs; Java 11+)
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-arrow-extensions" % "0.2.4"
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-arrow-extensions" % "0.2.5"
// Add the JDBC driver if you want to use it from Scala (optional)
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-jdbc-driver" % "0.2.4"
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-jdbc-driver" % "0.2.5"
```

```scala
Expand Down
14 changes: 7 additions & 7 deletions documentation/client/adbc_driver.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ Download the self-contained fat JAR for your Elasticsearch version:

| Elasticsearch | Artifact |
|----------------|-----------------------------------------------------|
| ES 6.x | `softclient4es6-adbc-driver-0.2.4.jar` |
| ES 7.x | `softclient4es7-adbc-driver-0.2.4.jar` |
| ES 8.x | `softclient4es8-adbc-driver-0.2.4.jar` |
| ES 9.x | `softclient4es9-adbc-driver-0.2.4.jar` |
| ES 6.x | `softclient4es6-adbc-driver-0.2.5.jar` |
| ES 7.x | `softclient4es7-adbc-driver-0.2.5.jar` |
| ES 8.x | `softclient4es8-adbc-driver-0.2.5.jar` |
| ES 9.x | `softclient4es9-adbc-driver-0.2.5.jar` |

### Maven / Gradle / sbt

Expand All @@ -36,20 +36,20 @@ Download the self-contained fat JAR for your Elasticsearch version:
<dependency>
<groupId>app.softnetwork.elastic</groupId>
<artifactId>softclient4es8-adbc-driver</artifactId>
<version>0.2.4</version>
<version>0.2.5</version>
</dependency>
```

**Gradle:**

```groovy
implementation 'app.softnetwork.elastic:softclient4es8-adbc-driver:0.2.4'
implementation 'app.softnetwork.elastic:softclient4es8-adbc-driver:0.2.5'
```

**sbt:**

```scala
libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-adbc-driver" % "0.2.4"
libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-adbc-driver" % "0.2.5"
```

---
Expand Down
10 changes: 5 additions & 5 deletions documentation/client/arrow_flight_sql.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,15 @@ Available images per ES version:
### Fat JAR

```bash
java -jar softclient4es8-arrow-flight-sql-0.2.4.jar
java -jar softclient4es8-arrow-flight-sql-0.2.5.jar
```

| Elasticsearch | Artifact |
|---------------|----------|
| ES 6.x | `softclient4es6-arrow-flight-sql-0.2.4.jar` |
| ES 7.x | `softclient4es7-arrow-flight-sql-0.2.4.jar` |
| ES 8.x | `softclient4es8-arrow-flight-sql-0.2.4.jar` |
| ES 9.x | `softclient4es9-arrow-flight-sql-0.2.4.jar` |
| ES 6.x | `softclient4es6-arrow-flight-sql-0.2.5.jar` |
| ES 7.x | `softclient4es7-arrow-flight-sql-0.2.5.jar` |
| ES 8.x | `softclient4es8-arrow-flight-sql-0.2.5.jar` |
| ES 9.x | `softclient4es9-arrow-flight-sql-0.2.5.jar` |

---

Expand Down
2 changes: 1 addition & 1 deletion documentation/client/download_analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ beacon to a public endpoint with exactly these fields:
|---------------|----------|-----------------------------------------------|
| `source` | `portal` | Where the count came from (the docs button) |
| `driver` | `jdbc` | Which driver family (`jdbc` or `adbc`) |
| `version` | `0.2.4` | The published artifact version |
| `version` | `0.2.5` | The published artifact version |
| `count_delta` | `1` | One download |

A timestamp is added on the server. That is the **entire** record.
Expand Down
14 changes: 7 additions & 7 deletions documentation/client/jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ Download the self-contained fat JAR for your Elasticsearch version. The JARs are

| Elasticsearch | Artifact |
|---------------|----------|
| ES 6.x | `softclient4es6-jdbc-driver-0.2.4.jar` |
| ES 7.x | `softclient4es7-jdbc-driver-0.2.4.jar` |
| ES 8.x | `softclient4es8-jdbc-driver-0.2.4.jar` |
| ES 9.x | `softclient4es9-jdbc-driver-0.2.4.jar` |
| ES 6.x | `softclient4es6-jdbc-driver-0.2.5.jar` |
| ES 7.x | `softclient4es7-jdbc-driver-0.2.5.jar` |
| ES 8.x | `softclient4es8-jdbc-driver-0.2.5.jar` |
| ES 9.x | `softclient4es9-jdbc-driver-0.2.5.jar` |

### Build Tool Integration

Expand All @@ -33,20 +33,20 @@ Download the self-contained fat JAR for your Elasticsearch version. The JARs are
<dependency>
<groupId>app.softnetwork.elastic</groupId>
<artifactId>softclient4es8-jdbc-driver</artifactId>
<version>0.2.4</version>
<version>0.2.5</version>
</dependency>
```

**Gradle:**

```groovy
implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.4'
implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.5'
```

**sbt:**

```scala
libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.4"
libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.5"
```

---
Expand Down
41 changes: 34 additions & 7 deletions documentation/client/repl.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,26 @@ It provides:
> 1.5.x and the JOIN engine is built on Apache Arrow 18.x — both ship Java-11
> bytecode. See [Extensions](#extensions-cross-index-joins-materialized-views).

**On Windows you do not have to install Java yourself.** Since `0.20.4`,
`install.ps1` (and therefore `install.cmd`) resolves Java in this order:

1. `%JAVA_HOME%\bin\java.exe` — when `JAVA_HOME` is set, that is the JVM tested,
not whatever `java` happens to be first on `PATH`; the two frequently differ.
2. the `java` on `PATH`, when `JAVA_HOME` is not set.
3. Neither is present, or the one found is **below the floor** for your ES
version ⇒ the installer downloads a portable **Temurin 17** JDK (a zip from
Adoptium, never an MSI, so it needs **no administrator rights**) and unpacks
it to `<install>\jdk`.

Java 17 satisfies both floors, so there is only ever one JDK to think about. The
bootstrapped JDK lives **inside the install directory**: `uninstall.ps1` removes
it along with everything else, and nothing machine-wide is modified — the
installer sets `JAVA_HOME` and `PATH` **for its own session only**. Later
sessions do not need them, because the generated launcher applies the same order
and finds `<install>\jdk` by relative path.

On Linux and macOS `install.sh` still expects a suitable Java to be present.

### Network Requirements

- Network access to JFrog repository (`softnetwork.jfrog.io`)
Expand Down Expand Up @@ -97,14 +117,18 @@ Windows client default is `Restricted`, and the machine may also be set to
`install.ps1` with `-ExecutionPolicy Bypass` **for that one process**, changing
nothing on the machine and needing no elevation.

Download **both** files, keep them in the same directory, then:
One file is enough. If `install.ps1` is not sitting next to it, `install.cmd`
downloads one:

```bat
curl -O https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.cmd
curl -O https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.ps1
install.cmd
```

A local `install.ps1` always wins, so a downloaded pair stays self-consistent —
put both files in the same directory when you want a pinned copy rather than
whatever is on `main`.

`install.cmd` accepts exactly the flags `install.ps1` does and forwards them
verbatim, so every option, default and fallback documented below applies
unchanged — there is no second implementation to drift:
Expand Down Expand Up @@ -163,8 +187,9 @@ install.cmd -ListVersions -EsVersion 8
• 0.20.1
• 0.20.2
• 0.20.3
• 0.20.4

Total: 2 version(s)
Total: 4 version(s)

To install a specific version:
./install.sh --es-version 8 --version <version>
Expand Down Expand Up @@ -192,7 +217,7 @@ install.cmd -ListVersions -EsVersion 8
./install.sh --list-versions --es-version 8

# Install specific version
./install.sh --es-version 8 --version 0.20.3
./install.sh --es-version 8 --version 0.20.4

# Install for Elasticsearch 9 (requires Java 17+)
./install.sh --es-version 9
Expand All @@ -201,7 +226,7 @@ install.cmd -ListVersions -EsVersion 8
./install.sh --target /opt/softclient4es

# Full custom installation
./install.sh --target ~/tools/softclient4es --es-version 7 --version 0.20.3
./install.sh --target ~/tools/softclient4es --es-version 7 --version 0.20.4
```

#### Windows
Expand All @@ -214,7 +239,7 @@ install.cmd -ListVersions -EsVersion 8
.\install.ps1 -ListVersions -EsVersion 8

# Install specific version
.\install.ps1 -EsVersion 8 -Version 0.20.3
.\install.ps1 -EsVersion 8 -Version 0.20.4

# Install for Elasticsearch 9 (requires Java 17+)
.\install.ps1 -EsVersion 9
Expand All @@ -223,7 +248,7 @@ install.cmd -ListVersions -EsVersion 8
.\install.ps1 -Target "C:\tools\softclient4es"

# Full custom installation
.\install.ps1 -Target "C:\tools\softclient4es" -EsVersion 7 -Version 0.20.3
.\install.ps1 -Target "C:\tools\softclient4es" -EsVersion 7 -Version 0.20.4
```

---
Expand All @@ -247,6 +272,8 @@ softclient4es/
│ # + extension jars and dependencies, see Extensions)
├── logs/ # Log files directory
│ └── softclient4es.log # (created at runtime)
├── jdk/ # Windows only, and ONLY when the installer had to
│ └── bin/java.exe # bootstrap a JDK — the launcher prefers it
├── LICENSE
├── README.md
├── VERSION
Expand Down
43 changes: 42 additions & 1 deletion documentation/sql/joins.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,48 @@ ORDER BY COUNT(*) DESC;
-- Engineering (3), Marketing (2) survive HAVING
```

> **Two ORDER BY gotchas:** the JOIN planner has two ordering restrictions — you cannot `ORDER BY` a **SELECT alias** (use `ORDER BY COUNT(*)`, not `ORDER BY cnt`), and you cannot `ORDER BY` a column that exists on **both** sides of the JOIN (order by a column unique to one side, e.g. `d.dept_name`, not the shared join key `d.dept_id`).
> **SELECT aliases and ordinals work here** — since arrow-extensions **0.2.5** (REPL bundle `0.20.4`, JDBC / ADBC / Flight SQL driver `0.2.5`). `ORDER BY cnt`, `HAVING cnt > 1`, `GROUP BY` on an alias and ordinal forms such as `ORDER BY 2` all resolve against the final SELECT list *after* the join, and alias matching is case-insensitive. Two limits remain: an alias is **not** legal in `SELECT`, `ON` or `WHERE` — nothing has been computed at that point — and it must be written **bare**, since `ORDER BY d.cnt` qualifies a name no table owns and fails inside DuckDB. Before 0.2.5 all of these were rejected with `Ambiguous column`.
>
> **The remaining ORDER BY gotcha:** you cannot `ORDER BY` a column that exists on **both** sides of the JOIN — order by a column unique to one side, e.g. `d.dept_name`, not the shared join key `d.dept_id`.

### JOIN cardinality — fan-out on a non-unique key

A JOIN on a key that is **not unique** on the other side multiplies rows. That is standard SQL and the engine is doing it correctly, but it is the easiest way to get plausible-looking wrong numbers, because the row multiplication is invisible in the output.

With one tenant that has **2** EU error rows, **3** US rows and **2** AP rows:

```sql
SELECT eu.tenant_id,
COUNT(*) AS eu_errors,
AVG(us.latency_ms) AS us_avg_latency,
AVG(ap.latency_ms) AS ap_avg_latency
FROM eu_events AS eu
JOIN us_events AS us ON eu.tenant_id = us.tenant_id
JOIN ap_events AS ap ON eu.tenant_id = ap.tenant_id
WHERE eu.level = 'ERROR'
GROUP BY eu.tenant_id;
```

`eu_errors` comes back as **12** — that is 2 × 3 × 2, one row per combination. The answer meant by the query is **2**.

**Why this is worse than one wrong column.** Within a group whose rows all fan out by the same factor:

| Aggregate | Under fan-out |
|---|---|
| `COUNT`, `SUM` | **inflated** by the fan-out factor |
| `AVG`, `MIN`, `MAX` | **unchanged** — uniform duplication preserves them |

So in the example above both averages are exactly right, and only the count is wrong. Three columns out of four corroborate a result that is 6× off, and nothing in the output signals that a fan-out happened. *Do not use "the averages look sensible" as a sanity check on a JOIN.* (If the factor varies across rows within a group — because you grouped by something coarser than the join key — then `AVG` is silently weighted too, and it is wrong as well.)

**What to do instead:**

- Count a key from **one** side rather than rows of the joined product: `COUNT(DISTINCT eu.event_id)`.
- Or aggregate **before** joining, so each side contributes one row per key — a materialized view per leg is the durable form of this.
- Sanity-check the row count against the left side alone before adding aggregates.

**`INNER JOIN` also drops rows.** A key absent from *any* joined table disappears from the result entirely — a tenant running in EU and US but not APAC vanishes from a query whose name says "every region". Use `LEFT JOIN` when the left side is the population you actually mean.

**In Federation specifically:** each leg is staged and joined coordinator-local, so a fan-out inflates the staged intermediate *and* consumes the joined-output row cap (`maxQueryResults`, Community 10,000 / Pro 1,000,000). Hitting that cap is reported — see [Row truncation at the result cap](#row-truncation-at-the-result-cap) — but a truncated fan-out is still an answer to a question you did not ask.

### ORDER BY … LIMIT (top-N)

Expand Down
45 changes: 38 additions & 7 deletions install.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ rem THIS process only, changes nothing on the machine, and needs no elevation.
rem
rem It is a wrapper and nothing else - every option, default, fallback and
rem message lives in install.ps1, so the two entry points can never drift.
rem Pass the same flags you would pass to install.ps1:
rem When install.ps1 is not sitting next to it, it downloads one, so
rem install.cmd on its own is a complete install. Pass the same flags you
rem would pass to install.ps1:
rem
rem install.cmd
rem install.cmd -ListVersions -EsVersion 8
Expand All @@ -28,14 +30,43 @@ rem ===========================================================================

setlocal

set "PS1_URL=https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/refs/heads/main/install.ps1"
set "PS1=%~dp0install.ps1"

if not exist "%PS1%" (
echo [ERROR] install.ps1 was not found next to install.cmd. 1>&2
echo [ERROR] Expected: %PS1% 1>&2
echo [ERROR] Download both files from the same release and keep them together. 1>&2
exit /b 1
)
rem A local install.ps1 always wins: a downloaded pair must stay self-consistent,
rem and a released bundle must never be silently mixed with main.
rem Written as goto + a separate errorlevel test rather than `if ... call ... ||`:
rem how cmd binds `||` inside an `if` body is ambiguous, and this is not a
rem platform where a subtlety can be settled by running it.
if exist "%PS1%" goto run
call :fetch_ps1
if errorlevel 1 exit /b 1

:run
powershell -NoProfile -ExecutionPolicy Bypass -File "%PS1%" %*
exit /b %ERRORLEVEL%

rem ---------------------------------------------------------------------------
rem Each line of a subroutine is parsed when it is reached, so %PS1% below sees
rem the value assigned on the previous line. The same code inside the `if not
rem exist (...)` block above would NOT: cmd expands every %VAR% in a
rem parenthesised block in ONE parse pass, before running any line in it.
rem ---------------------------------------------------------------------------
:fetch_ps1
where curl.exe >nul 2>&1
if errorlevel 1 (
echo [ERROR] install.ps1 is not next to install.cmd and curl.exe is unavailable. 1>&2
echo [ERROR] curl.exe ships with Windows 10 build 1803+ and Windows Server 2019+. 1>&2
echo [ERROR] Download install.ps1 manually into the same directory as install.cmd. 1>&2
exit /b 1
)
set "PS1=%TEMP%\softclient4es-install.ps1"
echo [INFO] install.ps1 not found next to install.cmd - downloading it...
echo [INFO] URL: %PS1_URL%
curl.exe -fsSL -o "%PS1%" "%PS1_URL%"
if errorlevel 1 (
echo [ERROR] Could not download install.ps1 from %PS1_URL% 1>&2
exit /b 1
)
echo [INFO] Using %PS1%
exit /b 0
Loading
Loading