Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ coverage/
test/*.db
stress-test-results.json
stress-test-results.md
benchmark/results/

# Generated documentation is now in build/docs/ (covered by build/ ignore)

Expand Down
29 changes: 28 additions & 1 deletion .valgrind.supp
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,18 @@
obj:*/libcrypto.so*
}

# OpenSSL state linked into the Node executable on Node 26.
{
Node_OpenSSL_Builtin_Compressions
Memcheck:Leak
match-leak-kinds: definite
fun:malloc
fun:CRYPTO_malloc
fun:ossl_load_builtin_compressions
fun:context_init
fun:default_context_do_init_ossl_
}

# pthread thread-local storage
{
pthread_TLS
Expand All @@ -77,6 +89,21 @@
fun:pthread_*
}

# The main Node environment starts one inspector agent thread even when no
# inspector client connects. Suppress only that TLS allocation stack.
{
Node_Inspector_TLS
Memcheck:Leak
match-leak-kinds: possible
fun:calloc
fun:calloc
fun:allocate_dtv
fun:_dl_allocate_tls
fun:allocate_stack
fun:pthread_create*
fun:_ZN4node9inspector5Agent5Start*
}

# dlopen and dynamic loading
{
dlopen_Reachable
Expand Down Expand Up @@ -145,4 +172,4 @@
# Napi::Reference -- one of the most likely real defects in an addon of this
# shape. If Node's own internals prove noisy here, suppress the specific
# node:: frame, never the napi_ entry point.
# See the matching note in .lsan-suppressions.txt.
# See the matching note in .lsan-suppressions.txt.
29 changes: 28 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,38 @@ console.log(users); // [{ id: 1, name: 'Alice' }]
db.close();
```

## Experimental async pool

Server workloads that must keep SQLite execution off the event loop can use the
experimental fixed-size connection pool:

```typescript
import { DatabasePool } from "@photostructure/sqlite/experimental";

await using pool = await DatabasePool.open("app.db", {
connections: 2,
connectionSetup: [
{ sql: "PRAGMA journal_mode=WAL" },
{ sql: "PRAGMA busy_timeout=5000" },
],
});

const user = await pool.get("SELECT * FROM users WHERE id = ?", [1]);
```

The experimental entry point deliberately offers only connection-independent
`run`, `get`, `all`, and `batch` operations. Review its authorizer, setup,
ordering, memory, and libuv tradeoffs in the
[async pool guide](./doc/experimental-async-pool.md) before using it in
production.

## Features

- API-compatible with Node.js v26.7.0 built-in `node:sqlite` module\*
- Zero dependencies - native SQLite implementation
- Synchronous API - no async overhead
- Stable synchronous API with no async overhead on the root entry point
- Native SQLite performance ([benchmarks and tradeoffs](./benchmark/README.md))
- Experimental async connection pool for off-event-loop SQLite execution
- Full SQLite feature set ([details](./doc/features.md))
- TypeScript support with complete type definitions
- Cross-platform prebuilt binaries (Windows/macOS/Linux, x64/ARM64)
Expand Down Expand Up @@ -75,6 +101,7 @@ in your application, [review the results and run the benchmark](./benchmark/READ
- [Working with Data](./doc/working-with-data.md)
- [Extending SQLite](./doc/extending-sqlite.md)
- [Advanced Patterns](./doc/advanced-patterns.md)
- [Experimental Async Pool](./doc/experimental-async-pool.md)

**Reference**

Expand Down
116 changes: 116 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,122 @@ Transaction` and `DELETE Bulk` each write roughly 1,000 rows per operation.

</details>

## Experimental async pool

The async pool has a separate focused benchmark because its promise and
concurrency semantics do not fit the synchronous driver harness above. The
benchmark runs every case against a fresh, identically seeded database and
reports repeated raw samples plus a median and distribution-free relative
margin of error. It compares:

- a warm synchronous connection with a reused statement;
- a fresh synchronous connection per operation;
- a `worker_threads` control that owns a `DatabaseSync` and receives one message
per operation;
- `strict` and `none` pool authorizers with one, two, three, and four connections;
- individual concurrent calls and explicit batches of 10 and 100 operations;
- `all()` results containing 1, 100, and 1,000 rows;
- 100/0, 90/10, and 0/100 read/write mixes;
- repeated identical SQL text versus 32 equivalent rotating texts, exposing
prepare and cache effects;
- pool reads while representative PBKDF2 or filesystem reads compete for
libuv workers; and
- operations and materialized rows per millisecond, with event-loop heartbeat
delay as a diagnostic.

Run it from the repository root:

```bash
npm run bench:async
```

For a reproducible comparison, run inside the benchmark directory and record
both Node's default libuv pool and a larger process-startup pool. Keep the
commands and all CLI values the same between runs:

```bash
cd benchmark
npm install

unset UV_THREADPOOL_SIZE
npm run bench:async -- \
--iterations=10000 \
--write-iterations=2000 \
--connections=1,2,3,4 \
--samples=6 \
--warmup=1 \
--output=results/async-pool-default.json

UV_THREADPOOL_SIZE=8 npm run bench:async -- \
--iterations=10000 \
--write-iterations=2000 \
--connections=1,2,3,4 \
--samples=6 \
--warmup=1 \
--output=results/async-pool-uv8.json
```

The reference summary below used Node 26.6.0 on Linux x64 with an AMD Ryzen 9
5950X (32 logical CPUs). Each figure is the median of six measured samples after
one warmup. The run predates the current addition of a three-connection scaling
case and covers its recorded one/two/four-connection matrix. Raw benchmark
reports are local artifacts and are not versioned.

| Reference scenario | Median ops/ms |
| -------------------------------------------------------- | ------------------------: |
| Warm sync, reused statement | 244.7 |
| Fresh sync connection per operation | 4.1 |
| `worker_threads` sync control | 123.9 |
| Pool `none`, one connection | 37.9 |
| Pool `strict`, one connection | 39.9 |
| Pool `none`, four connections | 132.7 |
| Pool `none`, batches of 100 | 131.6 |
| Pool `all()` with 1,000 rows | 0.9 ops/ms; 883.0 rows/ms |
| Four-connection pool plus PBKDF2, default libuv pool | 11.3 |
| Four-connection pool plus PBKDF2, `UV_THREADPOOL_SIZE=8` | 123.6 |

These numbers meet the goal of multiple simple operations per millisecond and
also make the global-libuv tradeoff concrete: increasing the startup-time pool
size mostly mattered when four SQLite jobs competed with four crypto jobs. It
did not materially improve the uncontended four-connection point-read case.

The commands above write local reports under `benchmark/results/`, which Git
ignores. The JSON schema records the package and SQLite versions, git revision
and dirty state, Node/V8/N-API/libuv versions, CPU and platform, effective
`UV_THREADPOOL_SIZE`, all CLI options, each scenario's settings, every raw
sample, and the computed summaries. When sharing a comparison, attach the raw
files and exact commands outside the repository. Absolute throughput is
machine-specific, so compare repeated samples on the same otherwise-idle
machine rather than treating one run as a performance guarantee.

Runtime and scope are fully controllable. List scenario IDs and groups with
`npm run bench:async -- --list`, or see all options with
`npm run bench:async -- --help`. For example, this quick sanity run exercises one
control and the result-size group without writing a report:

```bash
npm run bench:async -- \
--scenarios=worker-thread-sync-control,result-size \
--iterations=10 \
--result-sizes=1,10 \
--seed-rows=10 \
--samples=2 \
--warmup=0
```

Idle SQLite connections consume no libuv worker, but each active pool operation
competes for Node's process-global worker threads with filesystem, DNS, crypto,
and zlib work. More busy connections than `UV_THREADPOOL_SIZE` do not create
more simultaneous SQLite execution. Set the environment variable before Node
starts, only after measuring the complete application; the library never
changes it.

The heartbeat is diagnostic benchmark evidence, not a functional timing test.
Deterministic tests separately prove that long SQLite work leaves the event loop
responsive. See the [experimental async pool guide](../doc/experimental-async-pool.md)
for operational limits, authorizer policy, setup replay, result memory, and
shutdown behavior.

## Run the benchmarks

```bash
Expand Down
Loading
Loading