Skip to content

Add canonical pipeline benchmark and optimize hot paths#2946

Merged
schani merged 1 commit into
masterfrom
agent/optimize-canonical-benchmarks
Jul 15, 2026
Merged

Add canonical pipeline benchmark and optimize hot paths#2946
schani merged 1 commit into
masterfrom
agent/optimize-canonical-benchmarks

Conversation

@schani

@schani schani commented Jul 14, 2026

Copy link
Copy Markdown
Member

Summary

  • add a whole-pipeline benchmark covering JSON parsing, schema processing, inference and transformations, TypeScript/Rust generation, and maximum observed JS heap
  • add a canonical real-world set: USGS earthquakes, GitHub REST OpenAPI, NVD CVE 2024, HL7 FHIR R5, and Kestra 0.19
  • expose per-pass timing through an optional onTiming callback
  • document cache/download behavior and benchmark usage
  • optimize the hot paths found by profiling the canonical suite
  • add focused unit coverage for timing, JSON/YAML parsing, naming dependencies and forbidden names, and accumulated string cases

Root causes and optimizations

The Rust renderer's naming pass repeatedly rebuilt identical forbidden-name unions and rescanned namespaces that were already complete. This was especially expensive because Rust checks its global keyword namespace for properties, enum cases, and union members.

Large inferred JSON inputs also accumulated millions of redundant empty attribute entries and allocated a StringTypes object/map/set per observed string. Large JSON schemas were being parsed through the YAML lexer even when they were strict JSON.

This change:

  • caches each namespace's forbidden-name union
  • tracks only unfinished naming namespaces
  • skips redundant empty attribute entries and batches string-case counts
  • tries JSON.parse before retaining YAML as a compatibility fallback

Canonical results

Node 24.6.0, macOS arm64, three measured runs and one warmup per case:

Case Before After Change Max heap before Max heap after
USGS / TypeScript 250.9 ms 189.5 ms -24.5% 146.5 MiB 130.1 MiB
USGS / Rust 257.2 ms 186.4 ms -27.5% 148.4 MiB 121.1 MiB
GitHub OpenAPI / TypeScript 1.614 s 1.176 s -27.1% 203.1 MiB 197.3 MiB
GitHub OpenAPI / Rust 6.557 s 1.209 s -81.6% 204.8 MiB 195.8 MiB
NVD / TypeScript 5.558 s 3.450 s -37.9% 2674.2 MiB 1825.2 MiB
NVD / Rust 5.402 s 3.464 s -35.9% 2673.7 MiB 1831.6 MiB
FHIR / TypeScript 2.238 s 1.519 s -32.1% 499.3 MiB 204.7 MiB
FHIR / Rust 4.872 s 1.715 s -64.8% 492.1 MiB 245.8 MiB
Kestra / TypeScript 6.178 s 4.850 s -21.5% 633.1 MiB 530.3 MiB
Kestra / Rust 6.538 s 4.931 s -24.6% 635.0 MiB 530.9 MiB

The sum of the ten medians is 42.5% lower. Every case is faster and every measured maximum heap value is lower.

All ten generated canonical outputs were compared against an unoptimized build and matched byte-for-byte by SHA-256.

Validation

  • npm run build --workspace quicktype-core
  • npm run test:unit — 20 files, 136 tests
  • npm run lint
  • QUICKTEST=true FIXTURE=typescript npm run test:fixtures — 70 tests
  • full canonical benchmark, three measured runs plus one warmup
  • byte-for-byte SHA-256 comparison for all five inputs in TypeScript and Rust

The external Rust fixture compiler was not run because cargo is unavailable in this environment; Rust generation is covered by unit tests, the canonical runs, and output identity checks.

@schani
schani marked this pull request as ready for review July 14, 2026 21:04
@schani
schani merged commit 662ad5b into master Jul 15, 2026
24 checks passed
@schani
schani deleted the agent/optimize-canonical-benchmarks branch July 15, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant