feat: Download migrations and emulate bundles at runtime#203
feat: Download migrations and emulate bundles at runtime#203gjtorikian wants to merge 2 commits into
Conversation
Move the minimal ustar reader out of agent-sdk-assets.ts into npm-tarball.ts with a parameterized gzip-bomb cap, and share the spec tarball fixtures via __test-helpers__. Also export isCompiledBinary so the upcoming runtime bundle downloader can gate on the same detection. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generalize the Agent SDK first-run download into a runtime bundle mechanism (src/lib/runtime-assets.ts) so these packages can ship fixes independently of CLI releases. A generated manifest bakes each dep's semver range from package.json plus the tarball files to extract (the ESM bundle entrypoint and any __dirname-resolved sidecars, e.g. migrations' worker.js). At runtime the compiled binary resolves the newest non-deprecated version in range from the npm registry (24h disk-cached), downloads the tarball, verifies its SRI sha512 integrity before extracting, installs the files atomically under ~/.workos/cache/<name>/<version>/ (entrypoint last so its presence marks a complete install), and dynamic-imports the cached bundle. Every failure — resolution, download, verification, import, or a missing export — falls back first to the newest previously verified download and then to the compiled-in module, so the CLI is safe to ship before the packages publish dist/bundle.js. WORKOS_RUNTIME_DEPS=0 forces compiled-in with no network; =1 enables the mechanism from source.
Greptile SummaryThis PR decouples
Confidence Score: 3/5The download pipeline is safe to ship once the repeated-tarball-download issue is resolved; until then, every migrations, emulate, and dev invocation from a compiled binary will silently download a full npm tarball on startup. There is a concrete defect in the caching flow: writeResolutionCache is called after a successful metadata fetch but before a successful install. When the tarball exists on npm but does not yet contain dist/bundle.js — exactly the stated state of both companion packages today — the resolution is cached for 24 hours yet the entry path never appears on disk. Every subsequent process invocation will hit the cached resolution, find no entry path, and re-download the full tarball before failing and falling back to the compiled-in module. The failure is silent to users because logWarn only writes to the log file and to the console in debug mode. src/lib/runtime-assets.ts — specifically the ordering of writeResolutionCache relative to the downloadBundle call, and the pickHighestSatisfying candidate filter versus its docstring. Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (!resolved) { | ||
| try { | ||
| resolved = await fetchResolution(dep); | ||
| writeResolutionCache(cacheDir, resolved); | ||
| } catch (error) { | ||
| logWarn(`Version resolution for runtime dep ${dep.npmPackage} failed:`, error); | ||
| resolved = null; | ||
| } | ||
| } | ||
|
|
||
| if (resolved) { | ||
| try { | ||
| const bundlePath = entryPath(cacheDir, resolved.version, dep); | ||
| if (!existsSync(bundlePath)) { | ||
| await downloadBundle(dep, resolved, join(cacheDir, resolved.version)); | ||
| cleanupStaleVersions(cacheDir, resolved.version); | ||
| } | ||
| return await importBundle(bundlePath); | ||
| } catch (error) { | ||
| logWarn(`Runtime bundle install for ${dep.npmPackage}@${resolved.version} failed:`, error); | ||
| } | ||
| } |
There was a problem hiding this comment.
writeResolutionCache is called immediately after a successful metadata fetch, before the bundle install. When downloadBundle fails — for example because the currently-published tarball doesn't include dist/bundle.js (exactly the situation the PR describes for both @workos/migrations and @workos/emulate today) — the entrypoint path is never written to disk. On every subsequent CLI invocation within the 24-hour TTL window, readResolutionCache returns the cached version, existsSync(bundlePath) evaluates to false, and a full tarball download (up to TARBALL_TIMEOUT_MS = 30 s) is attempted again. Since logWarn is only visible in debug mode, users see no indication of what is happening. Every workos migrations, workos emulate, and workos dev invocation silently downloads the full npm tarball, fails extraction, then falls back to the compiled-in module until either the companion packages ship their bundles or the 24-hour cache expires.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/runtime-assets.ts
Line: 302-323
Comment:
**Repeated tarball download on every invocation during the transition period**
`writeResolutionCache` is called immediately after a successful metadata fetch, before the bundle install. When `downloadBundle` fails — for example because the currently-published tarball doesn't include `dist/bundle.js` (exactly the situation the PR describes for both `@workos/migrations` and `@workos/emulate` today) — the entrypoint path is never written to disk. On every subsequent CLI invocation within the 24-hour TTL window, `readResolutionCache` returns the cached version, `existsSync(bundlePath)` evaluates to false, and a full tarball download (up to `TARBALL_TIMEOUT_MS = 30 s`) is attempted again. Since `logWarn` is only visible in debug mode, users see no indication of what is happening. Every `workos migrations`, `workos emulate`, and `workos dev` invocation silently downloads the full npm tarball, fails extraction, then falls back to the compiled-in module until either the companion packages ship their bundles or the 24-hour cache expires.
How can I resolve this? If you propose a fix, please make it concise.| export function pickHighestSatisfying(metadata: AbbreviatedPackument, range: string): ResolvedVersion | null { | ||
| const versions = metadata.versions ?? {}; | ||
| const candidates = Object.keys(versions).filter((version) => { | ||
| const info = versions[version]; | ||
| return ( | ||
| valid(version) !== null && | ||
| !info?.deprecated && | ||
| typeof info?.dist?.tarball === 'string' && | ||
| typeof info?.dist?.integrity === 'string' | ||
| ); | ||
| }); | ||
| const version = maxSatisfying(candidates, range); | ||
| if (!version) return null; | ||
| const dist = versions[version]?.dist as { tarball: string; integrity: string }; | ||
| return { version, tarballUrl: dist.tarball, integrity: dist.integrity }; | ||
| } |
There was a problem hiding this comment.
pickHighestSatisfying docstring overstates sha512 pre-filtering
The JSDoc says it returns null for "versions missing a tarball URL or sha512 integrity" but the actual filter only requires typeof info?.dist?.integrity === 'string'. A version carrying only a sha1- or sha256- integrity string passes the candidate filter, gets written to resolution.json, triggers a tarball download on each invocation, then throws inside verifySriIntegrity (No sha512 hash…). The security contract is still upheld (verification always runs), but the candidate filter is looser than advertised, and such a version being cached in resolution.json feeds directly into the repeated-download issue described above. The filter should also check for the sha512- prefix, or the docstring should be corrected to accurately describe what is checked.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/runtime-assets.ts
Line: 84-99
Comment:
**`pickHighestSatisfying` docstring overstates sha512 pre-filtering**
The JSDoc says it returns null for "versions missing a tarball URL or sha512 integrity" but the actual filter only requires `typeof info?.dist?.integrity === 'string'`. A version carrying only a `sha1-` or `sha256-` integrity string passes the candidate filter, gets written to `resolution.json`, triggers a tarball download on each invocation, then throws inside `verifySriIntegrity` (`No sha512 hash…`). The security contract is still upheld (verification always runs), but the candidate filter is looser than advertised, and such a version being cached in `resolution.json` feeds directly into the repeated-download issue described above. The filter should also check for the `sha512-` prefix, or the docstring should be corrected to accurately describe what is checked.
How can I resolve this? If you propose a fix, please make it concise.1 participant
Context
The CLI ships as a Bun-compiled standalone binary (#195), with
@workos/migrationsand@workos/emulatecompiled in at build time. That couples their release cycles to ours: users only receive a new migrations or emulate version when a new CLI is released, even though those packages can move faster than the CLI.This PR is the CLI half of a cross-repo effort to decouple them. Companion branches in
workos-migrationsandemulatemake each package publish a self-contained single-file ESM bundle inside its normal npm tarball (exports subpath./bundle, tarball pathpackage/dist/bundle.js). This PR teaches the CLI to fetch and run those bundles at runtime — so amigrationsfix reaches users on their next invocation instead of waiting for a CLI release.How it works
scripts/gen-runtime-deps-manifest.ts, wired intobun run generate) bakes each dep's semver range frompackage.jsonat build time — ranges can't drift from what we compile against — plus the list of tarball files to extract (migrations ships aworker.jssidecar resolved via__dirname, so files are a list, entrypoint first).registry.npmjs.org(abbreviated metadata, 3s timeout) for the newest non-deprecated version satisfying the range; the answer is disk-cached for 24h.~/.workos/cache/<name>/<version>/— entrypoint written last, so its presence marks a complete install. Nothing that fails verification is ever imported.await import()ed and shape-checked (programfor migrations,createEmulatorfor emulate) by the calling commands.Fallback chain — every failure point (offline, timeout, no in-range version, bad integrity, failed import, missing export) falls back to the newest previously verified download, then to the compiled-in module. The packages haven't published bundles yet, so today this PR is behavior-neutral: every path lands on the compiled-in code, which is exactly what ships now.
Escape hatches —
WORKOS_RUNTIME_DEPS=0forces compiled-in with zero network. From source (dev/tests), the mechanism is off unlessWORKOS_RUNTIME_DEPS=1, keeping development and CI hermetic; it activates only in compiled binaries, mirroring the Agent SDK download.