Skip to content

fix(test): remove two flaky tests that failed CI on main (v1.55.2) - #188

Merged
paulkr merged 1 commit into
mainfrom
fix/flaky-ci-tests
Aug 11, 2026
Merged

fix(test): remove two flaky tests that failed CI on main (v1.55.2)#188
paulkr merged 1 commit into
mainfrom
fix/flaky-ci-tests

Conversation

@siddharth-bhansali

Copy link
Copy Markdown
Collaborator

Fixes the red CI on mainrun 31512619737.

Diagnosis

2 of 6 test jobs failed, on unrelated OS/Node combinations:

✗ test (node 22, ubuntu-latest)      ✓ test (node 22, windows-latest)
✗ test (node 20, windows-latest)     ✓ test (node 20, ubuntu-latest)
✓ test (node 18, ubuntu-latest)      ✓ test (node 18, windows-latest)

That pattern is flakiness, not a break. Two separate pre-existing flakes, neither introduced by #187 — they're newly visible, not newly broken: CI only started running the suite this morning (#183), and the enrich suites only stopped skipping once better-sqlite3 was bumped (#184).

1. calculateRelevance — a wall-clock race

expected: 0.7687777747569444
actual:   0.7687777748611111

The test scores two inputs that differ only in accessCount (one above the cap, one at it) and asserts they're equal. But calculateRelevance called Date.now() independently in each invocation, so the recency term differed whenever the two calls straddled a millisecond boundary.

Measured it rather than assuming:

unpinned clock: 13 mismatches / 50,000 pairs   (~0.03%)
pinned clock:    0 mismatches / 50,000 pairs

RelevanceInputs now takes an optional now, and the function reads the clock once per call. Omitting it preserves current behaviour exactly, so nothing in production changes.

Worth noting this isn't only a test fix: scoring a ranked list currently gives each record a marginally different clock. Pinning now lets a caller score a whole batch against one instant, which is what ranking actually wants.

2. writeFlowResultFile — a Windows teardown race

ENOTEMPTY: directory not empty, rmdir 'C:\...\Temp\one-flow-out-UDo0tZ'

rmSync without retries, against a directory whose file handle hadn't been released yet — routine on Windows. Node's rmSync has maxRetries/retryDelay for precisely this case.

Applied to all 15 recursive teardowns, not just the one that happened to fail. Every one of them is the same race; fixing only the observed instance would just move the flake. Several were already wrapped in try/catch, which hid the problem rather than solving it — those now retry properly too.

Verification

  • npx tsc --noEmit passes; 467 tests, 0 skipped, all passing.
  • Ran the two affected suites 5× consecutively — stable. (Though for flake 1 the real proof is the measurement above, not repeat runs: at 0.03% you'd expect ~1 failure per 3,000 runs, so green locally means little. The pinned clock makes it deterministic by construction.)

Run 31512619737 on main failed 2 of 6 test jobs — node 22 ubuntu and node 20
windows — while node 22 windows and node 20 ubuntu passed. Two unrelated
pre-existing flakes, neither introduced by #187.

1. calculateRelevance read Date.now() separately in each call, so two
   otherwise-identical inputs scored a millisecond apart differed in the
   recency term by ~1e-10 and failed an exact-equality assertion:

     expected: 0.7687777747569444
     actual:   0.7687777748611111

   Measured: 13 mismatches per 50,000 pairs (~0.03%) with an unpinned clock,
   0 when both calls share one instant.

   RelevanceInputs now takes an optional `now`, and the function reads the
   clock once per call. Production behaviour is unchanged when it is omitted.
   This also lets a caller score a whole batch against a single instant, which
   is what any ranked list actually wants.

2. `writeFlowResultFile` teardown did rmSync without retries and hit
   ENOTEMPTY on windows, where a just-closed handle can outlive the close.
   Node's rmSync has maxRetries/retryDelay for exactly this; added to all 15
   recursive teardowns rather than just the one that happened to fail, since
   every one of them is the same race.

Neither flake was visible before today: CI only started running the suite this
morning, and the enrich suites only stopped skipping once better-sqlite3
was bumped, so both are newly-visible rather than newly-broken.
@paulkr
paulkr merged commit 8ab2cd4 into main Aug 11, 2026
7 checks passed
@paulkr
paulkr deleted the fix/flaky-ci-tests branch August 11, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants