fix(test): remove two flaky tests that failed CI on main (v1.55.2) - #188
Merged
Conversation
Run 31512619737 on main failed 2 of 6 test jobs — node 22 ubuntu and node 20 windows — while node 22 windows and node 20 ubuntu passed. Two unrelated pre-existing flakes, neither introduced by #187. 1. calculateRelevance read Date.now() separately in each call, so two otherwise-identical inputs scored a millisecond apart differed in the recency term by ~1e-10 and failed an exact-equality assertion: expected: 0.7687777747569444 actual: 0.7687777748611111 Measured: 13 mismatches per 50,000 pairs (~0.03%) with an unpinned clock, 0 when both calls share one instant. RelevanceInputs now takes an optional `now`, and the function reads the clock once per call. Production behaviour is unchanged when it is omitted. This also lets a caller score a whole batch against a single instant, which is what any ranked list actually wants. 2. `writeFlowResultFile` teardown did rmSync without retries and hit ENOTEMPTY on windows, where a just-closed handle can outlive the close. Node's rmSync has maxRetries/retryDelay for exactly this; added to all 15 recursive teardowns rather than just the one that happened to fail, since every one of them is the same race. Neither flake was visible before today: CI only started running the suite this morning, and the enrich suites only stopped skipping once better-sqlite3 was bumped, so both are newly-visible rather than newly-broken.
paulkr
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the red CI on
main— run 31512619737.Diagnosis
2 of 6 test jobs failed, on unrelated OS/Node combinations:
That pattern is flakiness, not a break. Two separate pre-existing flakes, neither introduced by #187 — they're newly visible, not newly broken: CI only started running the suite this morning (#183), and the enrich suites only stopped skipping once better-sqlite3 was bumped (#184).
1.
calculateRelevance— a wall-clock raceThe test scores two inputs that differ only in
accessCount(one above the cap, one at it) and asserts they're equal. ButcalculateRelevancecalledDate.now()independently in each invocation, so the recency term differed whenever the two calls straddled a millisecond boundary.Measured it rather than assuming:
RelevanceInputsnow takes an optionalnow, and the function reads the clock once per call. Omitting it preserves current behaviour exactly, so nothing in production changes.Worth noting this isn't only a test fix: scoring a ranked list currently gives each record a marginally different clock. Pinning
nowlets a caller score a whole batch against one instant, which is what ranking actually wants.2.
writeFlowResultFile— a Windows teardown racermSyncwithout retries, against a directory whose file handle hadn't been released yet — routine on Windows. Node'srmSynchasmaxRetries/retryDelayfor precisely this case.Applied to all 15 recursive teardowns, not just the one that happened to fail. Every one of them is the same race; fixing only the observed instance would just move the flake. Several were already wrapped in
try/catch, which hid the problem rather than solving it — those now retry properly too.Verification
npx tsc --noEmitpasses; 467 tests, 0 skipped, all passing.