Skip to content

Don't re-derive token positions callers already know - #881

Open
alexander-beedie wants to merge 1 commit into
andialbrecht:masterfrom
alexander-beedie:perf/quadratic-rescans
Open

Don't re-derive token positions callers already know#881
alexander-beedie wants to merge 1 commit into
andialbrecht:masterfrom
alexander-beedie:perf/quadratic-rescans

Conversation

@alexander-beedie

Copy link
Copy Markdown
Contributor

Four places recomputed a token's position by scanning from the start of its parent, each on a path that runs once per group, resulting in potential quadratic slowdown at sufficient scale.

  • ReindentFilter._current_line_len()
    (O(position) scan plus an O(position) copy, once per group)

  • group_where()
    (re-derived an end index it had already been given by scanning from the statement head, once per WHERE clause)

  • group_comments()
    (re-scanned an all-comment tail once per comment token)

  • token_index()
    (sliced the token list before searching it, so the slice was a pure copy)

Tweaked each of these to avoid the potential blowup 👍

Speedup

Per vector, at its largest size:

vector before (ms) after (ms) speedup
VALUES list, no guards (62.9 kB) 2690 426 6.32x
VALUES list, capped (14.5 kB) 223 97 2.30x
UNION ALL chain (80.7 kB) 313 238 1.32x

@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.20%. Comparing base (111b35c) to head (4f6005f).
⚠️ Report is 25 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #881      +/-   ##
==========================================
+ Coverage   97.13%   97.20%   +0.07%     
==========================================
  Files          31       31              
  Lines        3663     3797     +134     
  Branches      328      346      +18     
==========================================
+ Hits         3558     3691     +133     
- Misses         63       65       +2     
+ Partials       42       41       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant