difflib._mdiff() buffers unmatched lines in lists and consumes them using
pop(0). For lopsided replacement blocks, each removal shifts the remaining
items, making line pairing quadratic.
Reproducer:
import difflib
a = ["a\n"] * 64_000
b = ["b\n"]
list(difflib._mdiff(a, b))
Using matched release builds from CPython main:
| Case |
Current |
With deque.popleft() |
_mdiff(), 64,000-to-1 |
264 ms |
61.9 ms |
HtmlDiff.make_table(), 64,000-to-1 |
398 ms |
206 ms |
_mdiff(), 128,000-to-1 |
999 ms |
126 ms |
The speedup increases with input size, reaching 7.94x at 128,000 lines. Patched
runtime scales approximately linearly.
Using collections.deque for the two pairing queues preserves FIFO behavior
while making removal constant-time. Focused tests, the complete test_difflib
suite, and randomized differential testing confirm identical output.
Tested on CPython main at ed1aa1f324f9 on Linux.
Linked PRs
difflib._mdiff()buffers unmatched lines in lists and consumes them usingpop(0). For lopsided replacement blocks, each removal shifts the remainingitems, making line pairing quadratic.
Reproducer:
Using matched release builds from CPython main:
deque.popleft()_mdiff(), 64,000-to-1HtmlDiff.make_table(), 64,000-to-1_mdiff(), 128,000-to-1The speedup increases with input size, reaching 7.94x at 128,000 lines. Patched
runtime scales approximately linearly.
Using
collections.dequefor the two pairing queues preserves FIFO behaviorwhile making removal constant-time. Focused tests, the complete
test_difflibsuite, and randomized differential testing confirm identical output.
Tested on CPython main at
ed1aa1f324f9on Linux.Linked PRs