Skip to content

gh-156505: Speed up difflib.HtmlDiff for lopsided replacements - #156506

Open
felix314159 wants to merge 1 commit into
python:mainfrom
felix314159:main
Open

gh-156505: Speed up difflib.HtmlDiff for lopsided replacements#156506
felix314159 wants to merge 1 commit into
python:mainfrom
felix314159:main

Conversation

@felix314159

@felix314159 felix314159 commented Aug 28, 2026

Copy link
Copy Markdown

difflib._mdiff() used lists as FIFO queues for unmatched lines and removed
items with pop(0). This repeatedly shifted the remaining items, causing
quadratic behavior for lopsided replacement blocks.

Use collections.deque and popleft() for constant-time FIFO removal. This
preserves the existing output while making the affected line-pairing step scale
approximately linearly.

Benchmarks on matched release builds from CPython main:

Case Before After Speedup
_mdiff(), 64,000-to-1 264 ms 61.9 ms 4.26x
HtmlDiff.make_table(), 64,000-to-1 398 ms 206 ms 1.93x
_mdiff(), 128,000-to-1 999 ms 126 ms 7.94x

The complete test_difflib suite passes. A focused regression test covers
lopsided replacements in both directions, and randomized differential testing
confirmed identical output for _mdiff() and HtmlDiff across context and
wrapping modes.

Fixes #156505

@bedevere-app

bedevere-app Bot commented Aug 28, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@python-cla-bot

python-cla-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@felix314159

Copy link
Copy Markdown
Author

I think the skip news label makes sense here as behavior is unchanged

@eendebakpt

Copy link
Copy Markdown
Contributor

Do you have benchmarks for more mixed scenario's? In particular cases where the changes could cause a regression?

@felix314159

Copy link
Copy Markdown
Author

I benchmarked the parent commit against this PR using 15 paired randomized samples with the same non-debug interpreter. So basically I tested identical inputs, sparse replacements, balanced replacements, and mixed insertion/deletion/replacement workloads. The larger mixed cases ranged from unchanged to like 2% faster. A four-line _mdiff case measured about 1.45% slower, but remember this is only around 0.3 µs per call. The corresponding HtmlDiff case differed by only 0.15%. So IMO there is no meaningful regression outside the lopsided case this PR fixes.

@StanFromIreland

Copy link
Copy Markdown
Member

CC @dg-pb @tim-one

@dg-pb dg-pb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sensible improvement. No new imports. Change makes sense in straight forward manner. No risk for degradation for any cases. Not much to add really.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

difflib.HtmlDiff has quadratic line pairing for lopsided replacements

4 participants