Optimize `unicode_eq` for short string comparisons by bypassing `memcmp` overhead

[rajat315315]

Feature or enhancement

Proposal:

Description

In CPython, string equality comparisons (unicode_eq inside Objects/stringlib/eq.h) are heavily used during dictionary lookups and set membership operations to resolve hash collisions. Currently, unicode_eq delegates the raw byte comparison to the standard C library's memcmp function:

return (memcmp(data1, data2, len * kind) == 0);

While glibc's memcmp is heavily optimized, it introduces substantial overhead for very short strings (which dominate Python dictionary keys and attribute names, typically 2 to 12 bytes):

1. Function Call Overhead: Invoking memcmp requires passing parameters on registers, pushing stack frames, and jumping (often traversing the Procedure Linkage Table in dynamic builds).
2. Branching & Alignment checks: The standard library must dynamically evaluate address alignment and length conditions to handle any size from 1 byte to megabytes.

Proposed Optimization

Introduce a fast inline helper unicode_memeq that bypasses memcmp for short sizes by leveraging inline integer comparisons and overlapping loads:

• Sizes 1, 2, 4, 8 bytes: Direct cast comparison using machine integer types (e.g., uint16_t or uint64_t), compiling to a single CPU register check.
• Sizes 9 to 16 bytes: Overlapping 8-byte loads (comparing the first and last 8 bytes of the string). This performs exactly two integer comparisons and avoids any branching loops.
• Sizes 17 to 32 bytes: Overlapping 16-byte SSE2 vector loads (using _mm_loadu_si128 and vector comparisons), avoiding loop checks.
• Sizes > 32 bytes: Fall back to standard memcmp.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-153563 feat: add unicode_memeq() to compare either directly or SSE2 vector #153564