A backreference does not match characters which are matched case-insensitively as literals

[serhiy-storchaka]

import re

print(re.fullmatch('(?i)ς', 'σ'))
print(re.fullmatch(r'(?i)(.)\1', 'ςσ'))
print(re.fullmatch('(?i)Σ', 'σ'))
print(re.fullmatch(r'(?i)(.)\1', 'Σσ'))

<re.Match object; span=(0, 1), match='σ'>
None
<re.Match object; span=(0, 1), match='σ'>
<re.Match object; span=(0, 2), match='Σσ'>

'ς' (GREEK SMALL LETTER FINAL SIGMA) matches 'σ' (GREEK SMALL LETTER SIGMA) when it is a literal in the pattern, but not when it is matched by a backreferenced group. 'Σ' (GREEK CAPITAL LETTER SIGMA) matches it in both cases.

This is because a literal is expanded at compile time into the alternatives listed in _casefix._EXTRA_CASES, which groups the characters having the same uppercase, so (?i)ς is compiled to a set containing both sigmas. A backreference has no compiled set to expand, and GROUPREF_UNI_IGNORE compares sre_lower_unicode() of the two characters, which lowercases 'Σ' to 'σ' but leaves 'ς' unchanged.

28 pairs are affected, among them 'µ' (MICRO SIGN) and 'μ' (GREEK SMALL LETTER MU), 'ſ' (LATIN SMALL LETTER LONG S) and 's' (LATIN SMALL LETTER S), the Greek symbol variants, and the Cyrillic historic letters added in Unicode 9.0.

sre_lower_unicode() can return a key which is the same for all characters matched case-insensitively, and then a backreference matches whatever a literal matches. This takes three steps:

• Compare the simple case folding instead of the lowercase. This unifies most of the pairs, 'µ' with 'μ' among them.

• Lowercase the result. A character whose full case folding is longer than one character is not unified with its case partners by the folding -- both SHARP S characters are folded to "ss", and every letter with ypogegrammeni to two characters -- so such a character is lowercased instead, which is what keeps it with them, since they share the lowercase. The folding is lowercased in turn, since it is not always downwards: Cherokee letters are folded to their uppercase.

• Hardcode the four pairs which no case mapping unifies: LATIN SMALL LETTER DOTLESS I ('i' and 'ı'), the two pairs of Greek letters with tonos and with oxia ('ΐ' and 'ΐ', 'ΰ' and 'ΰ'), and the two ST ligatures ('ſt' and 'st'). All eight characters are Unicode 1.1, and nothing added since has joined them.

_casefix._EXTRA_CASES is then unused and can be removed, together with the alternatives which the compiler expanded a literal into. Its generator keeps computing the groups of characters which have to be matched case-insensitively, and fails if sre_lower_unicode() does not fold such a group to a single code.

Linked PRs

• gh-156513: Match a backreference case-insensitively as a literal #156514