fix(normalizations): exclude -1 padding from token-norm length + guard mismatch (#1170)

[WatchTree-19]

problem

LogProbTokenNorm in src/lighteval/metrics/normalizations.py crashes with IndexError for belebele CFFormulation (#1170), and there's a related correctness bug @Rijgersberg flagged in the thread. two issues:

1. padding is counted in the token length. continuations are right-padded with -1 (pad_sequence(..., padding_value=-1) in transformers_model.py), and those -1 pads end up in choices_tokens. len(choices_tokens[ix]) then counts padding, so the per-choice length used to normalize the log-prob is inflated - e.g. the reporter's [236743, 236812, -1] is treated as length 3 instead of 2. this silently skews token-normalized scores whenever continuations differ in length, not only in the crashing case.

2. opaque IndexError on a length mismatch. the comprehension indexes choices_tokens[ix] over range(len(choices_logprob)) with no length check - unlike the LogProbCharNorm case just above, which validates and raises a clear ValueError. when the backend returns fewer output_tokens than choices (reporter: 3 token-lists for 4 choices/logprobs), you get an IndexError with no context.

fix

• exclude -1 padding from the per-choice token count via a small _num_continuation_tokens helper.
• validate len(choices_tokens) == len(choices_logprob) and raise a clear ValueError, matching the existing LogProbCharNorm guard.

tests

• test_token_norm_excludes_padding - fails on main (the -1 inflates the length and skews the value), passes here.
• test_token_norm_length_mismatch_raises - was IndexError, now a clear ValueError.
• existing test_token_norm / test_empty_input are unaffected.

remaining root cause (not fixed here, flagged for maintainers)

the reason belebele CFFormulation hits the mismatch is upstream: the model response's output_tokens comes back with fewer entries than logprobs/choices (reporter: 3 vs 4). that lives in the loglikelihood backend (_loglikelihood_tokens, where continuations are padded across both the length and the choice dimension) and needs a repro on the actual model to fix correctly. this PR makes the token count correct and turns the crash into an actionable error; happy to follow up on the backend alignment with a pointer if that's useful.

[ErenAta16]

Checked the padding-exclusion logic and the new length guard against the actual pad_sequence(..., padding_value=-1) usage in transformers_model.py — _num_continuation_tokens correctly counts only non-negative tokens, and the real or len(tokens) or 1 fallback chain avoids a divide-by-zero for the degenerate all-padding/empty case without changing behavior for any normal input. The two new tests reproduce both parts of #1170 (the padding-inflation case and the crash case) directly.

While comparing this to the sibling match arms in normalize_log_probs, I noticed the same missing-length-validation pattern this PR fixes for LogProbTokenNorm also exists in two other arms that aren't touched here:

• LogProbCharNorm(ignore_first_space=False) doesn't crash on a mismatch, but silently truncates: [choices_logprob[ix] / len(choice) for ix, choice in enumerate(choices_text)] iterates over choices_text, so if it's shorter than choices_logprob you just get a shorter result list back with no error, e.g. 3 logprobs + 2 texts silently returns 2 normalized values instead of 3.
• LogProbPMINorm() has the same shape as the old LogProbTokenNorm code: choices_logprob[ix] - unconditioned_logprob[ix] for ix in range(len(choices_logprob)) raises a bare IndexError if unconditioned_logprob is shorter.

Confirmed both concretely by replicating the exact comprehensions. Not asking to fix those here since they're outside the scope of #1170, but might be worth a quick follow-up PR applying the same guard-and-raise pattern to keep all four match arms consistent, since LogProbCharNorm(ignore_first_space=True) already has the same check this PR adds for LogProbTokenNorm.

This PR itself looks correct and ready.

[ErenAta16]

#1180 and #1290 are both open on the LogProbTokenNorm arm of normalize_log_probs, one since March, so here is the two of them side by side rather than in separate threads. @inakiLakunza @WatchTree-19 tagging you both.

Ran the three interesting inputs through main and through each PR's logic:

case                                  main                     #1180                    #1290
padded [1,2,-1] vs [4,5,6]            [3.3333, 6.6667]         [3.3333, 6.6667]         [5.0, 6.6667]
tokens shorter than logprobs          IndexError               [0.5, 1.0, 1.5]  SHORTER ValueError
zero-token choice [[]]                ZeroDivisionError        ZeroDivisionError        [-1.0]

Row 1 is the part I would not want to lose

transformers_model.py right-pads the continuations before they become output_tokens:

lighteval/src/lighteval/models/transformers/transformers_model.py

Line 1037 in 64f4f5a

pad_sequence(doc_continuation_tokens, batch_first=True, padding_value=-1)

and that padded tensor goes straight into ModelResponse.output_tokens at L1137, so len(choices_tokens[ix]) counts padding. Every choice in a document ends up divided by the length of the longest choice, which is the same divisor for all of them, so token normalization silently degenerates into no normalization at all whenever the choices differ in length. That is a wrong-score bug, not a crash, and it is only in #1290.

#1180 does not touch it. Its min(...) cap changes how many results come back but not what any of them is, so a run that does not crash keeps the inflated denominators.

Row 2 is where I would push back on #1180

Capping the loop with min(len(choices_logprob), len(choices_tokens)) turns a loud IndexError into a shorter return list. LoglikelihoodAcc then does:

best_choice = np.argmax(normalized_log_probs)
return int(best_choice in gold_ixs)

so the argmax runs over a truncated set of choices and a gold answer in one of the dropped positions scores 0. The logger.warning helps, but in a long eval run that is one line in a stream of progress output, and the failure mode it leaves behind is "accuracy is a bit lower than expected", which nobody investigates.

The stated trigger is a stale cache where a task's choices changed between runs. If that is the case, silently scoring against a stale subset seems worse than stopping and telling the user to clear the cache, which is what #1290's ValueError does. Its message even names the likely cause.

This is the same shape as the LogProbCharNorm arm right above, which silently truncates for one of its two settings today. I have a separate PR at #1317 for that one, and it goes the other way, raising rather than truncating, on the same reasoning. Worth the three landing consistently.

Two things on #1290

The -1 sentinel is backend-specific. _num_continuation_tokens filters on t >= 0, which is right for transformers_model.py since that is the only place using padding_value=-1 and real token ids are never negative. But vllm_model.py, sglang_model.py and endpoint_model.py all build output_tokens themselves without padding, so for those backends the filter is a no-op and the fix does not apply. That is fine, they do not have the bug, but a one-line comment saying the filter is specific to the padded path would stop someone later "generalising" it.

The fallback can produce a number where there is no answer. real or len(tokens) or 1 means an all-padding choice divides by the padding count, and an empty [] divides by 1. Both avoid the crash, which is the goal, but they also turn "this choice has no continuation tokens" into a plausible-looking normalized log prob that flows into the argmax. Given the mismatch case above is being made loud, it seems inconsistent to make this one quiet. Raising here, or at least logging which choice it was, would match.

Suggestion

#1290 is the one that fixes a score, and it fails loudly on the input #1180 is aimed at, so it covers #1180's case as well. My read is that #1290 is the one to take, with the fallback question above resolved and a note that the -1 filter is for the transformers path.

@inakiLakunza your diagnosis of the stale-cache trigger is worth keeping in the thread either way; #1290's error message would be more useful if it mentioned that as a cause, since right now it only suggests the backend returned too few token lists.