[Vulkan][Adreno] fp32 nn.Linear returns wrong results on ~23% of executions at large M

[msluszniak]

🐛 Describe the bug

A single nn.Linear(384, 384) lowered to Vulkan returns a wrong result on roughly a quarter of executions on an Adreno 840, from the same loaded program and the same input, with no error reported.

import torch, torch.nn as nn
class M(nn.Module):
    def __init__(self):
        super().__init__(); self.f = nn.Linear(384, 384)
    def forward(self, x): return self.f(x)
# lower with VulkanPartitioner, run on device with input (1, 1500, 384)

100 executions in one process, dumping every execution, on a Galaxy S26 Ultra (Adreno 840):

distinct outputs:                    24 / 100
executions wrong vs CPU by >1e-3:    23 / 100
max |deviation| vs CPU reference:    9.08
signal absmax:                       2.88
max spread across executions:        12.56

77 executions produce the identical, correct result. The other 23 are each uniquely wrong, and the error is more than 3x the magnitude of the signal itself, so this is not accumulation-order noise.

The corruption is tile shaped

The differing elements are confined to a contiguous band of rows aligned to 32, not spread across the output:

M	elements differing	rows affected	columns affected
1024	4089 / 393216	128 (rows 224 to 447)	32
1500	10447 / 576000	160 (rows 128 to 415)	128

That looks like specific workgroup tiles producing garbage rather than a numerical issue.

It is probabilistic, and the rate grows with M

nn.Linear(384, 384) over (1, M, 384), counting distinct outputs:

M	distinct	replays	max spread
128	1	20	0
254	1	200	0
384	1	20	0
512	1	20	0
768	1	20	0
800	2	40	9.07
832	1	40	0
896	3	40	9.08
960	2	40	2.82
992	3	40	9.07
1008	2	40	0.25
1024	2	20	8.98
1500	24	100	12.56

There is no clean cutoff. M = 832 is clean over 40 replays while 800 and 896 are not, which is consistent with a race whose probability rises with the dispatch size rather than a hard threshold. The small-M rows are only evidence of a low rate, not of safety, although M = 254 held over 200 replays.

Impact

This is what makes the Whisper encoder unusable on this GPU. Its n_ctx is 1500, so every q/k/v/out projection runs at M = 1500. Bisecting the encoder by exporting progressive prefixes:

stage	distinct / 20	max spread	signal absmax
conv1, conv2, +positional	1	0	5.98 to 8.49
after transformer block 0	9	31.1	5.41
after block 1	14	24.6	6.50
after block 2	18	40.3	36.84
after block 3	20	65.4	73.40

The two Conv1d layers are perfectly stable. Splitting block 0 further, the first unstable part is the q/k/v projection itself, before the attention matmuls.

Not reproducible on Mali

The same Whisper encoder program on a Mali-G76 (Galaxy S10+) is deterministic: 1 distinct output over 10 executions, correct to cosine 0.99999702. So this looks specific to the Adreno path.

This may share a root cause with #21938, which reports nondeterminism in the conv2d im2col GEMM. The failure here is in the plain fp32 linear path rather than a convolution, and the tiled fp32 linear shaders contain no barrier(), so if they are related the cause is below the shader level.

Versions

• ExecuTorch main (c27baa8031) plus my [ET-VK] Clamp the tanh argument in the gelu shader #22324 and [ET-VK] Do not return early before the barrier in the reduce shader #22326
• Device: Samsung Galaxy S26 Ultra (SM-S948B), Snapdragon SM8850, Adreno 840
• Compared against: Samsung Galaxy S10+ (SM-G975F), Mali-G76, where it does not reproduce

I have the repro harness and can run instrumented builds on both devices if that would help.

cc @SS-JIA @manuelcandales @digantdesai @cbilgin

---

Update: localised to the local work group shape, with a cheap workaround

The corrupted region is whole work groups

For nn.Linear(384, 384) over (1, 1500, 384), pick_linear_gwg dispatches {div_up_4(N), div_up(M, tile_m), B} = {96, 375, 1} with tile_m = 4, and pick_xy_square_lwg gives an 8 x 8 x 1 local group. Each invocation writes rows [4y, 4y+3] and columns [4x, 4x+3], so one work group covers 32 rows by 32 columns.

Mapping one corrupted execution onto that grid:

corrupted rows	five runs of exactly 32: 160-191, 224-255, 288-319, 352-383, 416-447
in work group y	5, 7, 9, 11, 13, that is every other one
corrupted columns	exactly 128-255, work group x 4-7
everything else	exact to 5e-7

So 20 entire work groups returned garbage and every other work group was perfect. The wrong values are finite and of plausible magnitude, not zeros or stale buffers, and they exceed the correct range (the output spans +/-2.88, the corrupted values reach -9.58).

This shader has no shared memory and no barrier(), and every invocation is independent, so a whole work group being wrong means its reads returned bad data rather than any cross-invocation interaction.

It is not the readback

Two extra probes at the same shape, 60 executions each:

graph	distinct	wrong
linear alone	5 / 60	4 / 60
relu(linear(x)) - 0.5, result consumed on device	7 / 60	6 / 60

The corruption survives having the result consumed on the GPU, so it is not an artifact of copying the output back.

The local work group shape decides it

Holding everything else fixed and forcing the local group for this dispatch, 60 executions each, same input:

local work group	distinct	wrong	max spread
8 x 8 x 1 (current default)	8	7	9.08
16 x 4 x 1	1	0	0
32 x 2 x 1	1	0	0
64 x 1 x 1	1	0	0
4 x 16 x 1	11	10	9.38
2 x 32 x 1	1	0	0
1 x 64 x 1	1	0	0

Only 8 x 8 and 4 x 16 fail. Every other shape of the same 64 invocations is bit-stable and matches the CPU reference exactly. Note 2 x 32 is clean while 4 x 16 is not, and 16 x 4 is clean while 8 x 8 is not, so this is not simply "2D is bad".

Since the shader is plain data-parallel code with no inter-invocation communication, a result that depends on the local group shape is a driver defect rather than a shader bug. But the choice of shape is ours, and it is what decides whether the output is correct.

Effect on the Whisper encoder

Whisper-tiny encoder on the same device, 40 executions:

local work group	distinct	cosine vs CPU	median time
8 x 8 x 1	18-20 / 20	0.72 to 0.999989	150.2 ms
16 x 4 x 1	1 / 40	0.99999696	152.6 ms, +1.5%
32 x 2 x 1	1	correct	158.4 ms, +5.4%
64 x 1 x 1	1 / 20	0.99999696	200.9 ms, +33.7%
2 x 32 x 1	1	correct	212.6 ms, +41.5%

16 x 4 x 1 makes the encoder fully deterministic and correct for 1.5% more time. That looks like a viable mitigation until the underlying driver issue is understood, though I have only measured it on this one GPU and would not want to change the default for other vendors without numbers from them.

Environment

Same as above: ExecuTorch main (c27baa8031), Galaxy S26 Ultra (SM-S948B), Adreno 840. Not reproducible on Mali-G76.

I am happy to turn 16 x 4 x 1 into a PR, gated on Adreno or applied generally, whichever the maintainers prefer. I can also run any instrumented build on both devices.

---

Update: this is not limited to linear

PR #22328 initially patched only Linear.cpp, and the Whisper encoder looked clean at 40 executions. It was not clean, and the scope was wrong.

Once the conv frontend was moved onto the conv2d im2col + GEMM path (#22329 / #22330), conv2d_gemm reproduced the same defect on its own, with Linear.cpp already fixed. Distinct outputs over 60 executions on the Adreno 840:

model	distinct / 60
Whisper encoder, conv1d frontend, Linear.cpp fixed	1
Whisper encoder, conv2d im2col frontend, Linear.cpp fixed	9
Whisper conv frontend alone, im2col + GEMM, Linear.cpp fixed	13

The bad runs are real corruption, not accumulation noise: deviations up to 1.29 on a signal magnitude of 17.2, roughly 7.5%, against a bit-exact 0 for the good runs.

conv2d_gemm dispatches with lwg = 8 x 8 x 1 from the same pick_xy_square_lwg, so this is one defect reached through thirteen call sites across Linear, Matmul, Conv2dGemm, Conv1dPW, Conv2dPW, SDPA, Q8taLinear, Q8taConv2dPW, QuantizedLinear and QuantizedConvolution. Any model whose conv2d reaches the im2col path on Adreno is affected today, which includes every conv2d with out_channels >= 128.

#22328 now fixes the shared helper instead of one caller. With that, all four models above are 1 / 60. The whole-encoder cost of applying it to every dispatch is +2.9% to +5.3% rather than the +1.5% measured for the linear dispatch alone.

Worth noting for anyone else measuring this: 40 executions is not enough to call a model deterministic here. The failure rate is roughly 10 to 20 percent per execution for an affected dispatch, but it varies with shape, and a clean 40-run sample said "fixed" when it was not.