RealVNC Viewer - Apple ARD Diffie-Hellman interop patch

A 10-byte binary patch for RealVNC Viewer 7.15.1 (r18), Windows x64, fixing two defects that prevent it from connecting to macOS Screen Sharing.

Symptom

Connecting to a modern macOS host fails at the security handshake:

Protocol error: key length too large

and, once that is fixed, immediately after:

Unsupported DH generator 5

What the server actually sends

Apple Screen Sharing, RFB security type 30 (Apple ARD, Diffie-Hellman). Raw bytes after selecting the security type:

00 05 02 00 FF FF FF FF FF FF FF FF C9 0F DA A2 21 68 C2 34 ...
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 gen   len   prime (RFC 3526 MODP Group 16)

Field	Value
generator	5
keyLength	0x0200 = 512 bytes (4096-bit)
prime	RFC 3526 MODP Group 16, a safe prime (p ≡ 23 mod 24)

Wire format is U16 generator, U16 keyLength, keyLength bytes prime, keyLength bytes peer public key - all big-endian.

Root cause

Two independent bugs, one behind the other.

Bug 1 - off-by-one on key length

The protocol parser and the DH layer disagree by exactly one byte about the maximum modulus size.

; DH layer - VA 0x14052ECB0 - NOT patched, shown for contrast
        mov     eax, [rbx]              ; modulus size in bytes
        sub     eax, 0x20
        cmp     eax, 0x1E0              ; 480
        ja      .unsupported_key_length ; accepts 32 .. 512 INCLUSIVE

; protocol parser - VA 0x1403E3230 - file offset 0x003E2630
        movzx   ecx, byte [rdx]
        shl     cx, 8
        movzx   eax, byte [rdx+1]
        or      cx, ax                  ; cx = keyLength (big-endian U16)
        mov     eax, 0x200              ; 512
        cmp     cx, ax
        jae     .throw                  ; rejects keyLength >= 512  <-- off by one

The DH implementation handles 512 bytes fine; the parser refuses to hand it over. A 4096-bit group lands exactly in that gap.

Bug 2 - generator whitelist too narrow

The generator is validated against a hardcoded set of {2, 23}. macOS uses 5.

The value is not used in any specialised code path - it goes straight into an MPI:

; VA 0x14052ED31
        mov     rcx, [rdi+0x18]
        mov     rdx, rsi                ; rsi = generator from the wire
        add     rcx, 0x20               ; &dhm->G
        call    mbedtls_mpi_lset        ; G = generator

So the whitelist is policy, not an implementation limit.

Patch A - key length off-by-one

File offset 0x003E2647, 1 byte: 83 → 87 (jae → ja)

        cmp     cx, ax                  ; ax = 512
-       jae     .throw                  ; 0F 83 ...  rejects >= 512
+       ja      .throw                  ; 0F 87 ...  rejects >  512

This makes the parser agree with the DH layer's own limit. The cap still exists - anything above 512 is still rejected.

Patch B - generator whitelist

File offset 0x0052E0C0, 14 bytes. Same instruction-block length, so nothing shifts.

; before
        cmp     esi, 2                  ; 83 FE 02
        je      .accept                 ; 74 09
        cmp     esi, 23                 ; 83 FE 17
        jne     .unsupported_generator  ; 0F 85 A1 01 00 00
.accept:

; after
        mov     eax, 0x00800024         ; B8 24 00 80 00   bit mask: bits 2, 5, 23
        bt      eax, esi                ; 0F A3 F0         CF = mask[esi mod 32]
        jnc     .unsupported_generator  ; 0F 83 A1 01 00 00
.accept:

before: 83 FE 02 74 09 83 FE 17 0F 85 A1 01 00 00
after:  B8 24 00 80 00 0F A3 F0 0F 83 A1 01 00 00

A bitmask was chosen over the simpler "change 23 to 5" so the accepted set is a superset of the original - nothing that worked before stops working.

Register safety: esi holds the generator and is read later at VA 0x14052ED35, so it must survive - the patch does not touch it. eax is dead at this point: its last write is the modulus size check above, and its next read is a call return value.

Why this is safe

Both patches leave the parameter validation that actually matters completely intact. Immediately after the generator check, the code still verifies with mbed TLS that P is prime and that (P-1)/2 is prime - 40 rounds of Miller-Rabin each - i.e. that the modulus is a genuine safe prime.

For a safe prime p = 2q+1 the only subgroup orders are 1, 2, q and 2q, so any generator other than 0, 1 and p-1 lands in a large subgroup. The mask rejects 0 and 1 (bits 0 and 1 are clear), and p-1 is a 4096-bit number that cannot fit the U16 wire field.

Known cosmetic side effect: bt with a 32-bit operand takes the bit index modulo 32, so generators congruent to 2, 5 or 23 mod 32 (34, 37, …) are also accepted. Harmless for the reason above.

Patch A is not a weakening at all - it is a bug fix that makes one bound match the other.

SHA-256 reference

State	SHA-256
Original, unpatched	39195C0ABE53DC8FC2A57BEEB6DF9ACCF72226D46688845BF864C4D639030D84
Patch A only	E9EC1ED67456A68FE29541C641E539BAE2FAD9BE92945AC657CAE2F0DE1C9051
Patch B only	6DFBA864629D0A08260A119E1C144E281C0DB76D9FDDA9E03DDD2602602C50C2
Patch A + B (target)	8E52F660B30A0B772C66B4A26BE316F7CF1390981169CC71D652D4C73389B33F

Files

File	Purpose
apply-patches.ps1	Applies both patches. Idempotent, hash-verified. Needs admin.
rollback.ps1	Restores the original from backup. Needs admin.
probe-dh.ps1	Reads and analyses the server's DH parameters. No admin.
probe-dh2.ps1	Raw byte dump across RFB versions × security types. No admin.

Usage

Run from a normal PowerShell prompt; it self-elevates via UAC.

Start-Process powershell -Verb RunAs -ArgumentList '-NoProfile','-ExecutionPolicy','Bypass','-NoExit','-File','C:\path\to\apply-patches.ps1'

The script refuses to touch anything unless the current SHA-256 matches a known state and the bytes at both offsets match. It backs up to vncviewer.exe.orig-7.15.1.bak next to the binary, verifies the result after writing, and restores from the backup automatically if verification fails.

Rollback:

Start-Process powershell -Verb RunAs -ArgumentList '-NoProfile','-ExecutionPolicy','Bypass','-NoExit','-File','C:\path\to\rollback.ps1'

Caveats

• Auto-update reverts this. RealVNC will overwrite vncviewer.exe and the bug comes back. Re-run apply-patches.ps1; if the version changed it aborts safely instead of corrupting the binary.
• Offsets are version-specific. They are valid for 7.15.1 (r18) x64 only.
• The Authenticode signature is broken by patching. The viewer does not self-verify, but SmartScreen may comment.

Re-deriving the offsets after an update

1. Find the ASCII string key length too large in .rdata.
2. Scan .text for rip-relative references to it - there is (should be) exactly one.
3. Walk back from the throw site to the cmp cx, 0x200 / jae pair. That is patch A.
4. Follow the call that consumes the generator (mov rdx, rsi → mbedtls_mpi_lset) to the DH setup function; the cmp esi,2 / cmp esi,23 block just above it is patch B.
5. Use .pdata (RUNTIME_FUNCTION entries) to get exact function bounds - note that one logical function may span several consecutive entries.

PowerShell gotcha when parsing big-endian fields: -shl preserves the [byte] type and truncates, so [byte]2 -shl 8 is 0, not 512. Cast to [int] first.