Skip to content

feat(payments): server-side idempotency for payment-send mutations (ENG-530) - #460

Open
islandbitcoin wants to merge 1 commit into
mainfrom
eng-530-payment-idempotency
Open

feat(payments): server-side idempotency for payment-send mutations (ENG-530)#460
islandbitcoin wants to merge 1 commit into
mainfrom
eng-530-payment-idempotency

Conversation

@islandbitcoin

Copy link
Copy Markdown
Contributor

Why

On 2026-07-23 a $140 USDT intraledger send fired twice ~1.5s apart (client double-fire; the second mutation began before the first settled) and the backend executed both — the sender was debited $280. Payment-send mutations had no idempotency protection: intraledger mints a fresh IBEX invoice and pays it with no dedupe key, so a double-tap / retry-on-timeout / replay double-pays. Fixes ENG-530.

What

Adds an optional client-supplied idempotencyKey to the payment-send mutations. A repeated request with the same key returns the original result instead of executing again — exactly-once settlement.

  • New withPaymentIdempotency() (src/app/payments/idempotency.ts) wraps the exported send functions. Dedupe scoped to (senderWalletId, idempotencyKey).
  • Reuses existing primitives: RedisCacheService for the result store (24h TTL) + a new LockService().lockPaymentIdempotencyKey (redlock .using, distinct namespace) for the in-flight guard.
  • Double-checked locking: fast-path cache read → on miss acquire the lock → re-check cache inside the lock → execute → persist → release. cache.set completes before the lock releases, so a concurrent second request either replays the cached result or gets a busy error — never a second payment.
  • Request-param fingerprinting: the cached envelope stores a sha256 fingerprint of the payment params (recipient/amount/bolt11). Same key + same params → replay; same key + different params → IdempotencyKeyReuseError (never a silent wrong-success, never a fresh execution).
  • Applied to all five send paths: intraledger BTC + USD wallet-id, ln-invoice, ln-no-amount. The ln-invoice resolver bypasses @app in this fork, so the inline Ibex.payInvoice is wrapped there too.
  • idempotencyKey is optional — existing clients are unaffected (mobile key-attachment is the follow-up, ENG-533).

Safety properties (verified in adversarial security review)

  • Exactly-once for the incident: concurrent same-key double-fire cannot execute two payments.
  • Fail-closed on Redis down: with a key present, a Redis outage returns a lock error without paying — never an unprotected payment.
  • No cross-user collision (key scoped to sender wallet); replays skip the ops-feed event (feat: Discord ops activity feed for key user flows #458 notify lives inside execute); errors are not cached (retryable).
  • Successful-pay-but-failed-cache.set now records a Critical exception so ops can catch a persist failure that could otherwise enable a later re-pay.

Documented residual (non-blocking)

  • IBEX-debited-but-errored: if IBEX moved money but returned an error, the result isn't cached, so a same-key retry can re-pay — no worse than pre-change; fully closed only once IBEX exposes its own idempotency. Documented in the helper header.
  • Client contract: a client receiving the busy/lock error MUST retry with the same key, never a new one. Documented.

Tests

37/37 payments specs (12 new: same-key replay, concurrent double-fire → single execution, fingerprint-mismatch conflict, no-key backwards-compat, no-double-ops-event, per-wallet scoping, failed-set records Critical, JSON round-trip replay). graphql + payments: 149/149. yarn tsc-check clean; make codegen regenerated the supergraph (4 clean idempotencyKey: String additions).

Out of scope

Cashout-initiate, the Fygaro credit path (will reuse this helper — ENG-440), and mobile key-attachment (ENG-533).

🤖 Generated with Claude Code

https://claude.ai/code/session_01UXnQYUbKQe5wUCWEvdAau7

…NG-530)

A double-fired intraledger send (client double-tap; the second mutation started
before the first settled) executed twice and double-debited the sender, because
payment-send mutations had no dedupe key. This adds an optional client-supplied
idempotencyKey so a repeated request returns the original result instead of
minting a fresh IBEX invoice and paying again — exactly-once settlement.

- New withPaymentIdempotency() helper (src/app/payments/idempotency.ts) keyed on
  (senderWalletId, idempotencyKey). Fast-path replays a cached result; otherwise
  acquires a short auto-releasing lock, executes, and persists the outcome (24h
  TTL). Reuses RedisCacheService for the store and a new
  LockService().lockPaymentIdempotencyKey — a redlock `.using` lock on a distinct
  resource namespace from the existing header-middleware lock.
- Request-param binding: the cached result is bound to a sha256 fingerprint of the
  payment params (recipient/invoice + amount). Same key + same params -> replay;
  same key + different params -> IdempotencyKeyReuseError (new, mapped in
  error-map.ts), never a silent replay that drops the new payment. Checked on both
  the fast path and the in-lock re-check.
- cache.set failure after a completed send is recorded via
  recordExceptionInCurrentSpan at Critical level (money moved but result not
  persisted -> a retry could double-pay; can't un-pay, so make it visible).
- Wrap the exported send functions (intraledger BTC/USD, payInvoiceByWalletId,
  no-amount BTC/USD). The #458 ops-event notify lives inside the wrapped body, so a
  cached replay emits no duplicate ops event and mints no second invoice.
- Thread idempotencyKey through the intraledger + ln-noamount resolvers; wrap the
  ln-invoice resolver's inline IBEX call (it bypasses @app in this fork). SDL +
  supergraph regenerated via `make codegen`.
- Optional field: absent = unchanged behavior. Only definitive outcomes are cached,
  so validation/transient errors can retry; the lock guards the concurrent window
  regardless.
- Tests: helper unit specs (replay, JSON round-trip, fingerprint-mismatch conflict,
  lock-busy, error-not-cached, set-fail records exception, per-wallet scoping) +
  integration specs on the wrapped intraledger send (same-key replay, concurrent
  double-fire, different keys, no key, no double ops event, scoping, conflict).

Client contract (documented in idempotency.ts): a busy/lock error MUST be retried
with the SAME key. Known residual gap (unchanged, documented): an IBEX
debited-but-errored return is uncached, so a same-key retry after the lock releases
can re-pay — mitigated once IBEX exposes request-level idempotency.

Out of scope (separate tickets): cashout-initiate, the Fygaro credit path, and
mobile key-attachment (ENG-533).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXnQYUbKQe5wUCWEvdAau7
@linear

linear Bot commented Jul 24, 2026

Copy link
Copy Markdown

ENG-530

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants