Skip to content

Fix NaN in gammpapprox lower branch for large a - #51

Open
gaoflow wants to merge 1 commit into
Axect:masterfrom
gaoflow:fix-gammpapprox-large-a-lower-branch-nan
Open

Fix NaN in gammpapprox lower branch for large a#51
gaoflow wants to merge 1 commit into
Axect:masterfrom
gaoflow:fix-gammpapprox-large-a-lower-branch-nan

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 28, 2026

Copy link
Copy Markdown

Problem

For a >= ASWITCH (100), gammp/gammq take the Gauss-Legendre quadrature
path in gammpapprox. Over the whole region 0.2a <= x <= a-1 they return
NaN, and invgammp panics for lower-tail p < 0.5.

gammp(100.0, 90.0)   // NaN   (mpmath P = 0.158220989186)
gammq(150.0, 120.0)  // NaN   (mpmath Q = 0.995436558696)
invgammp(0.1, 100.0) // panic: "Bad args in gammp"

These are the regularized incomplete gamma functions, i.e. the chi-square /
gamma / Poisson CDF, so for example a chi-square CDF with >= 200 degrees of
freedom (a = df/2 >= 100) is NaN over its entire lower half.

Root cause

gammpapprox computes the answer in log space:

let log_ans = log_max + sum.ln() + (xu - x).ln() + log_scale;

In the x <= a1 branch xu is set below x, so xu - x < 0 and
(xu - x).ln() is NaN, which propagates to both P and Q. invgammp
then feeds that NaN into gammp's assert!(x >= 0.0) (NaN >= 0.0 is
false) and panics.

The Numerical Recipes gammpapprox keeps ans = sum*(xu-x)*exp(...) as a
signed quantity (negative in this branch) and folds the sign into the return:
P = ans > 0 ? 1-ans : -ans, Q = ans >= 0 ? ans : 1+ans. The log-space
rewrite in 948acde kept only the x > a1 case and dropped the lower-branch
fold, while also taking ln of a negative width.

Fix

Take the log of the magnitude |xu - x| and restore the per-branch sign on
return:

  • lower branch (x <= a1): P = ans, Q = 1 - ans
  • upper branch (x > a1): P = 1 - ans, Q = ans (unchanged)

With gammpapprox correct, invgammp converges on both tails for large a
with no extra guard.

Verification

Reference values are mpmath (gammainc(a, 0, x, regularized=True)), stable at
dps 30 and 35. The new tests in tests/gammpapprox_large_a_test.rs cover:

  • gammp/gammq for a in {100, 150, 200, 500} with x spanning the former
    NaN region, x = a-1, x = a, and x > a; mid-range values (0.158, 0.00456,
    0.472, 0.972) asserted at full precision, plus P + Q == 1 and finiteness.
  • P(a, .) monotonic increasing across 0.2a .. a + 6*sqrt(a).
  • invgammp round-trip both tails (gammp(a, invgammp(p, a)) == p) and a
    direct match against mpmath findroot values.
  • the small-a gser/gcf path unchanged.

The old large-a quadrature test only exercised gammp(150, 150) (x = a, the
upper branch), so the lower branch was untested. Full suite passes; cargo fmt
clean; the change adds no new clippy warnings.

Fixes the large-a case of #4.

For a >= ASWITCH (100) the Gauss-Legendre quadrature path took the log of
the integration width (xu - x), which is negative in the x <= a-1 branch,
so (xu - x).ln() evaluated to NaN. That poisoned gammp/gammq (chi-square /
gamma / Poisson CDF returned NaN over the whole 0.2a <= x <= a-1 region)
and made invgammp panic for lower-tail p < 0.5 when the NaN reached
gammp's assert.

The log-space rewrite that introduced this kept only the x > a1 case of
the Numerical Recipes sign convention and dropped the lower-branch fold.
Take the log of |xu - x| and restore the per-branch sign on return: in the
lower branch the signed integral is negative, so P = ans and Q = 1 - ans;
the upper branch is unchanged (P = 1 - ans, Q = ans).

With gammpapprox correct, invgammp converges for large a on both tails
without any extra guard. Values verified against mpmath; the small-a
gser/gcf path is untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant