Default decile impacts to household net income and person-weighted groups

[anth-volk]

Fixes #485

Summary

• default DecileImpact.income_variable and calculate_decile_impacts to household_net_income
• replace the unweighted pandas.qcut fallback with shared microdf weighted ranking for both DecileImpact and IntraDecileImpact
• form computed household groups with household_weight * household_count_people, while other entities use their entity survey weights
• clamp computed groups to 1..quantiles; exclude negative, missing, and non-finite computed ranking values under the documented rules
• constrain microdf_python to >=1.3.0,<1.4, with tests locking in the reviewed max-rank tie behavior
• preserve explicit income concepts and precomputed grouping variables, including out-of-range sentinel values that intentionally exclude rows
• make UK economic-impact analysis pass its income concept explicitly so future shared-default changes do not alter country analysis silently
• calculate DecileImpact statistics with entity survey weights and IntraDecileImpact proportions with effective population weights
• calculate relative changes from weighted group means and the overall intra-decile row directly from the included population
• preserve empty-group values as JSON-safe nulls when collection DataFrames are converted to records
• add an approved behavior specification and tests covering weighting, ties, exclusions, invalid inputs, empty groups, alignment, serialization, and overall results

Breaking change

This intentionally changes default public decile results and is classified with a .breaking.md Towncrier fragment. The default income concept changes for callers that previously relied on equivalised HBAI income, and computed groups change from equal raw survey rows to person-weighted population groups.

The UK .py economic-impact bundle changes both its income concept and its grouping. The US bundle already selected household_net_income explicitly, so its income concept does not change, but its computed grouping also becomes person-weighted.

Root cause

The original UK report used equivalised HBAI income to rank households while calculating several separate outcomes within those deciles. When the generic DecileImpact output was extracted, the ranking variable and measured outcome were collapsed into one UK-specific default.

Separately, the generic fallback passed a weighted MicroSeries to pandas.qcut. Pandas ignored the survey weights, so the groups represented equal raw rows rather than approximately equal populations. The prior output code also used unweighted means/counts, averaged observation-level relative changes, and constructed the overall intra-decile row by averaging decile proportions.

Implementation

A shared _PreparedDecileAnalysis path now resolves the target entity, maps income values when needed, validates baseline/reform alignment and weights, creates baseline groups once, and supplies the two output types with their distinct analysis weights. The already-validated effective grouping weight is passed into group construction, avoiding duplicate calculation and validation.

Empty and excluded populations are represented explicitly rather than being reported as no-change populations. Nullable result columns use Pandas Float64, so dataframe.to_dict("records") returns None and strict JSON serialization emits null rather than non-standard NaN.

The behavior is documented in docs/decile-impact-spec.md, including indivisible survey rows, tied-income max-rank grouping, negative and non-finite ranking values, zero weights, precomputed exclusion sentinels, and null results for empty populations.

User impact

Default decile results now measure household net income, including effects represented in household_tax, and computed household groups are person-weighted. Consumers can still request equivalised HBAI income or pass a precomputed grouping variable explicitly. Household-level decile means and outcome counts remain household-survey-weighted, while intra-decile proportions are person-weighted.

Validation

• make format — passed
• make lint — passed
• focused and adjacent regression suite — 63 passed
• targeted mypy check of the four changed output modules — passed
• the prior commit passed the complete Python 3.13/3.14, documentation, changelog, lint, mypy, bundle-metadata, and smoke-import CI matrix; the updated branch has triggered a fresh CI run

[vahid-ahmadi]

Reviewing from the downstream side — UK Chat PR PolicyEngine/policyengine-uk-chat#217 is pinned to this commit, so we care about both the semantics and the release timing.

The diagnosis in the description is convincing, and both halves of the fix look right to me: separating the ranking variable from the measured outcome, and replacing pandas.qcut (which silently discarded the MicroSeries weights) with weighted ranking. Extracting calculate_decile_groups so DecileImpact and IntraDecileImpact cannot drift apart is the right shape, and making UK economic_impact_analysis pass its income concept explicitly is a good guard.

Substantive

• calculate_decile_groups clamps the top but not the bottom:

  groups = np.minimum(np.ceil(percentile_ranks * quantiles), quantiles).astype(int)

  rank(pct=True) appears to be cumulative-inclusive (the [10, 20, 30, 40] / person-weights [4, 1, 1, 1] fixture landing on [6, 8, 9, 10] implies 4/7 → 0.571). So the smallest rank is w_first / total, which is 0 whenever the lowest-income rows carry zero weight — and zero-weight households do occur in some datasets. Those rows then get group 0, which falls outside the range(1, quantiles + 1) loop in calculate_decile_impacts and disappears from every decile without an error. A matching np.maximum(..., 1) would close that off. A fixture with a leading zero-weight row would pin the behaviour either way.

• Consider framing this as a breaking change rather than .fixed. Default decile results move for every consumer of calculate_decile_impacts and compute_intra_decile_impacts, not only UK Chat — both the income concept and the grouping change, and published decile charts will shift. A heads-up to the US side and policyengine-api before release seems worth it.

• Cost: each DecileImpact.run() recomputes the full grouping, and calculate_decile_impacts constructs ten of them, so the weighted rank runs ten times over the whole frame. That was already true of qcut, so it is not a regression, but MicroSeries.rank is meaningfully more expensive and this now runs on FRS-sized data on every UK Chat decile request. Hoisting the grouping into calculate_decile_impacts and passing it down would be a straightforward follow-up if it shows up in profiles.

Minor

• microdf_python>=1.3.0 is a floor, not a pin, and the tie-handling behaviour the fixtures rely on is version-sensitive. Worth confirming the lockfile bump is enough for consumers installing outside the lock.
• The decile_variable branch returns baseline_data[decile_variable] unvalidated, so a precomputed variable outside 1..quantiles fails the same silent-drop way. Pre-existing, and out of scope here, but it is the same failure mode as the clamp point above.

Release timing

Whenever this lands, could you drop a note on the release version? UK Chat #217 is currently carrying an immutable GitHub-archive pin to a289cc4 in backend/requirements.txt, which we would rather not merge to production — we want to swap it for the first release containing this commit.

[vahid-ahmadi]

Review

Strong PR overall. The weighted-ranking fix is clearly correct relative to the old pd.qcut fallback (which silently ignored survey weights), the person-weighted grouping matches the country-package convention, and the direct-population decile=0 row fixes a real statistical bug (arithmetic mean of unequal decile proportions). The behavior spec in docs/decile-impact-spec.md plus the worked examples is excellent — it makes the tie/exclusion/empty-group semantics reviewable, and the test suite covers essentially every row of the exclusions table. The empty-group change (nulls instead of a fake "everyone no-change" group) is the right call.

A few points, none blocking:

1. Exact pin microdf_python==1.3.0 in a library (pyproject.toml). This will cause resolver conflicts for any downstream package that also depends on microdf with a different constraint (country packages, API). Since test_household_groups_keep_tied_incomes_together and the other grouping tests already lock in the max-rank tie semantics, they would catch a behavior change in a future microdf — consider >=1.3.0,<2 (or <1.4) and letting the tests guard the semantics instead of the pin. If the exact pin is deliberate policy, fine, but worth confirming.

2. Double weight computation/validation. _prepare_decile_analysis calls _get_decile_weights(...), and then calculate_decile_groups(...) calls it again internally for the computed-group path. Harmless but redundant — calculate_decile_groups could accept the already-validated effective_weight as a parameter.

3. Loosened test contract: test_computed_groups_reject_zero_total_effective_weight accepts (ValueError, ZeroDivisionError), but the code explicitly raises ValueError and the spec says "raise an error". Tightening to ValueError keeps the test honest about the contract.

4. Downstream null handling. baseline_mean/relative_change etc. can now be None (empty groups, zero baseline mean). The breaking-change fragment covers the semantics, but worth double-checking that consumers of results.dataframe (app charting, report generation) tolerate NaN rows rather than crashing — the UK/US bundles will now emit them whenever a decile ends up empty after exclusions.

5. Exact-zero count_no_change. income_change == 0 uses exact float equality; simulation noise (e.g. recomputation jitter) will classify near-identical households as winners/losers. That mirrors the previous behavior and the spec says "equal", so no change needed — just flagging it as a known sharp edge if winners/losers counts look noisy on real reforms.

Also verified: groups.isin(range(1, q+1)) treats pd.NA and out-of-range floats as excluded; zero-weight clamp-to-group-1 and negative-income -1 assignment match the spec's worked examples; calculate_decile_impacts prepares once and reuses across the ten deciles while standalone .run() stays self-contained.

Verdict: looks good to merge once the microdf pin question is settled.