Add numpy-only simulation/tuning harness for comparing aggregators by wxiao0421 · Pull Request #31 · Roblox/Sentinel

wxiao0421 · 2026-07-27T21:18:36Z

Summary

Picking an aggregator (and its hyperparameters) is data-dependent, but there was no lightweight way to measure that choice against labeled examples. This PR adds sentinel.simulation, a numpy-only harness for tuning — no Ray, S3, or experiment trackers required.

New module src/sentinel/simulation.py exposing LabeledGroup, score_groups, evaluate_groups, compare_aggregators, and run_grid_search, all re-exported from the public API in src/sentinel/__init__.py.
Score once, compare cheaply. score_groups runs the expensive model step a single time; compare_aggregators and run_grid_search then sweep all six aggregators and hyperparameter combinations over those cached scores. Sweeping min_score_to_consider is free because _apply_threshold re-applies the exact rule calculate_rare_class_affinity uses, on already-computed scores.
Three families of evaluation metrics so tuning isn't limited to ranking quality:
- Ranking: roc_auc, recall_at_n, precision_at_n, rank_ratio
- Threshold/classification: precision, recall, f1, false_positive_rate at a chosen or automatically best-F1 cutoff
- Separation/distribution: mean_separation, cohens_d, ks_statistic (threshold-free)

Results on real data

The notebook section runs end-to-end on 30 controversial vs 30 Lex Fridman episodes and its outputs are committed, so reviewers see the actual conclusion rather than just the code.

With the library defaults (top_k=5, min_score_to_consider=0.1) the aggregators are bunched together and the choice looks unimportant:

aggregator	roc_auc
skewness	0.894
max_score	0.860
percentile_score	0.852
top_k_mean	0.848
softmax_weighted_mean	0.797
mean_of_positives	0.796

The grid search is where it pays off:

aggregator	top_k	min_score_to_consider	roc_auc	f1
skewness	5	0.0	0.977	0.935
skewness	3	0.0	0.970	0.933
skewness	10	0.0	0.953	0.915
top_k_mean	3	0.0	0.904	0.885

Two conclusions. Sentinel's default aggregator is the right one — the top three configurations are all skewness. But the default noise floor of min_score_to_consider=0.1 is costing accuracy on this dataset; dropping it to 0.0 takes skewness from 0.894 to 0.977. That is exactly the kind of finding the harness exists to surface, and it is not visible without sweeping.

Also included

README.md: new "Simulation-based tuning" section with a worked snippet.
examples/sentinel_against_hate.ipynb: the new tuning section (comparison table, per-family bar charts, grid search, ROC-AUC heatmap) plus a "What we found" write-up.
Reproducibility fixes to the notebook. Two sampling steps were unseeded and the transcript list came back in filesystem order, so every run evaluated a different set of episodes. Both samples are now seeded and the glob is sorted. The 100 evaluation episodes are drawn with a seeded random.sample rather than an alphabetical slice — the corpus is grouped by show, so [:100] would have covered only the shows sorting first and skipped special-reports entirely.
Fixed the neutral-data download cell, which failed with poetry: command not found: a %%bash cell starts a non-login shell that never picks up ~/.local/bin. It now reuses sys.executable, which is already the Poetry venv.
Docs regenerated via docs/generate_docs.py (modules.rst, sentinel.rst).
Drive-by cleanups: fixed examples/Example_Threshold_Script.py (cache_model kwarg and index path), and renamed tests/test_sriracha_local_index.py to tests/test_sentinel_local_index.py to match the project name.

Notes for reviewers

The notebook diff is large because it was re-executed; no pre-existing cell's code changed apart from the sampling and download fixes noted above. The rest is regenerated output from one consistent run.
The "What we found" cell is deliberately qualitative. With 30 episodes per class, differences of a few thousandths of ROC-AUC are inside the margin of error, and separate runs did reorder the leading aggregators, so hard-coded figures in prose would go stale on the next run.
Sanity check: compare_aggregators and run_grid_search agree exactly where they overlap (skewness at top_k=5, min_score=0.1 is 0.894444 in both), confirming the cheap post-hoc thresholding matches the full re-scoring path.
tqdm progress-bar widgets and absolute local paths were stripped from the saved outputs.
Remaining caveat, pre-existing: the evaluation data is fetched at runtime from an unpinned third-party repo and Hugging Face, so the seeds make runs reproducible on one machine but not across arbitrary checkouts.

Test plan

pytest tests/test_simulation.py — 14 new tests covering all three metric families and score_groups plumbing
pytest tests/ — full suite passes (54 tests)
python docs/generate_docs.py leaves docs/source/ clean, so the CI docs-sync check passes
Notebook re-executed end-to-end from a clean kernel; all 21 code cells ran in order, nbformat.validate passes, no cell contains an error output

Add sentinel.simulation with score_groups, evaluate_groups, compare_aggregators, and run_grid_search. evaluate_groups reports three evaluation-metric families (ranking, threshold/classification, separation/distribution) so users can tune the summarize metric and hyperparameters for their use case, not just ranking. - New module + public API exports; docs regenerated (docs-sync passes) - tests/test_simulation.py covers all three metric families and score_groups plumbing - Demonstrate in sentinel_against_hate.ipynb (compare aggregators + grid search) - README: 'Simulation-based tuning' section - Cleanups: fix Example_Threshold_Script.py (cache_model kwarg + index path), rename test_sriracha_local_index.py -> test_sentinel_local_index.py Co-authored-by: Cursor <cursoragent@cursor.com>

The tuning section previously shipped as code with no saved output, so a reader could not see what the harness actually concludes. Re-executed the notebook against the full dataset (30 controversial vs 30 Lex Fridman episodes, 28,956 segments) and committed the real tables and charts. Result: skewness is confirmed as the best aggregator, but the default min_score_to_consider=0.1 is not optimal on this data - the top four grid configurations are all skewness, led by top_k=5 with min_score_to_consider=0.0 at 0.996 ROC-AUC. - Explain why the demo subsamples to 30 controversial episodes: it matches the 30 Lex Fridman episodes so the evaluation set is balanced, which changes what top_n, precision and f1 mean. - Fix the neutral-data download cell, which failed with "poetry: command not found" because a %%bash cell starts a non-login shell that never picks up ~/.local/bin. It now reuses sys.executable, which is already the Poetry venv. - Strip tqdm progress-bar widget outputs and absolute local paths from the saved outputs. Co-authored-by: Cursor <cursoragent@cursor.com>

…esults Two sampling steps were unseeded and the transcript list came back in filesystem order, so every run scored a different set of episodes and produced different numbers. Seed both samples and sort the glob so a re-run is comparable to the one recorded here. Add a "What we found" section interpreting the tuning results. It is written qualitatively on purpose: with 30 episodes per class, differences of a few thousandths of ROC-AUC are inside the margin of error, and separate runs did reorder the leading aggregators. The claims that survive are that the top configurations are all skewness, and that min_score_to_consider matters more for skewness than for the others because skewness is computed over the whole score array while the other five filter to scores > 0. Note: the saved outputs predate the seed, so re-running will produce slightly different figures until the notebook is executed again. Co-authored-by: Cursor <cursoragent@cursor.com>

…l head Sorting the transcript glob made the selection reproducible but biased it badly. The corpus holds three shows, so transcript_files[:100] took all 16 ajn-live episodes plus the first 84 alex-jones-show ones and never reached special-reports at all. Those episodes are also far shorter, which starves the aggregators of the extreme segments they rely on: the controversial episodes scored above zero on 1.4% of segments versus 1.2% for Lex Fridman, leaving almost nothing to separate, and ROC-AUC fell to 0.70. Draw a seeded random sample over the sorted pool instead, which is both deterministic and representative (86 alex-jones-show / 14 special-reports, matching the corpus). Segment-level separation returns to 1.9% versus 1.0%. Re-ran the notebook on that sample. Untuned, the aggregators cluster between 0.80 and 0.89 ROC-AUC with skewness ahead at 0.894; the grid search lifts skewness with top_k=5 and min_score_to_consider=0.0 to 0.977, and the top three configurations are all skewness. Cross-checked that compare_aggregators and run_grid_search agree exactly where they overlap. Co-authored-by: Cursor <cursoragent@cursor.com>

wxiao0421 · 2026-07-29T18:05:56Z

Stack map

This PR is the base of a four-PR stack implementing the improvement plan. Each PR targets the one below it, so each diff shows only its own changes.

main
 └── #31  Simulation/tuning harness  (this PR)
      └── #32  Persist the corpus + deterministic loading      [bug fixes]
           ├── #33  index.subsample()                          [new method]
           │    └── #34  Grid-search index size / ratio axes   [new args]
           └── #35  SentinelLocalIndex.from_texts()            [new classmethod]

Suggested review and merge order: #31 → #32 → #33 → #34, with #35 mergeable any time after #32.

#32 goes first among the fixes because there is no point tuning parameters while loading is non-deterministic — you cannot tell a real improvement from random variation in which negatives happened to survive. It is also the easiest to approve, since all three of its parts are defensibly bug fixes rather than features.

#35 deliberately branches off #32 rather than sitting on top of the stack: its only real dependency is the _take_rows helper, so it does not need to wait on #33 and #34.

PR	What it does	Risk
#32	`corpus.json`, corpus/embedding alignment, seeded loading	Low — additive and backward compatible
#33	Resize an index without re-encoding	Low — new method, nothing existing changes
#34	Sweep index size and ratio in `run_grid_search`	Low — new args default to `None`
#35	Build an index in one line	Low — new classmethod

Verified the stack combines cleanly: merging #35 into the tip of #34 produces no conflicts, and the combined tree passes all 99 tests with a clean flake8.

CI note: .github/workflows/test.yml only triggers on pull requests targeting main, so #32–#35 show no checks until they are retargeted. Each was run locally (pytest, flake8, docs/check_docs_sync.py) and each PR description records the results.

vcai4071 · 2026-07-29T18:08:02Z

+        observation_scores = np.asarray(
+            list(result.observation_scores.values()), dtype=float
+        )


Not a new change from this PR, but an observation: If there are repeated/duplicate messages, it seems calculate_rare_class_affinity dedupes when it builds a dict of text --> score. We could do something like

scores_by_text = result.observation_scores observation_scores = np.asarray( [scores_by_text[text] for text in observations], dtype=float )

to restore the full-length array.

This correction makes sense. Will fix with a follow up PR.

vcai4071 · 2026-07-29T18:10:34Z

+        in_top = labels[leaderboard_rank <= top_n_eff]
+        tp_in_top = int(np.sum(in_top == 1))
+        metrics["precision_at_n"] = tp_in_top / top_n_eff
+        metrics["recall_at_n"] = tp_in_top / n_pos


Edge case: with _average_ranks, ties are averaged, so it may be possible for precision_at_n > 1. For example, five positives tied at the top, the rank for all 5 will be 3. With top_n_eff=3, tp_in_top would be 5?

Will address in follow up PR as well.

vcai4071

LGTM! A couple of minor comments.

The "Tuning aggregation strategies" section is itself a 2.0 feature - the simulation harness arrived in #31, inside this release - so the appendix opened on a false premise by calling everything above it 1.0. Corrected, and it now points at that section as the worked example on real data. Adds section 7, which sweeps the axes that section could not: three index sizes x three negative ratios x three top_k x two thresholds over the real 30-vs-30 podcast set, 22,885 segments, 324 rows. It runs on the groups built earlier in the notebook rather than the appendix's toy fixture, and is marked as depending on them. That sweep is only practical because of the embedding cache. Encoding those segments takes 118s and a scoring pass afterwards takes 1.4s, so the 27 configurations are 2.6 minutes rather than 54. Three findings, all from the executed output. A larger index is reliably better, with no plateau at the full 1,516 positives. Fewer neighbours beat more, top_k=3 winning on both mean and max. And the aggregator ranking inverts depending on how you read it: top_k_mean has the best average ROC-AUC while skewness has by far the best achievable, 0.996 against 0.890, so skewness is the most configuration-sensitive of the six. Picking by average would have chosen top_k_mean and given up about a tenth of a point. The EditNotebook pass used for the intro correction stripped required fields from 41 stream outputs across 19 of the original cells, which invalidated the notebook. The original 32 cells are restored byte-for-byte from before this release's work, and the remaining outputs repaired, so the file validates again. Co-authored-by: Cursor <cursoragent@cursor.com>

…lt columns, and document the release (#36) * Reuse observation embeddings across a sweep, and guard result columns Three related changes for 2.0.0. run_grid_search re-encoded the observation texts on every scoring pass, even though an observation's embedding depends on the encoder and never on the index it is scored against. subsample() shares the parent's sentence model and encoding kwargs, so one set of embeddings is valid for a whole sweep. calculate_rare_class_affinity now accepts sample_embeddings, simulation gains encode_observations(), and run_grid_search hoists the encoding out of its loops. On a 2x2x3 sweep over 320 observations that is 2.99s to 0.52s, a 5.7x saving, with byte-identical rows. Kept behind cache_observation_embeddings so callers short of memory can opt out, and indices that cannot pre-encode fall back rather than failing, which keeps the harness usable with the duck-typed doubles it advertises support for. Result rows are plain dicts so they drop into a DataFrame, but that also means a second write to a key silently destroys the first - which is exactly how an index size once replaced the positive-group count. The grid-search columns are now named constants applied through _add_columns, which raises instead of overwriting. run_grid_search's arguments were ordered by the accident of when each was added, leaving the two index axes separated from the other sweep axes by unrelated plumbing. They are all keyword-only, so regrouping them by role breaks nothing; they now read in the same order as the cost-ordered loop the docstring describes. On SentinelLocalIndex the opposite was true: everything was positional-or-keyword, so a new argument could only be appended. sample_embeddings belongs beside text_samples, so the parameters after it become keyword-only, as do those of from_texts and load. Every call site in the repo already used keywords. Also converts __init__.py from tabs to spaces, the only such file in the repo and the source of all 29 W191 warnings. Co-authored-by: Cursor <cursoragent@cursor.com> * Document the 2.0 features and how to migrate Adds V2_FEATURES.md covering each addition since 1.0 with what it is, why it exists and how to use it, plus migration notes for the two deliberate breaks: keyword-only arguments after the first, and the index_-prefixed grid-search columns. In the README, replaces the "What's New" section with a 2.0 summary table pointing at the relevant sections, adds a section on subsample(), and documents the index sweep axes and the observation embedding cache with its measured saving. Also separates the two things called "metrics" in the tuning section, since the aggregator is swept while the evaluation metrics are all reported. Co-authored-by: Cursor <cursoragent@cursor.com> * Demonstrate the 2.0 features in the hate-speech notebook Appends a self-contained appendix showing from_texts(), the persisted corpus and seeded loading, subsample(), the grid-search index axes, and the observation embedding cache, with executed outputs. Self-contained deliberately: it builds a small index from a dozen example sentences rather than depending on the data-loading cells above, so it runs in seconds on its own. The shipped example index is used only for the subsample demonstration, where a realistic size is the point - it predates corpus support, so its explanations would show row numbers rather than text. Appended rather than woven in, so the diff is 789 insertions and no deletions and the existing 32 cells are byte-identical. The final aggregator comparison reports the whole table rather than picking a winner: all six separate this example perfectly, which says the example is easy, not that the aggregators are equivalent. Sorting by Cohen's d there produced a number like 8.5e15, because the within-class variance is zero on a fixture this clean. The real comparison is the earlier section, on real data. Co-authored-by: Cursor <cursoragent@cursor.com> * Cover the remaining V2 additions in the notebook appendix An audit of the public API added since 0e924ae against the appendix found four things demonstrated nowhere: evaluate_groups, DEFAULT_AGGREGATORS, load_corpus, and the sample_embeddings parameter itself, which the appendix only reached indirectly through encode_observations. Adds a section introducing the harness by cost - score once, then evaluate with one aggregator, then all of them, then sweep - which is where evaluate_groups and DEFAULT_AGGREGATORS naturally belong, and which also shows the three metric families on a single row. The group fixture moves there, since that section is now the first to use it. Extends the corpus section with load_corpus, reading the texts back without the embeddings. It returns (None, None) for the shipped example index, which is a concrete demonstration of the pre-2.0 format rather than an assertion about it. Extends the caching section with the underlying parameter, including the refusal when the embeddings do not line up with the text. Co-authored-by: Cursor <cursoragent@cursor.com> * Add a real-data configuration sweep and its conclusions to the appendix The "Tuning aggregation strategies" section is itself a 2.0 feature - the simulation harness arrived in #31, inside this release - so the appendix opened on a false premise by calling everything above it 1.0. Corrected, and it now points at that section as the worked example on real data. Adds section 7, which sweeps the axes that section could not: three index sizes x three negative ratios x three top_k x two thresholds over the real 30-vs-30 podcast set, 22,885 segments, 324 rows. It runs on the groups built earlier in the notebook rather than the appendix's toy fixture, and is marked as depending on them. That sweep is only practical because of the embedding cache. Encoding those segments takes 118s and a scoring pass afterwards takes 1.4s, so the 27 configurations are 2.6 minutes rather than 54. Three findings, all from the executed output. A larger index is reliably better, with no plateau at the full 1,516 positives. Fewer neighbours beat more, top_k=3 winning on both mean and max. And the aggregator ranking inverts depending on how you read it: top_k_mean has the best average ROC-AUC while skewness has by far the best achievable, 0.996 against 0.890, so skewness is the most configuration-sensitive of the six. Picking by average would have chosen top_k_mean and given up about a tenth of a point. The EditNotebook pass used for the intro correction stripped required fields from 41 stream outputs across 19 of the original cells, which invalidated the notebook. The original 32 cells are restored byte-for-byte from before this release's work, and the remaining outputs repaired, so the file validates again. Co-authored-by: Cursor <cursoragent@cursor.com> * Cover the summarize metrics, explainability and Docker in the 2.0 notes The release range starts at #21, not after it, so three things the notes had omitted or mis-attributed are in scope. 1.0 shipped two summarize metrics, mean_of_positives and skewness. 2.0 has six, adding top_k_mean, percentile_score, softmax_weighted_mean and max_score. An earlier draft of this file claimed the opposite - that all six predated the release - which was wrong, and understated it: the release both widened the choice and supplied the evidence to make it. Also documents the explainability fields on RareClassAffinityResult, and the Dockerfiles, neither of which appeared anywhere in the notes. Replaces the flat list of links with a table of the main changes and a one-line summary of each, so the file opens with what matters rather than an index. Co-authored-by: Cursor <cursoragent@cursor.com> * Require Python 3.10, and resync the lock 1.0 declared support for Python 3.9 in both the constraint and the classifiers, but CI has only ever run 3.10 to 3.12, so the claim was never verified. It had also stopped being possible to honour: current torch requires 3.10 or newer, and the constraint here allows anything up to 3.0, so a 3.9 install had to silently resolve to an older torch that nobody tests. An unverified promise that quietly hands users a different dependency set is worse than no promise. The source itself was already 3.9-clean - every file in src/ parses under ast.parse(feature_version=(3, 9)) and there are no 3.10-only runtime APIs - so this is about what the metadata claims rather than about fixing breakage. Raising the minimum is a breaking change, which is why it goes in the major release rather than waiting. Relocking drops three backports whose functionality is in the standard library from 3.10: importlib-metadata, importlib-resources and zipp. Also removes 200-odd python_version < "3.10" markers that can no longer fire. No package versions change, and no packages are added. The lock had to be regenerated because it records the python constraint and a content-hash derived from pyproject.toml; leaving it stale would fail the poetry install that CI runs. Regenerated with Poetry 2.4.1 to preserve lock-version 2.1 - relocking with the older Poetry on this machine silently rewrote it to 2.0. Both versions validate the result. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

wxiao0421 and others added 4 commits July 27, 2026 14:18

vcai4071 reviewed Jul 29, 2026

View reviewed changes

vcai4071 approved these changes Jul 29, 2026

View reviewed changes

wxiao0421 merged commit 8179021 into main Jul 29, 2026
4 checks passed

wxiao0421 deleted the feat/simulation-tuning-harness branch July 29, 2026 19:58

Navigation Menu

Uh oh!

Add numpy-only simulation/tuning harness for comparing aggregators - #31

Add numpy-only simulation/tuning harness for comparing aggregators#31
wxiao0421 merged 4 commits into
mainfrom
feat/simulation-tuning-harness

wxiao0421 commented Jul 27, 2026 •

edited

Loading

Uh oh!

wxiao0421 commented Jul 29, 2026

Uh oh!

vcai4071 Jul 29, 2026

Uh oh!

wxiao0421 Jul 29, 2026

Uh oh!

vcai4071 Jul 29, 2026 •

edited

Loading

Uh oh!

wxiao0421 Jul 29, 2026

Uh oh!

vcai4071 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wxiao0421 commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Results on real data

Also included

Notes for reviewers

Test plan

Uh oh!

wxiao0421 commented Jul 29, 2026

Stack map

Uh oh!

vcai4071 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

wxiao0421 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

vcai4071 Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wxiao0421 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

vcai4071 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wxiao0421 commented Jul 27, 2026 •

edited

Loading

vcai4071 Jul 29, 2026 •

edited

Loading