Skip to content

Carry coarse label-source provenance into Under the Hood reports - #52

Open
Pitchfork-and-Torch wants to merge 26 commits into
xai-org:mainfrom
Pitchfork-and-Torch:cursor/uth-label-source-provenance-dd47
Open

Carry coarse label-source provenance into Under the Hood reports#52
Pitchfork-and-Torch wants to merge 26 commits into
xai-org:mainfrom
Pitchfork-and-Torch:cursor/uth-label-source-provenance-dd47

Conversation

@Pitchfork-and-Torch

@Pitchfork-and-Torch Pitchfork-and-Torch commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Fixes #39.

README says Under the Hood should show whether labels were manually applied outside automated systems. The daily job already reads TweetSafetyLabelEvent.label but kept only timestamps; SafetyLabelSource was dropped, and the serving types had no field to carry it.

reportJson.source is one of four tokens:

  • automated ← BotMakerAction
  • manual ← ToolAction
  • llm ← a Grok/LLM union member, detected by productPrefix or getName plus the last non-empty . / $ segment (GrokAnnotation*), skipping a trailing synthetic $ and getSimpleName (can throw on Scala $ names), so this compiles if that case is absent from spam.rtf
  • unknown ← unset snapshot / tweet-flag / unmapped-source rows (serving clamp)

Jobs persist only automated | manual | llm after trim/lowercase. Unmapped, missing, leftover tokens, and tweet-flag rows stay unset (source = None is set explicitly on flag writes). Daily counts clamp the last-apply source before grouping. When an event apply and an unset snapshot apply are merged, the later timestamp wins; a persistable event source is kept only if the later row is a snapshot gap-fill (isSnapshot), not a later unmapped event. latestAsOf treats the newest asOf as a full rewrite of (user, authored day, label): it keeps every persistable source from that asOf and drops older asOf rows so a later sourced rewrite cannot sit beside an earlier unset row. The month MH writer reconstructs rows with named Scrooge fields and pattern-matches leftover same-day carried / removed (not best._4 / best._5). Serving clamps any other stored string to unknown, merges leftover rows that clamp to the same (label, source) by unioning days (larger carried/removed per day, then sum; groupBy.toList before flatMap so Map.flatMap cannot drop rows), pattern-matches that leftover group key as (name, source) rather than t._1 / t._2, and sorts that list by label then source so reportJson order is stable.

postLabelSource is a closed function (match + function closer before formatPercentage). Post-label about / effect look up by the raw label id, same as the account path; display name is only written to label.

Does not emit rule_id, actor_ldap, agent_tool, or VF-client type names. Snapshot and tweet-flag inputs stay unset (no invented source).

Based on current xai-org/x-algorithm main (c65aa179). Lane is under-the-hood/ only.

Test plan

  • Static review of the event → daily → month → reportJson path.
  • under-the-hood/ has no published build/test target in this snapshot; no job was executed.
  • Confirm Scrooge optional field source (ids 3 / 10) is backward-compatible with existing parquet/MH rows.
  • Confirm reportJson.source can only be automated | manual | llm | unknown.
  • Confirm snapshot/flag/unmapped/leftover-token rows coalesce to a single unknown row per label, including after daily count, latestAsOf, and reportJson day-union.
  • Confirm a later unset snapshot apply does not wipe an earlier persistable event source.
  • Confirm a later unmapped event apply does not inherit an earlier persistable source.
  • Confirm a newer asOf that fills source replaces older unset rows for the same (user, day, label) and does not emit both.
  • Confirm same-asOf automated and manual rows for one label both survive latestAsOf.
  • Confirm tweet-flag daily/backfill rows set source = None.
  • Confirm Automated / padded tokens normalize to the allowlist instead of unknown.
  • Confirm post-label rows are ordered by (label, source).
  • Confirm leftover serving groupBy keys (name, source) by pattern match, not t._1 / t._2.
  • Confirm leftover month same-day counts use pattern-matched carried / removed, not best._4 / best._5.
  • Confirm llm matching uses productPrefix or getName plus the last non-empty . / $ segment, not getSimpleName or String.split("$").
  • Confirm postLabelSource closes before formatPercentage.
  • Confirm postLabelAbout / postLabelEffect receive the raw label id, not the display name.

cursoragent and others added 26 commits August 14, 2026 22:09
@cursoragent @Pitchfork-and-Torch
Carry coarse label-source provenance into Under the Hood reports
5e88956 
TweetSafetyLabelEvent already has SafetyLabelSource (BotMaker vs Tool),
but the daily job dropped it and the report had no field to show whether
a label was applied manually. Persist a coarse category only — no
rule_id or actor_ldap — and emit it on each post-label row.

Fixes xai-org#39

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Tighten UTH reportJson source to automated|manual|llm|unknown
816315c 
Replace other/unavailable with unknown. reportJson clamps to the four
tokens. GrokAnnotationAction is not on the published spam.rtf IDL the
jobs compile against, so it is not matched and folds to unknown.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Collapse unset UTH sources and map Grok to llm without a hard case
e43ac6f 
Persist only automated|manual|llm so snapshot/flag/unmapped rows share
one unset key and do not split into duplicate unknown report rows.
Detect a GrokAnnotation* union via Product prefix so llm works when
that variant exists without naming a case that may be absent from
spam.rtf.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Note that unset UTH source becomes reportJson unknown
a0178ea 
Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Coalesce leftover UTH source tokens with unset
f0dd4fd 
Treat stored unknown/other/unavailable as unset so earlier daily rows
do not split aggregates. Also match GrokAnnotation on the union class
name when the value is not a Product.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Rewrite latest-as-of UTH rows to a persistable source token
4185ebf 
Group-by already coalesced leftover tokens; the kept row still carried
the raw string. Normalize after reduce so month assembly always sees
automated|manual|llm or unset, including single-row groups.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Clamp UTH source before daily label counts
50dddaf 
Count keys used the raw last-apply source. Leftover tokens that persist
to unset could split a label, then collapse at write and drop counts.
Normalize in actionAgg and the count key so one persistable token is
counted once.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Normalize UTH source tokens before persist and report
d19e74b 
Trim and lowercase so Automated / ' llm ' do not miss the allowlist,
split aggregates, or render as unknown.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Avoid Scrooge .copy when rewriting latest-as-of source
bafda60 
UthDailyPostLabel is constructed with named args everywhere else in
this lane. Reconstruct the row the same way so source rewrite does not
depend on a case-class copy method the IDL compiler may not emit.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Write month post-label source with named Scrooge fields
b29d203 
Positional UthPostLabelAggregate(label, days, source) depends on field
order. Use named args and persistToken so the MH row matches the daily
rewrite path.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Dedupe UTH reportJson post labels that clamp to the same source
6d0d4b7 
Leftover month rows stored as unset vs unknown/other collapse to the
same report token. Keep one row per (label, source), taking the larger
post count so the same posts are not shown twice.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Sort UTH reportJson post labels by label and source
58445ef 
groupBy after source dedupe left row order hash-dependent. Sort so
the same month aggregate always emits a stable reportJson list.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Tighten Grok union name match for UTH llm source
418b20c 
Match the last simple class segment with startsWith(GrokAnnotation)
instead of contains(), so an unrelated name cannot be classified llm.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Leave tweet-flag UTH rows with an explicit unset source
e3b92eb 
Flag inputs have no SafetyLabelSource. Set source = None at both daily
and backfill writes so a future thrift default cannot invent provenance.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Merge leftover UTH post-label days instead of taking max posts
4bdb34e 
max(posts) dropped disjoint leftover days after source clamp. Union
days and keep the larger carried/removed per day, matching month
assembly, then sum.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Avoid Map.flatMap when merging UTH leftover source rows
90a0711 
groupBy returns a Map; flatMap there expects key/value pairs and can
drop or fail to type the report rows. Convert toList first.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Fix Grok class-name split: $ is regex end-of-string
5d842d5 
String.split('$') does not split inner-class names. Take the segment
after the last '.' or '$' and also check getName so binary names like
SafetyLabelSource$GrokAnnotationAction still map to llm.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Pattern-match UTH leftover source groups instead of _1._1
fb5ece6 
Nested tuple accessors can silently pick the wrong slot if the group
shape changes. Destructure (name, source) and day rows by case.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Close postLabelSource and look up about/effect by raw id
1a097b1 
postLabelSource was missing its function closer, so formatPercentage
was nested inside it. Post-label about/effect must use the raw label
id, matching the account path; display name stays on label only.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Drop older asOf UTH rows when source is rewritten
089a909 
latestAsOf grouped by persistable source, so an earlier unset row
could survive next to a later sourced rewrite of the same user/day/label.
Keep every source from the latest asOf only.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Skip trailing synthetic $ when matching UTH llm source
988af0c 
lastIndexOf('$') on GrokAnnotationAction$ (Scala module suffix) left
an empty segment and missed llm. Take the last non-empty . / $ piece.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Keep event source when an unset snapshot apply is later
0425294 
mergeActionAgg took the later apply's source even when that row was
an unset snapshot duplicate, wiping BotMaker/Tool/Grok provenance.
Last apply still wins the timestamp; fall back to the other persistable
source when the later row is unset.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Pattern-match UTH post-label leftover group keys
6a8944e 
groupBy still used t._1 / t._2 after raw was added as a third
tuple field. Name the display name and source in the key, same as
the leftover flatMap.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Skip getSimpleName when detecting UTH llm source
8db548b 
Class.getSimpleName can throw InternalError on Scala $ names and
would fail the daily job for an unmapped SafetyLabelSource. getName
plus the last non-empty segment already yields the simple class name.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Pattern-match leftover UTH month day counts
d98fe35 
Same-day leftover rows still used best._4 / best._5 after source
was added to the tuple. Take carried/removed by pattern match.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@cursoragent @Pitchfork-and-Torch
Only inherit source from an unset snapshot gap-fill
7b6ca47 
mergeActionAgg fell back to an earlier persistable source for any
later unset apply, so a later unmapped event inherited BotMaker/Tool
provenance. Thread isSnapshot and fall back only for snapshot rows.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
@kiri234-code

kiri234-code commented Aug 22, 2026

Copy link
Copy Markdown

Thank you for carrying coarse label-source provenance through the Under the Hood reporting path. This appears to address the reporting gap discussed in Issue #39.

I would like to clarify one important point about the scope of this change.

I have a real-world case where a post for which I voluntarily applied a content warning before publication appears in my Under the Hood report as NSFW_HIGH_PRECISION. The report shows one such post out of 73 total posts, with no account-level labels currently present.

This PR adds source to UthDailyPostLabel and UthPostLabelAggregate, but I do not see a corresponding change to safety-label-user-agg. Am I correct that this PR changes only how post-label provenance is reported, and does not change how those post labels are counted toward account-level NSFW labels?

If account aggregation evaluates NSFW_HIGH_PRECISION by label type without considering its source, an appropriately author-labeled post could still contribute to an account-level restriction in the same way as content that was left unlabeled and subsequently detected or reported.

Could you please clarify:

  1. Whether an author-applied content warning is represented as ToolAction / manual, or whether it reaches this reporting path without a mapped SafetyLabelSource and therefore appears as unknown.
  2. Whether safety-label-user-agg distinguishes author-applied labels from automated detections, reports, or X manual enforcement.
  3. If it does not, whether preserving an author_applied category—or otherwise making source information available to account aggregation—should be addressed in this PR or tracked separately.

The provenance added by this PR is valuable for transparency, but the most important practical question for affected users is whether responsible self-labeling can itself contribute to account-level visibility restrictions.

@Pitchfork-and-Torch

Pitchfork-and-Torch commented Aug 24, 2026

Copy link
Copy Markdown
Author

@kiri234-code You are reading the scope correctly.

This PR only changes the Under the Hood reporting path (under-the-hood/). It adds a coarse source on UthDailyPostLabel / UthPostLabelAggregate and the reportJson tokens automated | manual | llm | unknown. It does not edit safety-label-user-agg/ and it does not change how post labels are counted toward account-level labels.

On the three questions, from the published dump:

  1. Author-applied content warnings are not a separate author_applied token. UTH maps SafetyLabelSource.ToolAction to manual, BotMakerAction to automated, and a Grok/LLM union member (GrokAnnotation*) to llm. Snapshot, tweet-flag, unmapped, and missing sources stay unset and clamp to unknown in the report. Whether a pre-publish content warning is stored as ToolAction vs a tweet-flag / unmapped snapshot is not proven in this dump, so a self-labeled post can show up as manual or unknown depending on how that event was written.

  2. safety-label-user-agg does not distinguish source. The processor fetches baseTweetSafetyLabelMap and keeps labels.keys (label types only). Default rules trigger on NSFW_HIGH_PRECISION / SOFT_NSFW by name, then count matching posts in a window. No SafetyLabelSource check.

  3. Making source available to account aggregation (or an author_applied category) should be a separate PR. This one is the reporting gap in issue Under the Hood does not carry label-source provenance into the report, contrary to README:411 #39. Expanding it into serving/agg would mix classes.

One post labeled NSFW_HIGH_PRECISION with no account-level labels is consistent with the default agg windows (account NSFW_HIGH_PRECISION wants 3 matching posts in a 5-post window). UTH can still show that one post.

xAI owns whether self-labeling should be excluded from account restrictions. This PR does not claim to change that.

@kiri234-code

kiri234-code commented Aug 25, 2026

Copy link
Copy Markdown

@Pitchfork-and-Torch
Thank you very much for the detailed and clear explanation.

This confirms the distinction I wanted to understand: this PR affects only the Under the Hood reporting path, while the published account aggregation logic currently evaluates label types without checking their source.

I also understand that the published code does not establish whether an author-applied warning is written as ToolAction / manual, or enters through a tweet flag or unmapped path and becomes unknown. Therefore, whether self-labeling produces the relevant post label remains a separate unresolved question.

Your explanation of the default 3-in-5 aggregation window also clarifies why one NSFW_HIGH_PRECISION post can appear in Under the Hood without producing an account-level label.

This answers my questions about the scope of the PR and the behavior visible in the published code. I appreciate you taking the time to examine each point so carefully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Under the Hood does not carry label-source provenance into the report, contrary to README:411

3 participants

@Pitchfork-and-Torch @kiri234-code @cursoragent