Skip to content

Reduce main thread and network overhead on high pop servers - #10

Open
ch3rn1k wants to merge 1 commit into
RustyMoose:mainfrom
ch3rn1k:perf/high-pop-overhead
Open

Reduce main thread and network overhead on high pop servers#10
ch3rn1k wants to merge 1 commit into
RustyMoose:mainfrom
ch3rn1k:perf/high-pop-overhead

Conversation

@ch3rn1k

@ch3rn1k ch3rn1k commented Aug 2, 2026

Copy link
Copy Markdown

went looking into what gets expensive when server's full, ended up in three spots: per-player invokes, network counters, and the uploader

hot path

  • player second-stats was registering one InvokeRepeating per player. now theres a single ticker at 100ms that walks a tenth of activePlayerList each tick. still one sample per player per second, just spread out, and no more invoke register/cancel on every connect and disconnect
  • the Message.Type dictionaries (counters and names) are now plain arrays indexed by enum value, so NetWrite.Send isnt hashing anything anymore
  • NetworkUpdateData.Bytes was an int, and write.Length * count was int math too, overflows inside a single report window at high pop so both are long now
  • MetricsTimeStorage.LogTime was doing TryGetValue plus an indexer set, so two lookups on every invoke, RPC and work queue job. now it just keeps a small mutable accumulator instead

uploader

the Queue<string> plus the ever growing StringBuilder is now a ring buffer with an 8MB cap, drops oldest lines once its full. batches capped at 60k chars too so they stay off the LOH. whatever gets dropped shows up in servermetrics.status instead of just vanishing

the actual request moved off the main thread, one background thread, pooled HttpClient, gzip, three attempts, queue bounded to four batches. main thread just grabs a batch and hands over the bytes, roughly 0.07ms even if the endpoint takes a full second to respond. errors still get logged from the main thread since InvokeHandler isnt thread safe

also added a one second coalescing window, otherwise it fires off a ton of tiny requests. on an idle test server that went from 349 requests/min down to 60, same line protocol volume either way (~440KB), but actual bytes on the wire went 127KB down to 85KB

bug fix

OnUnloaded was only destroying the MetricsLogger component, so the GameObject holding the uploader survived harmony.unload. old coroutine kept running after unload, and the new upload thread leaks one per reload. it destroys the whole object now

config

new config key "Compress submitted metrics with gzip", defaults true, gets written into existing configs and can be toggled with reloadcfg. status has an In Flight line now

testing

tested against a local InfluxDB stub on a procedural 1000 map. normal operation, db going down and coming back, unload/reload, disable/enable via config, gzip on and off

buffer and uploader logic also has a reflection based test runner, covers batching, wrap around, overflow accounting, gzip round trip, retries, connection reuse

one thing worth flagging

a batch already handed off to the worker is lost if the db is unreachable, only whats still in the ring buffer survives. can make the worker hold onto failed batches and retry with backoff if thats preferred

build note

LangVersion was set to 14 which needs the .NET 10 SDK, and nothing here actually uses any C# 14 features, so bumped it back to 13. also dropped two Unity module references that arent in the game build anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant