Reduce main thread and network overhead on high pop servers - #10
Open
ch3rn1k wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
went looking into what gets expensive when server's full, ended up in three spots: per-player invokes, network counters, and the uploader
hot path
InvokeRepeatingper player. now theres a single ticker at 100ms that walks a tenth ofactivePlayerListeach tick. still one sample per player per second, just spread out, and no more invoke register/cancel on every connect and disconnectMessage.Typedictionaries (counters and names) are now plain arrays indexed by enum value, soNetWrite.Sendisnt hashing anything anymoreNetworkUpdateData.Byteswas anint, andwrite.Length * countwas int math too, overflows inside a single report window at high pop so both arelongnowMetricsTimeStorage.LogTimewas doingTryGetValueplus an indexer set, so two lookups on every invoke, RPC and work queue job. now it just keeps a small mutable accumulator insteaduploader
the
Queue<string>plus the ever growingStringBuilderis now a ring buffer with an 8MB cap, drops oldest lines once its full. batches capped at 60k chars too so they stay off the LOH. whatever gets dropped shows up inservermetrics.statusinstead of just vanishingthe actual request moved off the main thread, one background thread, pooled
HttpClient, gzip, three attempts, queue bounded to four batches. main thread just grabs a batch and hands over the bytes, roughly 0.07ms even if the endpoint takes a full second to respond. errors still get logged from the main thread sinceInvokeHandlerisnt thread safealso added a one second coalescing window, otherwise it fires off a ton of tiny requests. on an idle test server that went from 349 requests/min down to 60, same line protocol volume either way (~440KB), but actual bytes on the wire went 127KB down to 85KB
bug fix
OnUnloadedwas only destroying theMetricsLoggercomponent, so the GameObject holding the uploader survivedharmony.unload. old coroutine kept running after unload, and the new upload thread leaks one per reload. it destroys the whole object nowconfig
new config key
"Compress submitted metrics with gzip", defaults true, gets written into existing configs and can be toggled withreloadcfg.statushas anIn Flightline nowtesting
tested against a local InfluxDB stub on a procedural 1000 map. normal operation, db going down and coming back, unload/reload, disable/enable via config, gzip on and off
buffer and uploader logic also has a reflection based test runner, covers batching, wrap around, overflow accounting, gzip round trip, retries, connection reuse
one thing worth flagging
a batch already handed off to the worker is lost if the db is unreachable, only whats still in the ring buffer survives. can make the worker hold onto failed batches and retry with backoff if thats preferred
build note
LangVersionwas set to 14 which needs the .NET 10 SDK, and nothing here actually uses any C# 14 features, so bumped it back to 13. also dropped two Unity module references that arent in the game build anymore