Skip to content

chunk() returns a sparse array when one event exceeds maxKB, stalling the event queue (regression in 2.23.0) #1309

Description

@AlvYuste
  • analytics-react-native version: 2.24.0 (regression introduced in 2.23.0; 2.21.3 and 2.22.0 unaffected)
  • Integrations versions (if used): @segment/analytics-react-native-plugin-advertising-id, @segment/analytics-react-native-plugin-idfa
  • React Native version: 0.83.10 (Hermes, New Architecture, Expo SDK 55)
  • iOS or Android or both? Both

chunk() in packages/core/src/util.ts can return a sparse array. Since 2.23.0 the upload path iterates its results with for...of, which does not skip holes, so a single oversized event now makes every flush throw and the event queue stops draining.

chunk assigns into an index rather than appending:

if (maxKB !== undefined) {
  rollingKBSize += sizeOf(item);
  if (rollingKBSize >= maxKB) {
    chunks[++currentChunk] = [item];   // on index 0 this skips chunks[0]
    return chunks;
  }
}

When the first item alone is >= maxKB, currentChunk goes 0 -> 1 and chunks[0] is never created. MAX_PAYLOAD_SIZE_IN_KB is 500, so any single queued event serialising to 500KB or more triggers it.

Array.prototype.map preserves the hole, Promise.all resolves it to undefined, and aggregateErrors in packages/core/src/plugins/SegmentDestination.ts then reads result.status off undefined:

const results: BatchResult[] = await Promise.all(batches.map((batch) => this.uploadBatch(batch)));
const aggregation = this.aggregateErrors(results);   // for (const result of results) { switch (result.status)

This was harmless before 2.23.0. 2.21.x used chunkedEvents.map(async (batch) => { ... }) and never iterated the results, so the hole was silently skipped and the oversized event simply never uploaded. 2.23.0 added aggregateErrors with for (const result of results), which made the same sparse array fatal.

Steps to reproduce

chunk and sizeOf are pure, so the sparse array reproduces standalone (both copied verbatim from packages/core/src/util.ts):

const sizeOf = (obj) => (encodeURI(JSON.stringify(obj)).split(/%..|./).length - 1) / 1024;

const chunk = (array, count, maxKB) => {
  if (!array.length || !count) return [];
  let currentChunk = 0, rollingKBSize = 0;
  return array.reduce((chunks, item, index) => {
    if (maxKB !== undefined) {
      rollingKBSize += sizeOf(item);
      if (rollingKBSize >= maxKB) { chunks[++currentChunk] = [item]; return chunks; }
    }
    if (index !== 0 && index % count === 0) { chunks[++currentChunk] = [item]; }
    else { if (chunks[currentChunk] === undefined) chunks[currentChunk] = []; chunks[currentChunk].push(item); }
    return chunks;
  }, []);
};

// one event over MAX_PAYLOAD_SIZE_IN_KB (500), then two normal ones
const big = { messageId: 'a', properties: { blob: 'x'.repeat(520 * 1024) } };
const batches = chunk([big, { messageId: 'b' }, { messageId: 'c' }], 100, 500);

// [ <1 empty item>, [ {messageId:'a'} ], [ {messageId:'b'} ], [ {messageId:'c'} ] ]
//   ^ the hole                            ^ 'b' and 'c' should have shared a batch;
//                                           rollingKBSize is never reset, so they don't
console.log(batches);
console.log(0 in batches);   // false  <-- hole

Promise.all(batches.map((b) => ({ status: 'success', messageIds: [] }))).then((results) => {
  for (const result of results) { void result.status; }   // TypeError
});

In-app, track any event whose serialised size is >= 500KB while it is the only or first entry in the persisted queue, then let a flush policy fire.

Expected behavior

chunk returns a dense array of non-empty batches. An item that alone exceeds maxKB gets its own batch; the server rejects it with a 4xx, default4xxBehavior: 'drop' drops it, and the queue continues to drain.

Actual behavior

errorHandler receives ErrorType.FlushError with Flush failed: TypeError: Cannot read property 'status' of undefined (Hermes wording) on every flush.

Because the throw escapes sendEvents after Promise.all has uploaded the batches but before processUploadResults runs, nothing is ever dequeued. The queue therefore never drains, the same events are re-uploaded on every flush, and the device recovers only when pruneExpiredEvents discards the events at maxTotalBackoffDuration — 12 hours by default. We saw this on roughly 200 devices across both platforms in a single release, each reporting a FlushError every 30 seconds.

There is a second, independent problem in the same function: rollingKBSize is never reset when a new chunk starts. Once the cumulative size crosses maxKB, the size branch fires for every remaining item and each one becomes its own batch, so the count limit becomes unreachable. Measured with 1200 events of about 1KB: 700 batches, 699 of them single-event, against 3 batches once fixed. That is 700 HTTP requests per flush where 3 would do.

Suggested fix

Build the chunks by appending, and reset the accumulator per chunk. This removes the hole, restores the count limit, and keeps an oversized item isolated in its own batch:

export const chunk = <T>(array: T[], count: number, maxKB?: number): T[][] => {
  if (!array.length || !count) {
    return [];
  }

  let rollingKBSize = 0;

  return array.reduce((chunks: T[][], item: T) => {
    const itemKBSize = maxKB === undefined ? 0 : sizeOf(item);
    const currentChunk = chunks[chunks.length - 1];
    const isOverMaxKB = maxKB !== undefined && rollingKBSize + itemKBSize >= maxKB;

    if (currentChunk === undefined || currentChunk.length >= count || isOverMaxKB) {
      rollingKBSize = itemKBSize;
      chunks.push([item]);
      return chunks;
    }

    rollingKBSize += itemKBSize;
    currentChunk.push(item);
    return chunks;
  }, []);
};

Happy to open a PR if that would help.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug fix

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions