Skip to content

feat(billing): track E2B and Daytona Function sandbox usage - #7184

Open
BillLeoutsakosvl346 wants to merge 3 commits into
stagingfrom
feat/e2b-daytona-usage-billing
Open

feat(billing): track E2B and Daytona Function sandbox usage#7184
BillLeoutsakosvl346 wants to merge 3 commits into
stagingfrom
feat/e2b-daytona-usage-billing

Conversation

@BillLeoutsakosvl346

@BillLeoutsakosvl346 BillLeoutsakosvl346 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • meter remote workflow Function code and shell sandboxes for E2B and Daytona
  • calculate provider runtime cost in-process and emit it as ordinary Function output.cost
  • reuse the existing trace/span cost summary and workflow usage-ledger pipeline used by hosted-key tools
  • attribute direct Function cost to its Function span; fold nested Function calls into Agent or Pi toolCost
  • keep local JavaScript, custom Function tools, non-workflow calls, and sandboxProfile: mothership unmetered

Trace-native billing flow

E2B/Daytona elapsed resource usage
  -> Function output.cost
  -> direct Function span or Agent/Pi toolCost
  -> existing cost summary
  -> existing workflow logger and usage ledger

There is no sandbox-specific outbox, reconciliation worker, billing-period machinery, database schema, or new ledger path.

Runtime begins immediately before provider creation and ends when Function cleanup starts. Pricing uses the provider-effective TTL as a cap, the configured production multiplier once, and eight-decimal final rounding. Cleanup time is Sim infrastructure overhead.

Outcome policy

Billable completed sandbox outcomes:

  • successful code and shell execution
  • returned user-code exceptions
  • returned nonzero shell exits
  • workspace export failures after the sandbox already completed
  • every provider-completed Function retry attempt

Uncharged outcomes:

  • provider creation or execution throws
  • sandbox/provider crashes
  • provider-reported timeout or E2B lifetime limit
  • cancellation
  • provisioning, corrupt-output, or sandbox-file collection failure
  • an enclosing Agent or Pi provider/memory failure before the parent produces its authoritative cost output

The parent-failure rule intentionally matches hosted-key trace collection: nested tool cost is published through the completed parent span, not billed independently from partial child execution.

Pricing and scope

  • E2B: 2 vCPU and 4 GiB RAM
  • Daytona: 2 vCPU, 4 GiB RAM, and 10 GiB provisioned disk
  • provider list rates are multiplied by the existing COST_MULTIPLIER
  • provider credits, free allowances, and negotiated discounts are intentionally excluded
  • no Mothership-profile, local-JavaScript, custom Function-tool, or historical-backfill change

Accepted limitations

  • provider allocations that never return a usable result are not charged
  • process crashes can lose usage
  • nested Agent/Pi Function costs are aggregated into the parent rather than itemized per call
  • if an Agent/Pi parent fails before publishing its final output, nested Function cost is not charged
  • persisted trace UI visibility continues to follow the existing per-span display policy

Validation

Latest focused run after rebasing onto staging:

  • 439 tests passed across remote-sandbox conformance, Function execution, Function transformation, and shared tool dispatch
  • apps/sim type-check passed
  • apps/sim lint passed
  • API validation passed
  • tool-request and monorepo boundary checks passed
  • git diff --check passed

Ticket: https://sim-ai.slack.com/archives/C093DF8MA21/p1787866922654329

@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 29, 2026 3:01am

Request Review

@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR meters remote E2B and Daytona Function sandboxes and propagates chargeable user-code failure costs through Function, Agent, Pi, retry, trace, and workflow-cost accounting.

  • Adds provider-specific sandbox pricing and resource snapshots.
  • Carries trusted sandbox cost through failed Function executions and retry aggregation.
  • Reconciles sandbox charges through existing workflow trace and usage-ledger accounting.
  • Leaves workflow-scoped calls with missing workflow identifiers reachable as unmetered remote executions.

Confidence Score: 4/5

The PR is not yet safe to merge because workflow-scoped remote Function calls with missing workflow identifiers can still create unmetered E2B or Daytona sandboxes.

The displayed reply, whose author name is blank in the supplied thread data, claims incomplete workflow attribution now fails closed, but current HEAD still derives meterUsage solely from the optional workflowId and forwards false to all remote execution paths without rejecting incomplete attribution.

Files Needing Attention: apps/sim/lib/function-execution/execute-request.ts

Important Files Changed

Filename Overview
apps/sim/lib/function-execution/execute-request.ts Introduces the central remote-sandbox metering and failure-cost flow, but still gates metering on optional workflowId without enforcing complete workflow attribution.
apps/sim/lib/execution/remote-sandbox/index.ts Coordinates sandbox lifecycle and attaches provider-priced usage to chargeable terminal results.
apps/sim/executor/execution/block-executor.ts Aggregates trusted Function costs across retries and preserves them in successful or failed block logs.
apps/sim/executor/handlers/function/function-handler.ts Transfers trusted costs from Function tool results to returned outputs and thrown errors.
apps/sim/providers/cost-policy.ts Incorporates Function sandbox tool costs into provider cost accounting.
apps/sim/lib/billing/sandbox-pricing.ts Defines frozen provider resource pricing, TTL capping, multiplier application, and billed-cost rounding.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Function, Agent, or Pi call] --> B{Remote E2B or Daytona sandbox?}
  B -- No --> C[Existing unmetered execution]
  B -- Yes --> D{workflowId present and standard tool?}
  D -- No --> E[Remote execution without metering]
  D -- Yes --> F[Snapshot sandbox pricing]
  F --> G[Execute code or shell]
  G --> H{Terminal outcome}
  H -- User-code exception or nonzero exit --> I[Attach trusted cost]
  H -- Success --> I
  H -- Timeout, cancellation, or provider failure --> J[No user charge]
  I --> K[Aggregate retries and tool consumers]
  K --> L[Trace cost and workflow usage ledger]
Loading

Reviews (3): Last reviewed commit: "fix(billing): charge function user-code ..." | Re-trigger Greptile

Comment thread apps/sim/lib/function-execution/execute-request.ts Outdated
Comment thread apps/sim/lib/billing/sandbox-usage-outbox.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 17 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/execution/remote-sandbox/index.ts Outdated
Comment thread apps/sim/lib/function-execution/execute-request.ts Outdated
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile-apps @cubic-dev-ai please re-review the latest changes on this PR. The prior findings have been addressed or resolved with the documented scope decision, and the branch is updated with current staging.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 28, 2026

Copy link
Copy Markdown

@greptile-apps @cubic-dev-ai please re-review the latest changes on this PR. The prior findings have been addressed or resolved with the documented scope decision, and the branch is updated with current staging.

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 22 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/function-execution/execute-request.ts Outdated
Comment thread apps/sim/lib/execution/remote-sandbox/index.ts Outdated
Comment thread apps/sim/lib/billing/sandbox-usage-outbox.ts Outdated
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai please re-review the latest changes in dd7aa1c. The three findings from the prior run have been addressed with focused E2B/Daytona, outbox, and Function response regressions, and all threads are resolved.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 28, 2026

Copy link
Copy Markdown

@cubic-dev-ai please re-review the latest changes in dd7aa1c. The three findings from the prior run have been addressed with focused E2B/Daytona, outbox, and Function response regressions, and all threads are resolved.

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 22 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/billing/sandbox-usage-outbox.ts Outdated
Comment thread apps/sim/lib/execution/remote-sandbox/index.ts Outdated
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai please re-review the latest changes in a3a6dd2. Both latest sandbox billing findings are fixed, focused regressions pass across E2B and Daytona, and the full requested validation set is green.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 28, 2026

Copy link
Copy Markdown

@cubic-dev-ai please re-review the latest changes in a3a6dd2. Both latest sandbox billing findings are fixed, focused regressions pass across E2B and Daytona, and the full requested validation set is green.

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 22 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

Implemented the sandbox durability follow-up in c88112edf2.

Review fixes included:

  • Persist a V2 sandbox.usage.finalize intent before any E2B/Daytona create call, then correlate the provider sandbox through E2B metadata or Daytona name/labels.
  • Distinguish definitive create rejection from ambiguous timeout/transport/conflict/rate-limit/server failures. Ambiguous creates return non-retryable 503 sandbox_create_uncertain; user code never runs while identity is uncertain.
  • Bound intent/attachment DB admission, provider recovery, and termination. Pre-expiry provider outages defer for 60 seconds without consuming outbox attempts.
  • Recover ambiguous creates by correlation. Known sandboxes are terminated and billed once; known identities past TTL bill exactly to frozen TTL without provider access. Identities that can never be confirmed are never charged, remain on hourly non-consuming reconciliation, and emit structured operator alerts.
  • Persist ranked terminal evidence (provider_confirmed > not_found_observed > provider_expiry), retry terminal patches, and reload before pricing so concurrent live cleanup wins.
  • Preserve immutable V1 pricing replay while rejecting non-positive multipliers for new intents.
  • Coordinate intent admission and cycle close through the payer subscription lock; a close waits for outstanding old-period sandbox events, while post-close admission refreshes only the effective period for the same payer.

Focused validation:

  • 374 tests passed across sandbox pricing/finalization, cycle close, core outbox, E2B/Daytona conformance, and Function execution responses.
  • apps/sim type-check passed.
  • apps/sim lint passed.
  • API validation audit passed.
  • Monorepo boundary check passed.

Policy decisions remain unchanged: unresolved provider identities are no-charge; Daytona account allowances/credits are ignored in favor of configured list-rate pricing; Mothership scope is unchanged.

Out-of-scope follow-ups identified for separate work:

  1. General long-running workflow settlement across billing-period close (broader than sandbox intent coordination).
  2. Internal log-detail partial totals while independently ledgered costs are still settling.

@cubic-dev-ai please re-review the latest commit, especially the durable pre-create intent, correlation recovery, terminal evidence precedence, and billing-period lock coordination.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 28, 2026

Copy link
Copy Markdown

Implemented the sandbox durability follow-up in c88112edf2.

Review fixes included:

  • Persist a V2 sandbox.usage.finalize intent before any E2B/Daytona create call, then correlate the provider sandbox through E2B metadata or Daytona name/labels.
  • Distinguish definitive create rejection from ambiguous timeout/transport/conflict/rate-limit/server failures. Ambiguous creates return non-retryable 503 sandbox_create_uncertain; user code never runs while identity is uncertain.
    ...

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/billing/cycle-close.ts Outdated
Comment thread apps/sim/lib/billing/sandbox-usage-outbox.ts Outdated
@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from c88112e to e48e74f Compare August 28, 2026 16:42

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/function-execution/execute-request.ts Outdated
@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from e48e74f to 3125336 Compare August 28, 2026 18:28
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

Implemented the complete sandbox lifecycle review follow-up in 3125336, rebased onto staging at b094b4d.

Finding-by-finding:

  1. Durable creation boundary: added a V3 monotonic state machine (intent admitted → create dispatched → provider attached → cleanup requested → terminal/no-charge). Local failures before dispatch preserve the original error; every no-handle failure after dispatch is an uncertain non-retryable 503 and is recovered by correlation.
  2. Crash recovery: intent-only crashes complete no-charge, dispatched creates are searched while live, recovered sandboxes are terminated and billed once, and identities still unconfirmed at frozen TTL complete as auditable identity_unconfirmed no-charge.
  3. Provider bounds: Daytona now bounds the full create lifecycle, uses one absolute timeout across lookup+delete, and observes cancellation before late delete. E2B receives the lifecycle signal/request timeout and freezes the SDK-equivalent whole-second TTL.
  4. Cleanup truth: every metered live-handle kill is preceded by a durable cleanup_requested checkpoint. Failed checkpoints skip direct kill; successful cleanup records provider_confirmed evidence. Weak not-found/expiry evidence waits briefly for stronger live confirmation.
  5. Ledger durability: finalization row-locks and reloads the latest payload before pricing; usage insertion and ledgerRecordedAt are atomic and idempotent by provider+sandbox ID. Infrastructure failures defer without consuming attempts; malformed/semantic payloads retain finite dead-letter behavior.
  6. Handler lease safety: provider and database work share a 60-second sub-lease with abort checks, leaving margin before the generic 90-second outbox handler timeout.
  7. Billing-period coordination: intent admission, ledger insertion, and cycle close serialize on the payer subscription. Null-marker and initialized rollover paths preserve adjacent elapsed-period writers, exact adjacent threshold mismatches hand off to cycle close, unrelated mismatches remain errors, and cycle close revalidates its ledger snapshot under lock.
  8. Actor/payer attribution: trusted executor actorUserId is forwarded separately from attributedUserId, forged body identity is ignored, valid actor/workspace-payer splits succeed, and incomplete metered attribution fails before provider creation.

Validation after the final staging rebase:

  • 483 focused Vitest tests across the 10 requested suites
  • apps/sim type-check
  • apps/sim full Biome lint
  • API validation
  • tool-request boundary
  • monorepo boundaries
  • git diff --check

Policy remains unchanged: an unresolved provider identity is no-charge; Daytona allowances/credits are ignored; pricing and multiplier behavior are unchanged; Mothership, local JavaScript, and custom Function-tool scope are unchanged. The broader long-workflow settlement and internal terminal-log partial-total concerns remain separate follow-up work.

@cubic-dev-ai please re-review the new head 3125336.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 28, 2026

Copy link
Copy Markdown

Implemented the complete sandbox lifecycle review follow-up in 3125336, rebased onto staging at b094b4d.

Finding-by-finding:

  1. Durable creation boundary: added a V3 monotonic state machine (intent admitted → create dispatched → provider attached → cleanup requested → terminal/no-charge). Local failures before dispatch preserve the original error; every no-handle failure after dispatch is an uncertain non-retryable 503 and is recovered by correlation.
    ...

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 29 files

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/execution/remote-sandbox/daytona.ts Outdated
Comment thread apps/sim/lib/billing/sandbox-usage-outbox.ts Outdated
@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from d1413fb to 204e028 Compare August 28, 2026 21:28
@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from 204e028 to 843a194 Compare August 28, 2026 21:29
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

Final delivery head is 843a194ea5, rebased onto current staging at 836b87f5c4.

The only post-review-request adjustment was replacing a banned crypto.randomUUID() string inside a test fixture; CI's check:utils audit now passes. No production behavior changed. The 394 focused tests, type-check, lint, API validation, utils audit, tool-request boundary, monorepo boundaries, and diff check all pass on the rebased head.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 25 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from 843a194 to 5b4cb21 Compare August 28, 2026 22:46
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

Pushed 5b4cb21569 after rebasing onto current staging.

This is a focused correction to the simplified v1 implementation:

  • Ambiguous admission acknowledgement now retains the deterministic provider+sandbox event ID, starts cleanup, and best-effort records infrastructure_error instead of leaving a committed event to bill to TTL.
  • Metered code and shell execution race cancellation after durable admission, and usage is released at cleanup start without waiting for provider deletion to settle.
  • Standard-versus-custom Function classification is stamped from the registered tool ID, so forged Agent/Pi arguments cannot bypass billing.
  • Workflow cost_total reconciliation is monotonic, so a base-only resume cannot erase externally ledgered Mothership/model spend.

Daytona continues to use the full provisioned 10 GiB disk at list rate. Provider credits, free allowances, and negotiated discounts remain intentionally excluded. The known-provider-ID guarantee, accepted v1 limitations, schemas, APIs, dependencies, and Mothership/local/custom-tool scope are unchanged.

Validation passed: 560 tests across 10 focused/adjacent suites, type-check, lint, API validation, utilities audit, tool-request boundaries, monorepo boundaries, and git diff --check. Two read-only audits found no remaining blocker, high, or medium issue.

@cubic-dev-ai please review the focused follow-up commit 5b4cb21569.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 28, 2026

Copy link
Copy Markdown

Pushed 5b4cb21569 after rebasing onto current staging.

This is a focused correction to the simplified v1 implementation:

  • Ambiguous admission acknowledgement now retains the deterministic provider+sandbox event ID, starts cleanup, and best-effort records infrastructure_error instead of leaving a committed event to bill to TTL.
    ...

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 27 files

Confidence score: 4/5

  • In apps/sim/lib/logs/execution/logger.test.ts, costTotalUpdate filters SQL template values to strings/numbers while the assertions expect workflowExecutionLogs.costTotal in params, creating a test expectation mismatch; align the helper or assertions with the actual parameter representation.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/logs/execution/logger.test.ts">

<violation number="1" location="apps/sim/lib/logs/execution/logger.test.ts:1050">
P2: The `costTotalUpdate` helper filters the sql template's `values` to string/number, but the assertions expect `'workflowExecutionLogs.costTotal'` to appear in `params`. The interpolated `workflowExecutionLogs.costTotal` in the `sql\`GREATEST(COALESCE(${...}, 0), ...)\`` template is a drizzle Column object (its empty slot is visible in the asserted `text`), so the filter removes it and the actual `params` are only `['1.705']`/`['1.205']`. Either the assertions never match (failing tests) or, if drizzle happened to inline the column, the params contract the helper claims is wrong. Align the helper and assertions: assert the column reference is carried as a non-string `value` (or its rendered SQL text) rather than as a string `param`.</violation>
</file>

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/logs/execution/logger.test.ts Outdated
@icecrasher321

Copy link
Copy Markdown
Collaborator

@BillLeoutsakosvl346 we should probably put this under a new usage log source? Is that already the case? What is this under right now?

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@BillLeoutsakosvl346 we should probably put this under a new usage log source? Is that already the case? What is this under right now?

Right now it’s under workflow as a tool called Code sandbox, with E2B/Daytona details in the metadata. That keeps it included in the workflow’s total cost. We could add a separate sandbox source, but that would require a migration and some billing/reporting updates.

@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from 5b4cb21 to 6fc6977 Compare August 29, 2026 01:56
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

Rebuilt this PR from current staging around the existing trace-cost pipeline.

What changed:

  • Removed the sandbox-specific outbox, reconciliation, billing-period, and ledger machinery.
  • Successful remote workflow Function code/shell runs now emit provider runtime cost as function_execute output.cost.
  • Direct Function spans use the existing workflow tool-cost path; Agent already folds it into toolCost; Pi now has the matching small accumulator.
  • Standard Function vs custom-tool metering is stamped from the registered tool identity.
  • E2B and Daytona use frozen list-rate resource pricing, the production multiplier, eight-decimal rounding, and provider-effective TTL caps.
  • Local JavaScript, custom Function tools, non-workflow calls, and mothership-profile sandboxes remain excluded.
  • Failed, timed-out, cancelled, and discarded retry attempts remain uncharged, matching hosted-key semantics.

Scope is now 20 files, +602/-50, with no schema, outbox, logger, usage-ledger, or billing-period changes.

Validation:

  • 697 focused tests passed after rebasing onto staging.
  • Type-check, lint, API validation, tool-request boundary, monorepo boundaries, and diff-check passed.
  • Browser acceptance passed for E2B and Daytona: each successful Function run emitted a cost and appeared under the existing Function cost line; a failed Daytona run emitted no sandbox charge.

Accepted limitations remain: no charge when a provider never returns an allocation ID or the process crashes, cleanup is platform overhead, failed attempts are uncharged, Agent/Pi nested Function costs are folded into the parent toolCost, and per-span trace cost visibility follows existing UI policy.

@cubic-dev-ai please review the rewritten trace-native implementation.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 29, 2026

Copy link
Copy Markdown

Rebuilt this PR from current staging around the existing trace-cost pipeline.

What changed:

  • Removed the sandbox-specific outbox, reconciliation, billing-period, and ledger machinery.
  • Successful remote workflow Function code/shell runs now emit provider runtime cost as function_execute output.cost.
    ...

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 20 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile-apps @cubic-dev-ai please re-review the latest commit 546b19e. Remote Function code exceptions and nonzero shell exits are now charged through the existing trace-cost pipeline, including retries and direct Function, Agent, and Pi consumers. Provider/sandbox failures, timeouts, and cancellation remain uncharged. Focused validation: 587 tests passed, plus type-check, lint, API validation, tool-boundary, monorepo boundaries, and diff-check.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 29, 2026

Copy link
Copy Markdown

@greptile-apps @cubic-dev-ai please re-review the latest commit 546b19e. Remote Function code exceptions and nonzero shell exits are now charged through the existing trace-cost pipeline, including retries and direct Function, Agent, and Pi consumers. Provider/sandbox failures, timeouts, and cancellation remain uncharged. Focused validation: 587 tests passed, plus type-check, lint, API validation, tool-boundary, monorepo boundaries, and diff-check.

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 31 files

Confidence score: 3/5

  • apps/sim/lib/function-execution/execute-request.ts and apps/sim/executor/handlers/pi/pi-handler.ts can lose provider runtime or billed Function-call costs when export, backend, memory persistence, or retry handling fails, causing undercharging and incomplete trace/error billing; preserve and propagate the cost through these failure paths.
  • apps/sim/lib/execution/remote-sandbox/index.ts misclassifies E2B provider-limit failures as billable shell or user-code failures, which can charge users for provider-side limits; classify the limit before assigning billableResult while retaining the failure response.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/function-execution/execute-request.ts">

<violation number="1" location="apps/sim/lib/function-execution/execute-request.ts:1828">
P2: When remote code completes but workspace-file export fails, the response drops `sandboxCost`, so the existing trace-cost pipeline cannot charge the provider runtime. Thread the cost through `exportFailure` and include it in every export-error response.</violation>
</file>

<file name="apps/sim/executor/handlers/pi/pi-handler.ts">

<violation number="1" location="apps/sim/executor/handlers/pi/pi-handler.ts:537">
P2: When a Local Pi run executes a billed Function call and then the backend or memory persistence fails, `runPi` throws before exposing `functionToolCost`, so the error and any retry/final block log lose that charge. Preserve the accumulated cost on the thrown error before rethrowing, as the Function handler does.</violation>
</file>

<file name="apps/sim/lib/execution/remote-sandbox/index.ts">

<violation number="1" location="apps/sim/lib/execution/remote-sandbox/index.ts:660">
P2: When E2B reaches its provider limit, this branch treats `E2BProviderLimitError` as a billable user-code exception. Exclude provider-limit errors from `billableResult` while still returning the failure response.</violation>

<violation number="2" location="apps/sim/lib/execution/remote-sandbox/index.ts:800">
P2: When E2B reports its provider limit through the sentinel stderr and exit code, this branch bills it as an ordinary shell failure. Classify the provider-limit result before assigning `billableResult` so provider failures remain uncharged.</violation>
</file>

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/lib/function-execution/execute-request.ts
Comment thread apps/sim/executor/handlers/pi/pi-handler.ts
Comment thread apps/sim/lib/execution/remote-sandbox/index.ts Outdated
Comment thread apps/sim/lib/execution/remote-sandbox/index.ts Outdated
@BillLeoutsakosvl346
BillLeoutsakosvl346 force-pushed the feat/e2b-daytona-usage-billing branch from 546b19e to 97c0c8a Compare August 29, 2026 03:01
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai please re-review the latest commit 97c0c8a. Completed sandbox outcomes now retain cost through workspace export failures, E2B provider lifetime-limit results are typed and remain uncharged for both code and shell, and the Pi parent-failure behavior is documented as the intentional hosted-key-compatible completion boundary. The four focused suites passed 439 tests, and type-check, lint, API validation, tool-boundary, monorepo boundaries, and diff-check are green.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 29, 2026

Copy link
Copy Markdown

@cubic-dev-ai please re-review the latest commit 97c0c8a. Completed sandbox outcomes now retain cost through workspace export failures, E2B provider lifetime-limit results are typed and remain uncharged for both code and shell, and the Pi parent-failure behavior is documented as the intentional hosted-key-compatible completion boundary. The four focused suites passed 439 tests, and type-check, lint, API validation, tool-boundary, monorepo boundaries, and diff-check are green.

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 31 files

Confidence score: 3/5

  • In apps/sim/executor/execution/block-executor.ts, a successful Function sandbox followed by a post-processing exception can lose the trusted returned cost because the catch reads cost only from the later error; preserve and attach the Function output’s cost to keep execution accounting accurate.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/executor/execution/block-executor.ts">

<violation number="1" location="apps/sim/executor/execution/block-executor.ts:671">
P2: When a Function sandbox succeeds but block post-processing throws, this catch reads cost only from the later error. Retain the returned Function output's trusted cost and attach it to post-processing errors so completed sandbox usage is not under-billed.</violation>
</file>

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Fix all with cubic | Re-trigger cubic

return softOutput
}

const trustedExecutionCost = readTrustedExecutionCost(error)

@cubic-dev-ai cubic-dev-ai Bot Aug 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a Function sandbox succeeds but block post-processing throws, this catch reads cost only from the later error. Retain the returned Function output's trusted cost and attach it to post-processing errors so completed sandbox usage is not under-billed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/executor/execution/block-executor.ts, line 671:

<comment>When a Function sandbox succeeds but block post-processing throws, this catch reads cost only from the later error. Retain the returned Function output's trusted cost and attach it to post-processing errors so completed sandbox usage is not under-billed.</comment>

<file context>
@@ -620,8 +668,10 @@ export class BlockExecutor {
       return softOutput
     }
 
+    const trustedExecutionCost = readTrustedExecutionCost(error)
     const errorOutput: NormalizedBlockOutput = {
       error: errorMessage,
</file context>
Fix with cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants