feat(speculation): add selector + selection-limit extensions

Makefile

@@ -364,7 +364,7 @@ local-stovepipe-stop: ## Stop the Stovepipe service
 
 mocks: ## Generate mock files using mockgen
 	@echo "Generating mocks..."
-	@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/enumerator/... ./submitqueue/extension/speculation/dependencylimit/... ./submitqueue/extension/speculation/pathscorer/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
+	@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/enumerator/... ./submitqueue/extension/speculation/dependencylimit/... ./submitqueue/extension/speculation/selector/... ./submitqueue/extension/speculation/selectionlimit/... ./submitqueue/extension/validator/... ./submitqueue/extension/speculation/pathscorer/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
 	@echo "Mocks generated successfully!"
 
 proto: ## Generate protobuf files from .proto definitions

submitqueue/entity/speculation_tree.go

@@ -134,7 +134,7 @@ type SpeculationPathInfo struct {
 // path's identity and its freshly computed predicted-success score. It is the
 // scorer seam's only output — the controller merges scores into the tree by
 // path ID and persists them; tree structure and status never pass through the
-// scorer. Like SpeculationPathDecision, it is ephemeral and never persisted.
+// scorer. Like PathDecision, it is ephemeral and never persisted.
 type PathScore struct {
 	// PathID identifies the scored path (SpeculationPathInfo.ID) within the
 	// tree the scorer was handed.
@@ -143,14 +143,17 @@ type PathScore struct {
 	Score float32
 }
 
-// SpeculationPathDecision is a seam's decision for a single path: the action the
+// PathDecision is a seam's decision for a single path: the action the
 // controller should take for it. It is the output of both the selector (per
 // batch) and the prioritizer (queue-wide), and is not persisted. A seam returns
 // a decision only for the paths it wants to act on; omitted paths are left
-// as-is.
-type SpeculationPathDecision struct {
-	// Path identifies the speculation path the action applies to.
-	Path SpeculationPath
+// as-is. A seam must return at most one decision per path — the controller
+// treats conflicting duplicates as a policy bug, applying the first and
+// logging and skipping the rest.
+type PathDecision struct {
+	// PathID identifies the speculation path the action applies to
+	// (SpeculationPathInfo.ID), within the tree(s) the seam was handed.
+	PathID string
 	// Action is what the controller should do for the path.
 	Action SpeculationPathAction
 }

submitqueue/extension/speculation/selectionlimit/BUILD.bazel

@@ -0,0 +1,8 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["selectionlimit.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/selectionlimit",
+    visibility = ["//visibility:public"],
+)

submitqueue/extension/speculation/selectionlimit/README.md

@@ -0,0 +1,17 @@
+# Speculation Selection Limit
+
+Vendor-agnostic "how much" policy that bounds how many paths a batch may build in parallel.
+
+See the [Speculation RFC](/doc/rfc/submitqueue/speculation.md) for the end-to-end design and how limits fit into the two-layer speculation model.
+
+## Selection Limit
+
+The selection limit is the [selector](../selector)'s companion. The selector decides **which** of a batch's paths are worth building — its ranking over the tree; the selection limit decides **how many** of them may run at once. Keeping "which" and "how much" separate keeps selector logic free of resource accounting and lets the bound scale with build resources without touching that logic.
+
+The value is **signal-driven**, not a fixed constant. Its primary input is the build system's available capacity, but a policy may also weigh historical pass rates, cost budgets, time of day, or an experiment toggle.
+
+Unlike the dependency limit — which the controller holds and applies as an eligibility gate — the selection limit is **injected into the seam that uses it**: the selector is constructed with it and calls it itself, never receiving it as a method parameter. This follows the repo's extension-contract pattern (dependencies injected at the `Factory`), keeps the selector interface limit-free and stable, and lets the limit be swapped independently of selector logic.
+
+## Factory
+
+A per-queue factory returns the limit policy for a queue, following the repo's extension contract. It is handed only the queue identity; the signals a policy weighs — a capacity feed, historical metrics, config — are injected at construction by the integrator in the wiring layer, which is also where the limit is handed to the selector. Computing the limit itself takes no further inputs.

submitqueue/extension/speculation/selectionlimit/fake/BUILD.bazel

@@ -0,0 +1,19 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["fake.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/selectionlimit/fake",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/extension/speculation/selectionlimit:go_default_library"],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["fake_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+    ],
+)

submitqueue/extension/speculation/selectionlimit/fake/fake.go

@@ -0,0 +1,52 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package fake provides a programmable selectionlimit.SelectionLimit for tests
+// and examples. New sets the value returned by Limit; FailWith injects an error
+// on every call. It is intended for examples and tests only, never production.
+package fake
+
+import (
+	"context"
+
+	"github.com/uber/submitqueue/submitqueue/extension/speculation/selectionlimit"
+)
+
+// SelectionLimit is a programmable selectionlimit.SelectionLimit.
+type SelectionLimit struct {
+	limit int
+	err   error
+}
+
+// New returns a fake SelectionLimit whose Limit returns the given value.
+func New(limit int) *SelectionLimit {
+	return &SelectionLimit{limit: limit}
+}
+
+// FailWith makes every Limit call return err.
+func (l *SelectionLimit) FailWith(err error) *SelectionLimit {
+	l.err = err
+	return l
+}
+
+// Limit returns the configured value, or the injected error if FailWith was set.
+func (l *SelectionLimit) Limit(_ context.Context) (int, error) {
+	if l.err != nil {
+		return 0, l.err
+	}
+	return l.limit, nil
+}
+
+// ensure the fake satisfies the interface.
+var _ selectionlimit.SelectionLimit = (*SelectionLimit)(nil)

submitqueue/extension/speculation/selectionlimit/fake/fake_test.go

@@ -0,0 +1,36 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package fake
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestLimit_ReturnsConfiguredValue(t *testing.T) {
+	got, err := New(2).Limit(context.Background())
+	require.NoError(t, err)
+	assert.Equal(t, 2, got)
+}
+
+func TestLimit_FailWith(t *testing.T) {
+	sentinel := errors.New("boom")
+	_, err := New(2).FailWith(sentinel).Limit(context.Background())
+	require.ErrorIs(t, err, sentinel)
+}

submitqueue/extension/speculation/selectionlimit/mock/BUILD.bazel

@@ -0,0 +1,12 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["selectionlimit_mock.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/selectionlimit/mock",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/extension/speculation/selectionlimit:go_default_library",
+        "@org_uber_go_mock//gomock:go_default_library",
+    ],
+)

submitqueue/extension/speculation/selectionlimit/selectionlimit.go

@@ -0,0 +1,57 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package selectionlimit
+
+//go:generate mockgen -source=selectionlimit.go -destination=mock/selectionlimit_mock.go -package=mock
+
+import "context"
+
+// SelectionLimit is the "how much" policy that bounds how many paths a batch may
+// build in parallel.
+//
+// It is the selector's companion: the selector decides *which* of a batch's
+// paths are worth building (its ranking); the selection limit decides *how many*
+// of them may run at once. Separating the two keeps selector logic free of
+// resource accounting and lets the bound scale with build resources without
+// touching that logic.
+//
+// The value is dynamic: it may change between calls, so the selector reads it
+// each pass rather than caching it.
+//
+// Unlike the dependency limit, this limit is injected into the seam that uses it
+// — the selector is constructed with it and calls it itself — never passed as a
+// method parameter, keeping the selector interface limit-free and stable.
+type SelectionLimit interface {
+	// Limit returns the current maximum number of paths a batch may build in
+	// parallel. The selector caps its Promote decisions at this. It takes no
+	// parameters; anything an implementation needs is injected at construction.
+	Limit(ctx context.Context) (int, error)
+}
+
+// Config carries the per-queue identity handed to a Factory. The system knows
+// only the queue name; everything a policy needs to compute the limit (a
+// capacity feed, historical metrics, config) is injected at construction by the
+// integrator.
+type Config struct {
+	// QueueName identifies the queue this SelectionLimit serves.
+	QueueName string
+}
+
+// Factory builds the SelectionLimit for a queue. Implementations are provided by
+// integrators (and tests) and inject whatever signals they need at construction.
+type Factory interface {
+	// For returns the SelectionLimit for the given queue.
+	For(cfg Config) (SelectionLimit, error)
+}

submitqueue/extension/speculation/selector/BUILD.bazel

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["selector.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/selector",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/entity:go_default_library"],
+)

submitqueue/extension/speculation/selector/README.md

@@ -0,0 +1,21 @@
+# Speculation Path Selector
+
+Vendor-agnostic interface for deciding what the orchestrator should do with each path in a batch's enumerated speculation tree.
+
+See the [Speculation RFC](/doc/rfc/submitqueue/speculation.md) for the end-to-end design and how selection fits into the orchestrator pipeline.
+
+## Selector
+
+A selector is the **policy** — the part that decides how aggressively to spend build resources. *Given the candidate paths in the batch's tree and their current status, what should we do with each, right now?* It returns an **action** per path — `Promote` (advance it one stage toward running) or `Cancel`. Strategies span a spectrum: build only the single optimistic path (cheapest — bet on the happy case), build every candidate (maximum parallelism, maximum build cost), or a top-K / budget-bounded subset in between.
+
+The selector decides only where to spend build resources. It does **not** decide merging: a path becomes mergeable when its build passed and its base matches what actually landed, which is deterministic, not a policy choice — so the controller finalizes it on its own.
+
+The controller hands the selector the batch's **speculation tree** directly — the subject it decides over. The controller is the single writer: it reconciles each path's status (candidate, selected, prioritized, building, passed, failed, cancelling, cancelled) from the latest builds and dependency states, and it maps each of the selector's decisions to a status transition — `Promote` → `Selected`, `Cancel` → `Cancelling` (or `Cancelled` when nothing is building) — applied under the tree's optimistic lock and persisted. The selector's only output is decisions, each naming a path by its ID, at most one per path; it **never** writes status. This keeps it a deterministic policy over the tree it is given.
+
+Selection expresses **desire, not admission**: `Selected` means this batch wants to spend a build slot on the path, while `Prioritized` means the queue-wide prioritizer has actually admitted it under the shared build budget and it is cleared to build. A promoted path waits in `Selected` until the prioritizer clears it — with free budget it would move on promptly, under contention it may wait indefinitely or be dropped. That split exists because a selector sees only its own batch and cannot ration a budget shared across the whole queue.
+
+Because it is re-run on every build signal, a selector can start narrow — build the optimistic path first — and widen later, committing more paths only once earlier bets resolve. Returning no action for a path leaves it as-is. Policy parameters — a top-K cap, a build budget, an experiment toggle — are configured when the selector is constructed rather than passed through this contract.
+
+## Factory
+
+A per-queue factory returns the selector for a queue, following the repo's extension contract. It is handed only the queue identity and nothing else; policy knobs — a top-K cap, a build budget, an experiment toggle — are injected at construction by the integrator in the wiring layer, which resolves per-queue settings through `queueconfig`. Selection itself stays config-free.