Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ Start at the [docs index](docs/) for the full map. Quick links:
- [Configuration](docs/configuration.md) — Config options, env-var overrides, tier shortcuts
- [Architecture](docs/architecture.md) — System design, codegen pipeline, plugin model, multi-CSP vision
- [Services Matrix](docs/services-matrix.md) — services, coverage status, boto3 compatibility
- [Plugin API](docs/plugin-api.md) — the stable `ServicePlugin` contract and v1.x stability policy
- [Compatibility Policy](docs/compatibility-policy.md) — what v1.0 guarantees across 1.x, and what it explicitly does not
- [Fidelity Manifest](docs/fidelity-manifest.md) — per-operation tiers: how much to trust any given call
- [Plugin API](docs/plugin-api.md) — the in-tree `ServicePlugin` contract for contributors
- [Roadmap](docs/roadmap.md) — Phased plan toward multi-CSP support
- [FAQ](docs/faq.md) / [Troubleshooting](docs/troubleshooting.md) — Common questions and errors
- [Contributing](docs/contributing.md) — Development setup, adding new services
Expand Down
5 changes: 5 additions & 0 deletions changes/unreleased/Added-20260809-230000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Added
body: A published compatibility policy (docs/compatibility-policy.md) stating what v1.0 guarantees across the 1.x line — config keys, environment variables, the CLI, admin API response keys, fidelity tier names, and the response shape of hand-verified operations covered by the boto3 compatibility suite — what it explicitly does not guarantee, and the deprecation procedure that precedes any removal
time: 2026-08-09T23:00:00.000000+09:00
custom:
Issue: "129"
9 changes: 8 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,14 @@ This directory holds DevCloud's technical documentation. Use the map below to ju
- **[Configuration](configuration.md)** — YAML options, env-var overrides, tier shortcuts
- **[Architecture](architecture.md)** — system design, plugin model, codegen pipeline, multi-CSP vision
- **[Roadmap](roadmap.md)** — phased plan toward multi-CSP support
- **[Services Matrix](services-matrix.md)** — 101 services, coverage status, boto3 pass rate
- **[Services Matrix](services-matrix.md)** — 104 services, coverage status, boto3 pass rate

## What you can rely on

- **[Compatibility Policy](compatibility-policy.md)** — what v1.0 guarantees across 1.x, what it explicitly does not, and how deprecation works
- **[Fidelity Manifest](fidelity-manifest.md)** — per-operation tiers: how much to trust any given call
- **[CRUD Engine](crud-engine.md)** — how engine-served operations behave, and where they stop
- **[Plugin API](plugin-api.md)** — the in-tree `ServicePlugin` contract for contributors

## Per-service references

Expand Down
131 changes: 131 additions & 0 deletions docs/compatibility-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Compatibility Policy

What DevCloud **v1.0** promises, and what it deliberately does not.

This document covers the surfaces you touch as a *user* of DevCloud — the config file, the
environment, the CLI, the admin API, and the AWS wire protocol. For the in-tree Go contract
that service implementations are written against, see
[plugin-api.md](plugin-api.md#api-stability); that surface lives under `internal/` and is not
importable from another module.

Versions follow [Semantic Versioning](https://semver.org). "Across 1.x" below means every
release from v1.0.0 up to but not including v2.0.0.

## Guaranteed across 1.x

### Configuration file

These keys keep their name, type and meaning. New keys may be added; existing ones are not
removed or repurposed. Defined in [`internal/config/config.go`](../internal/config/config.go).

| Key | Type | Meaning |
|---|---|---|
| `server.port` | int | Listen port. Default `4747` when absent or `0`. |
| `services` | map | Presence of the block is authoritative — only the services it lists run. Absent means every registered service runs. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Honor empty services blocks before guaranteeing this behavior

For a config containing services: {}, this promise is false: YAML creates an empty map, but Config.Service checks len(c.Services) > 0, so it treats that explicitly present block exactly like an absent block and enables every registered service. This can unexpectedly start all 104 services when a user relies on the newly published authoritative-block guarantee; distinguish a nil map from an empty non-nil map, or narrow the policy.

Useful? React with 👍 / 👎.

| `services.<id>.enabled` | bool | Whether that service starts. |
| `services.<id>.data_dir` | string | Where that service stores data. |
| `admin.enabled` | bool | Whether the admin API is served. Default `false`. |
| `logging.level` | string | Log level. |
| `logging.format` | string | Log format. |

### Environment variables

| Variable | Meaning |
|---|---|
| `DEVCLOUD_PORT` | Overrides `server.port`. |
| `DEVCLOUD_SERVICES` | Service filter. `all`, a comma-separated list of service ids, or the `tier1` / `tier2` / `tier3` shortcuts. Unknown tokens are treated as literal service names. |
| `DEVCLOUD_DATA_DIR` | Base directory; each service stores under `<base>/<id>`. Overrides `data_dir`. |

Environment overrides config file, and that precedence is guaranteed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Make the service environment filter actually override YAML

This precedence guarantee does not hold for DEVCLOUD_SERVICES: with YAML listing only s3 and DEVCLOUD_SERVICES=sqs, Config.Service("sqs") first accepts the environment filter but then rejects sqs because the nonempty YAML map does not list it, leaving no service enabled. The added test masks this case by selecting s3, which is already present in YAML; either make the environment selection authoritative or document that it can only further restrict the YAML block.

Useful? React with 👍 / 👎.


### Command line

`-config <path>` keeps its meaning. With no flag, DevCloud uses `./devcloud.yaml` if present
and the embedded defaults otherwise — zero-config startup keeps working.

### Admin API

Served at `/devcloud/api/` when `admin.enabled: true`. These routes keep responding, and their
JSON responses **only gain fields** — no documented key is removed or repurposed.

| Route | Guaranteed response keys |
|---|---|
| `GET /devcloud/api/services` | array of `id`, `name`, `status`, `resourceCount` |
| `GET /devcloud/api/services/{id}/resources` | array of `type`, `id`, `name` |
| `GET /devcloud/api/logs` | array of `method`, `path`, `status`, `duration`, `timestamp`, `service`; newest first; `?limit=` honoured |
| `GET /devcloud/api/fidelity` | object keyed by service id, each with `modelBacked` and `counts`; `?service=<id>` adds `operations` |

### Fidelity tier names

`hand-verified`, `auto-crud` and `unimplemented` keep the meanings given in
[fidelity-manifest.md](fidelity-manifest.md). The set does not shrink, and a name is never
reused for a different meaning. Every reachable operation carries one — enforced by
`TestFidelityManifestCoverage` in [`cmd/devcloud/fidelity_test.go`](../cmd/devcloud/fidelity_test.go),
which fails the build if an operation is unclassified.

### Wire behaviour — scoped to the compatibility suite

**A hand-verified operation covered by a test in [`tests/compatibility/`](../tests/compatibility/)
keeps its response shape across 1.x.**

That suite — 775 tests driving real boto3 clients — *is* the guarantee. It runs in CI on every
push and again against the tagged commit before a release publishes, so the promise is enforced
by a failing build rather than by review discipline. If a response shape you depend on is not
Comment on lines +68 to +73

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Enforce the complete response shape promised by the policy

The compatibility suite does not currently enforce an operation's full response shape. For example, tests/compatibility/test_lambda.py:27-38 covers the hand-verified CreateFunction operation but asserts only FunctionName and FunctionArn; removing fields such as Runtime, Handler, or MemorySize from functionConfig would still leave that test green while breaking users under this newly stated guarantee. The policy must scope stability to fields explicitly asserted by tests, or the suite must snapshot/validate every promised response field before claiming a failing build enforces the guarantee.

Useful? React with 👍 / 👎.

covered there, it is not covered by this policy; adding a test is the way to bring it in scope,
and such contributions are welcome.

## Not guaranteed

Depending on any of the following will break, and breaking it is **not** a major-version event.

- **`auto-crud` response content.** 948 operations are served by the
[generic CRUD engine](crud-engine.md) at fidelity that is deliberately *plausible, not
faithful*: store-backed responses echoing your input plus synthesized ids and ARNs, with no
validation, no cross-resource integrity, no pagination correctness and no business logic.
Their shape and content may change in any release. Use them to wire an SDK up, nothing more.
- **Hand-verified operations with no compatibility test.** Of 4,496 hand-verified operations,
only what the suite covers is promised. The rest are best-effort.
- **Data durability.** Stores are local development stores. Several are in-memory and
per-process; on-disk layouts under `data_dir` may change format between releases without a
migration. Do not treat DevCloud as a database.
- **`unimplemented` → served transitions.** An operation that returns an error today may start
returning a response. This is additive, and ships in a minor release.
- **Service coverage.** New services may be added in a minor release. The 104 services present
at v1.0 are a floor, not a ceiling.
- **Error message wording.** Error *codes* and HTTP status of `unimplemented` operations are
documented in [fidelity-manifest.md](fidelity-manifest.md); the human-readable message text
is not stable.
Comment on lines +95 to +97

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Align guaranteed unimplemented errors with provider responses

This newly guarantees error codes and statuses that the linked fidelity manifest describes incorrectly for many classified operations. For example, IAM classifies AcceptDelegationRequest as unimplemented, but the default branch in internal/services/iam/provider.go returns NotImplemented with HTTP 501, while docs/fidelity-manifest.md:17 says Query services return InvalidAction with HTTP 400; numerous JSON providers have the same 501 behavior. Publishing those values as stable therefore gives v1 users a contract the current binary already violates, so either normalize the providers or document their actual per-service errors before guaranteeing them.

Useful? React with 👍 / 👎.

- **Log output.** Format, levels and wording of server logs are operational, not an API.
- **Everything under `internal/`.** Go forbids importing it from another module, and DevCloud
reserves the right to restructure it freely across 1.x — explicitly including the planned
intermediate representation and `ModelSource` work on the [roadmap](roadmap.md). Internal
churn is not a compatibility event.
Comment on lines +99 to +102

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exempt the stable plugin contract from the internal exclusion

The blanket statement that everything under internal/ may be restructured freely across 1.x directly conflicts with the compatibility contract added to docs/plugin-api.md:130-142, which says the in-tree ServicePlugin, PluginConfig, Response, Resource, and ProtocolType surfaces are stable and require a major bump for breaking changes. Maintainers therefore cannot tell whether changing those types is permitted in a 1.x release; carve that explicitly stable subset out of this exclusion or remove the competing contract.

Useful? React with 👍 / 👎.

- **Behavioural parity with AWS.** No release of DevCloud promises AWS's validation, business
logic, eventual-consistency timing, rate limits, or IAM enforcement. Credentials are accepted
without signature verification.

## Deprecation procedure

Removing anything from the guaranteed list is a **major** version bump. Before that can happen:

1. **Deprecate in a minor release.** The old form keeps working and emits a runtime warning
naming its replacement. The precedent is the `dashboard` → `admin` config rename: the old key
still enables the admin API, warns, and yields to an explicit `admin` block
([`config.go`](../internal/config/config.go)).
2. **Document it** — in the release notes for that version, and here.
3. **Remove no earlier than the next major.** At least one released version must have shipped
the warning.

Silence is not deprecation. A removed key that YAML would otherwise drop without comment is
kept in the parser purely to warn — that is why `auth` still produces a message telling you
SigV4 is not enforced rather than being ignored.

The pre-flight checklist in [release.md](release.md#pre-flight-checklist) makes this a step in
cutting a release, not a thing to remember.

## Reporting a break

If a 1.x release breaks something on the guaranteed list, that is a bug — please
[open an issue](https://github.com/skyoo2003/devcloud/issues) with the DevCloud version and a
reproducing snippet. If it breaks something on the not-guaranteed list, an issue is still
useful: it is evidence for tightening the policy in a future major.
6 changes: 6 additions & 0 deletions docs/plugin-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,12 @@ The registry also exposes `RegisteredServices()` (all registered IDs) and

## API stability

This section is the **in-tree** contract — it constrains contributors writing
service plugins inside this repository. `internal/plugin` cannot be imported
from another Go module, so it is not the promise a *user* of DevCloud depends
on. That is [compatibility-policy.md](compatibility-policy.md), which covers the
config file, environment variables, CLI, admin API and wire behaviour.

Starting at **v1.0**, `ServicePlugin`, `PluginConfig`, `Response`, `Resource`,
and the `ProtocolType` constants are stable within the `v1.x` series:

Expand Down
6 changes: 5 additions & 1 deletion docs/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,11 @@ The rest are only caught here.
with a warning first. The precedent is the `dashboard` → `admin` rename in
[`internal/config/config.go`](../internal/config/config.go): the old key kept working,
emitted a warning, and only then became removable. Removing without that overlap is a
major-version change.
major-version change. The full procedure, and the surfaces it applies to, is
[compatibility-policy.md](compatibility-policy.md).
- [ ] **Compatibility review.** If this release changes anything on the guaranteed list in
[compatibility-policy.md](compatibility-policy.md), it is a major bump — or it is a bug.
Additive change (a new config key, a new response field, a new service) is a minor bump.

## Cutting a release

Expand Down
83 changes: 83 additions & 0 deletions internal/admin/api_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,89 @@ func newTestRegistry(p *mockServicePlugin) *plugin.Registry {
return reg
}

// getJSON issues GET path against h, asserts the status and Content-Type the
// compatibility policy guarantees, and decodes the body into v.
func getJSON(t *testing.T, h http.Handler, path string, v any) {
t.Helper()
w := httptest.NewRecorder()
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
require.Equal(t, http.StatusOK, w.Code, path)
require.Equal(t, "application/json", w.Header().Get("Content-Type"), path)
require.NoError(t, json.NewDecoder(w.Body).Decode(v), path)
}

// surfaceAPI builds an API with one service, one resource and one log entry —
// enough for every guaranteed route to return a non-empty body.
func surfaceAPI(t *testing.T) http.Handler {
t.Helper()
p := &mockServicePlugin{
id: "s3",
name: "Amazon S3",
resources: []plugin.Resource{{Type: "bucket", ID: "my-bucket", Name: "my-bucket"}},
}
lc := NewLogCollector(10)
lc.Add(RequestLog{
Method: "GET", Path: "/s3/my-bucket", Status: 200,
Duration: "1.000ms", Timestamp: time.Now(), Service: "s3",
})
return NewAPI(newTestRegistry(p), lc).Handler()
}

// TestGuaranteedAdminSurface_Collections locks the wire keys of the three
// list-returning routes in docs/compatibility-policy.md.
//
// It decodes into map[string]any deliberately. The other tests in this file
// decode into the internal structs (serviceInfo, RequestLog), so renaming a
// JSON tag renames both sides of the assertion and they stay green while every
// consumer breaks. Asserting key *presence* rather than the whole payload keeps
// additive change — which the policy allows — from failing the build.
func TestGuaranteedAdminSurface_Collections(t *testing.T) {
h := surfaceAPI(t)

for _, tc := range []struct {
route string
keys []string
}{
{"/devcloud/api/services", []string{"id", "name", "status", "resourceCount"}},
{"/devcloud/api/services/s3/resources", []string{"type", "id", "name"}},
{"/devcloud/api/logs", []string{"method", "path", "status", "duration", "timestamp", "service"}},
} {
var got []map[string]any
getJSON(t, h, tc.route, &got)
require.NotEmpty(t, got, "%s returned no entries to check", tc.route)

for _, key := range tc.keys {
if _, ok := got[0][key]; !ok {
t.Errorf("%s: entry is missing guaranteed key %q — guaranteed by docs/compatibility-policy.md",
tc.route, key)
}
}
}
}

// TestGuaranteedAdminSurface_Fidelity locks both shapes of the fidelity route:
// the summary carries counts only, and naming a service adds its operations.
func TestGuaranteedAdminSurface_Fidelity(t *testing.T) {
h := surfaceAPI(t)

var summary map[string]map[string]any
getJSON(t, h, "/devcloud/api/fidelity", &summary)
require.Contains(t, summary, "s3")
for _, key := range []string{"modelBacked", "counts"} {
if _, ok := summary["s3"][key]; !ok {
t.Errorf("/devcloud/api/fidelity: missing guaranteed key %q — guaranteed by docs/compatibility-policy.md", key)
}
}
assert.NotContains(t, summary["s3"], "operations",
"the unfiltered summary must not carry every operation")

var detail map[string]map[string]any
getJSON(t, h, "/devcloud/api/fidelity?service=s3", &detail)
require.Contains(t, detail, "s3")
assert.Contains(t, detail["s3"], "operations",
"?service= must add the per-operation tiers")
}

// TestAPI_Services registers a mock plugin and verifies the
// /devcloud/api/services endpoint returns it.
func TestAPI_Services(t *testing.T) {
Expand Down
72 changes: 72 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,78 @@ func TestParse_AdminKeyWinsOverDeprecated(t *testing.T) {
assert.False(t, cfg.Admin.Enabled, "explicit admin.enabled=false should win over deprecated dashboard.enabled=true")
}

// TestGuaranteedConfigSurface locks the config keys docs/compatibility-policy.md
// promises will keep their name and meaning across 1.x. Adding a key is fine;
// removing or repurposing one of these is a major-version event, and this test
// is what makes that visible instead of silent. The deprecated 'dashboard' and
// removed 'auth' keys are covered by TestParse_DeprecatedDashboardKey and
// TestParse_RemovedAuthKeyWarns.
func TestGuaranteedConfigSurface(t *testing.T) {
isolateEnv(t)

cfg, warnings, err := parse([]byte(`
server:
port: 5555
services:
s3:
enabled: true
data_dir: ./custom/s3
admin:
enabled: true
logging:
level: debug
format: json
`))
require.NoError(t, err)
assert.Empty(t, warnings, "the guaranteed surface must parse without warnings")

// t.Errorf per key, not require: one removed field should not hide the rest.
for _, tc := range []struct {
key string
got any
want any
}{
{"server.port", cfg.Server.Port, 5555},
{"services.<id>.enabled", cfg.Service("s3").Enabled, true},
{"services.<id>.data_dir", cfg.Service("s3").DataDir, "./custom/s3"},
{"services (block is authoritative)", cfg.Service("sqs").Enabled, false},
{"admin.enabled", cfg.Admin.Enabled, true},
{"logging.level", cfg.Logging.Level, "debug"},
{"logging.format", cfg.Logging.Format, "json"},
} {
if tc.got != tc.want {
t.Errorf("%s: got %v, want %v — guaranteed by docs/compatibility-policy.md", tc.key, tc.got, tc.want)
}
}
}

// TestGuaranteedEnvSurface locks the three environment overrides the policy
// guarantees, including their precedence over the config file.
func TestGuaranteedEnvSurface(t *testing.T) {
isolateEnv(t)
t.Setenv("DEVCLOUD_PORT", "6060")
t.Setenv("DEVCLOUD_SERVICES", "s3")
t.Setenv("DEVCLOUD_DATA_DIR", "/tmp/dc")

cfg, _, err := parse([]byte(`
server:
port: 4747
services:
s3:
enabled: true
data_dir: ./custom/s3
sqs:
enabled: true
`))
require.NoError(t, err)

assert.Equal(t, 6060, cfg.Server.Port, "DEVCLOUD_PORT must override server.port")
assert.True(t, cfg.Service("s3").Enabled, "DEVCLOUD_SERVICES must keep the service it names")
assert.False(t, cfg.Service("sqs").Enabled, "DEVCLOUD_SERVICES must filter out services it does not name")
assert.Equal(t, filepath.Join("/tmp/dc", "s3"), cfg.Service("s3").DataDir,
"DEVCLOUD_DATA_DIR must rebase data dirs, overriding data_dir")
}

// TestParse_EmptyData_FillsDefaults verifies that parsing an empty YAML
// payload yields a Config with at least the default server port populated,
// so downstream code sees a usable config rather than a zero-value one.
Expand Down