feat: route to the site's content store by benpeter · Pull Request #263 · adobe/da-universal

benpeter · 2026-08-03T11:02:58Z

requires #261

Description

before

one store, hardcoded: admin.da.live.
a site on the hlx6 source bus keeps its content on api.aem.live instead.
so on a migrated site

GET served da-admin's pre-migration copy, or the empty template at HTTP 200
POST wrote that copy back to da-admin, where it is not served
/Media/Holiday.PNG 404'd (daCtx.sourcePath lowercases the whole path, the source bus keeps
directory and extension case)

now

a read does both lookups at once. a write fires only the second.

GET {HLX_ADMIN}/ping/{org}/{site} answers sites in the Fastly edge dictionary in
~37ms with x-api-upgrade-available: true. that means source bus,
and the base is built as {AEM_API}/{org}/sites/{site}/source. helix-api-service refuses a source
url naming another site (src/contentproxy/source/utils.js). no other base is legal for this site.

x-api-upgrade-available is usable as a "yes" but not as a "no". helix-admin sets it when the
resolved config has content.source.url under https://api.aem.live/
(helix-admin/src/index.js:100), so its absence covers

a legacy site
a config that would not resolve, and
a site that does not exist

so a read takes the fast answer when the store replies 200, or 401/403.
on any other status it waits for the config read.
so a wrong yes costs one 404 and cannot produce the empty template.

GET {AEM_API}/{org}/sites/{site}/sidekick reads contentSourceUrl.

sidekick lookup says	result
url under `{AEM_API}/`	read and write that url
url under `https://content.da.live/`	read and write `env.DA_ADMIN`
401 or 403	that status, no store touched
404, 5xx, timeout, unparseable	503, no store touched

needs code:read, answers for legacy sites too (one config service behind both stores).

a write reads only the config: one request where 460ms does not matter, and a wrong
store cannot be reverted.

per store

source bus reads the raw body and types it from the path extension. it parses no form data
da-admin takes the data form part, unchanged
source-bus paths keep directory and extension case with the stem lowercased. da-admin gets
daCtx.sourcePath

only 404 from a store means absent. 401, 403, 429, 5xx pass through
(they used to fall through to the empty template at HTTP 200)

a 401 or 403 on an html GET is answered with the da:401 shell the authorbus extension matches
to refetch /gimme_cookie and refresh. a 503 has no aue:system:ab meta, so the recovery cannot
start. a store that throws is a 503 too. it used to reach withCorsHeaders, which reads
response.headers: an opaque 500 without CORS.

considered but dropped:

If-Match on a write. only the source bus sets an etag on a read, and the editor is given the
connection uri once, at page load. UES posts back to it per edit
(universal-editor-service-plugin-da/src/io.ts:29,52) and returns updates[], so the iframe is
not reloaded and the etag is stale after the first save. a write response has no etag either, so
it cannot be refreshed
a query param on the urn:adobe:aue:system:ab meta content. it does round-trip through UES, so
a read could name its store there for a write to compare against. the one use was a 409 when they
disagreed, which needs content.source.url to change between the read and the save. to not make
it more complex, this is relying on authors being hands-off during migration

Related Issue

Unblocks adobe/da-live#1066. Supersedes #242.

How Has This Been Tested

41 checks driving the handlers against api.aem.live, admin.hlx.page and admin.da.live, on a
scratch site, objects read back and deleted.

/ping answered the enrolled site in 31ms and the legacy site in 434ms, so the fast yes is
the edge and its absence is an origin request
an AEM_API that is unset, scheme-less or has a trailing space answers 503 on a real /ping
yes. it used to build undefined/org/sites/..., which threw out of worker.fetch
an image only in the source bus, 404 on content.da.live and on aem.page: was a 404 through
getHandler, now 200 on ue.da.live, on preview.da.live and with ?quick-edit.
that is #1066's 404
6 saves of one page: was 201, 412, 412, 412 with If-Match on the read's etag. now 6× 201,
and the sixth is what is stored
the multipart envelope stored its boundary lines as the page at 201, now the document
an expired token was 503 "please retry", now 401 with the da:401 shell. 403 passes through
a store that throws was an opaque 500 without CORS, now 503 with CORS (in workerd)
/Media/LiveProbe.PNG was a 404, now 200
legacy read and save unchanged, still the data form part

image reads race the store against *.aem.page, which predates this. store at 200 wins, then
the published copy at 200, then the store's own 503. that last one was the proxy's 404, claiming
the image did not exist.

the shipped sidekick does not rely on api.aem.live/.../sidekick answering for legacy sites
(aem-sidekick/src/extension/project.js:245 picks the host off /ping), so a 404 fallback to
admin.hlx.page/sidekick/.../config.json would be needed if that changes.

covers daCtx.sourcePath, the /source/... url handed to env.daadmin.fetch across GET/HEAD/POST, and a 415 for a multipart File that is not text/html. red on main, where / reads /.html and /page.html reads /page.html.html. Relates to #256.

sourcePath comes from the filename rather than ext, so / reads /index.html and /page.html stops reading /page.html.html. ext is unchanged, so the compose vs raw proxy split at da-admin.js:93 holds. drops six reader-less daCtx fields. daSourcePost now 415s a File whose declared type is not text/html, which would otherwise corrupt a binary body once the write lands on the key GET reads, and returns a response instead of undefined so withCorsHeaders cannot throw. Relates to #256.

4 red: text/html with a charset param is refused, octet-stream and text/plain are allowed through onto the key GET reads. the two case rows pass in node because undici normalizes File.type; they are workerd regression guards.

workerd preserves a part's declared type verbatim and reports '' when the client declared none, so the check normalizes before comparing and no longer allows octet-stream or text/plain onto the key GET reads.

3 red. the html serializer rewrites any body, so posting to /Media/Logo.PNG, a string part, or /sheet.json currently writes html onto the canonical key that GET reads. on main those went to a doubled key nothing reads.

ext is html iff sourcePath ends .html, so this covers every non-html target and refuses nothing the universal editor produces. main sent those writes to a doubled key; sourcePath sends them to the key GET reads.

the non-form fallthrough asserted only instanceof Response, so a 200 passed. the part-type tests targeted a non-html path, where the path check now answers first. adds the urlencoded string-part path, which had no test.

drops the intermediate object. on a duplicated param this takes the first value where fromEntries took the last; only 'local' is acted on, at ue/scaffold.js:36.

the workerd empty-type behaviour it described is in the acfa57e commit message.

the four call sites had the same construction in three spellings. org, site and sourcePath drop out of three destructures as a result.

the sidekick config names the store and 404s when config resolution failed, so it can answer "unknown" where /ping reports legacy.

answers sourcebus with the store base url, legacy, or unknown. a config that could not be resolved lands on unknown rather than being reported as legacy.

the source bus keeps directory and extension case and reads a raw body; da-admin lowercases the whole path and reads a data form part.

each store builds its own url and its own write body, since neither parses the other's shape.

a source-bus read carries its etag so the write is conditional; anything a stamp cannot be trusted to say is not trusted.

carries the store and, on the source bus, the etag the read returned, so the write can be made conditional on it.

503 when the source is unresolved, only 404 means absent, and a UE read stamps the connection uri with the store and etag it read.

a throw escapes into withCorsHeaders, which reads response.headers and throws again.

503 when the content source is unresolved, since guessing serves a migrated site its stale copy. only 404 means the document is absent. a UE read stamps the connection uri with the store and etag it read, which the write that follows uses as its precondition. the write path routes here too and gains that precondition in the next commit.

a source-bus save carries the etag its read returned, so a wrong-store or stale write is refused by the store rather than landing. 409 when the read and the save find different stores.

the stamp the read left carries the etag, so a save conditioned on it cannot land on a document the author never saw: a wrong store or a changed page is refused by the store with 412. where the stamp and a fresh lookup name different stores the site moved while the page was open, which is answered 409.

the base is used verbatim as a store url and the author token goes with it. found by a mutation that swapped startsWith for includes and survived.

an image falls through to the published copy on *.aem.page, since the handlers already race the two and an image cannot become a write. mp4 is not raced, so its refusal reaches the caller.

the editor keeps the stamp it was served at page load and posts back to it for every edit, so a precondition pinned to a version lands the first save and refuses the rest.

the editor keeps the connection uri it was served at page load and posts back to it for each edit, so an If-Match on the read's etag lands the first save and 412s the rest. the source bus sets no etag on a write response either, so nothing can refresh it. the stamp now says only which store the read used and whether it found a document. store identity is still carried, by the 409 when the stamp and a fresh lookup disagree.

a previewed page is one worker request per image, each looking the store up again at ~460ms. writes never reuse, and an unresolved answer is never kept.

a previewed page is one worker request per relative image src, each looking the store up again. the sidekick config is no-store and ~460ms, so 8 identical lookups spend 3.8s of origin time. reads reuse an answer for 10s; a write always asks, and an unresolved answer is never kept.

a stale answer would let the 409 miss a site that moved stores inside the reuse window, which is the one case that overwrites a live page. found by a surviving mutation.

the lookup moves to api.aem.live/{org}/sites/{site}/sidekick behind AEM_API. it needs code:read, which every role that can save already has, and it answers for legacy sites too since both stores read the same config service. gone with it: the connection-uri marker and the 409 it fed, which caught only a migration mid-session and those are managed; the write preconditions, which no read could refresh; and the 10s lookup reuse, since nothing else on this path caches and the config owner sends no-cache upstream and no-store down.

…ers 503 reporting an expired session as unresolved answers a retryable 503, so the client never re-authenticates and the da:401 recovery the authorbus extension has never fires. and a throw from either store's fetch escapes into withCorsHeaders, which reads response.headers and turns it into an opaque 500 with no CORS.

benpeter · 2026-08-03T11:40:32Z

+ * could not be reached, and when `source.kind` is unauthorized or unknown
+ */
+async function readSource(env, daCtx, init) {
+  const config = resolveContentSource(env, daCtx);


await only if needed, clear answer from `/ping/ wins

benpeter · 2026-08-03T11:42:01Z

+
+    // the payload is settled, so the only question left is where it goes. A write is the one
+    // operation a wrong guess cannot be walked back from.
+    const source = await resolveContentSource(env, daCtx);


no /ping on write, the slower lookup doesn't matter for one request, and a wrong store here can't be undone

benpeter · 2026-08-03T11:50:06Z

-      response = aemProxyRes.value;
+    if (storeRes?.status === 200) {
+      response = storeRes;
+    } else if (aemRes?.status === 200) {


published 200 beats store 5xx on the read path (same in head.js)

# Conflicts: # src/responses/index.js # src/routes/da-admin.js # test/routes/da-admin.test.js

benpeter · 2026-08-03T12:12:36Z

+
+  return {
+    url: new URL(`/source/${org}/${site}${sourcePath}`, env.DA_ADMIN),
+    fetch: (input, init) => env.daadmin.fetch(input, init),


a service binding's fetch from the binding throws Illegal invocation in workerd, so it
has to be called on env.daadmin

tripodsan · 2026-08-06T08:20:37Z

+    ['/folder/content', '/folder/content.html', 'appends .html when the request had no extension'],
+    ['/', '/index.html', 'names the root document index.html'],
+    ['/sub-folder/', '/sub-folder/index.html', 'names a directory index'],
+    ['/Media/Holiday.PNG', '/Media/holiday.PNG', 'lowercases the stem, keeps directory and extension case'],


Suggested change

['/Media/Holiday.PNG', '/Media/holiday.PNG', 'lowercases the stem, keeps directory and extension case'],

['/Media/Holiday.PNG', '/media/holiday.png', 'lowercases the stem, keeps directory and extension case'],

tripodsan · 2026-08-06T08:20:54Z

+    ['/', '/index.html', 'names the root document index.html'],
+    ['/sub-folder/', '/sub-folder/index.html', 'names a directory index'],
+    ['/Media/Holiday.PNG', '/Media/holiday.PNG', 'lowercases the stem, keeps directory and extension case'],
+    ['/A/B/c.JSON', '/A/B/c.JSON', 'keeps every directory segment as requested'],


Suggested change

['/A/B/c.JSON', '/A/B/c.JSON', 'keeps every directory segment as requested'],

['/A/B/c.JSON', '/a/b/c.json', 'keeps every directory segment as requested'],

tripodsan · 2026-08-06T08:21:04Z

+    ['/sub-folder/', '/sub-folder/index.html', 'names a directory index'],
+    ['/Media/Holiday.PNG', '/Media/holiday.PNG', 'lowercases the stem, keeps directory and extension case'],
+    ['/A/B/c.JSON', '/A/B/c.JSON', 'keeps every directory segment as requested'],
+    ['/Sub-Folder/', '/Sub-Folder/index.html', 'keeps directory case on a directory index'],


Suggested change

['/Sub-Folder/', '/Sub-Folder/index.html', 'keeps directory case on a directory index'],

['/Sub-Folder/', '/sub-folder/index.html', 'keeps directory case on a directory index'],

tripodsan · 2026-08-06T08:22:57Z

+export function sourceBusPath({ path, sourcePath }) {
+  const dirEnd = path.lastIndexOf('/');
+  const base = sourcePath.slice(sourcePath.lastIndexOf('/') + 1);
+  const baseDot = base.lastIndexOf('.');
+  const requestedDot = path.lastIndexOf('.');
+  // an extension the request carried keeps its case; the `.html` we appended does not have one
+  const ext = requestedDot > dirEnd ? path.slice(requestedDot) : base.slice(baseDot);
+  return `${path.slice(0, dirEnd)}/${base.slice(0, baseDot)}${ext}`;
+}


we should also sanitize the folder names. see https://github.com/adobe/helix-api-service/pull/352

extensions are also lowercased

tripodsan · 2026-08-06T08:30:14Z

+    if (response?.status === 200) return { source: fast, response };
+    // a refusal is about the token, not about which store, so the config read would repeat it
+    if (response?.status === 401 || response?.status === 403) return { source: fast, response };


why do you fall through other errors? like 500, or 400, ... 429 ?

Suggested change

if (response?.status === 200) return { source: fast, response };

// a refusal is about the token, not about which store, so the config read would repeat it

if (response?.status === 401 || response?.status === 403) return { source: fast, response };

if (response && response.status !== 404) return { source: fast, response };

tripodsan · 2026-08-06T08:30:53Z

+    console.warn(`503 ${store.url}, the store could not be reached: ${e.name}: ${e.message}`);
+    return undefined;


why? why are exceptions "swallowed" ? isn't that, well, a exception?

tripodsan · 2026-08-06T08:35:59Z

+    if (response.status !== 200) return undefined;
+    if (response.headers.get(UPGRADE_HEADER) !== 'true') return undefined;
+  } catch (e) {
+    return undefined;


why? if we can't determine the state, we can't just ignore it. then, the request should be aborted.

tripodsan · 2026-08-06T08:36:22Z

+  try {
+    url = new URL(`/ping/${org}/${site}`, env.HLX_ADMIN);
+  } catch (e) {
+    return undefined;


this should never happen, so ignoring the exception is wrong

tripodsan · 2026-08-06T08:37:21Z

+  try {
+    url = new URL(`/${org}/sites/${site}/sidekick`, api);
+  } catch (e) {
+    return unknown(org, site, `AEM_API is not a url: ${e.message}`);


same... ignoring the exception is wrong. the url must not be invalid

tripodsan · 2026-08-06T08:38:18Z

+    return unknown(org, site, 'no org or site in the request');
+  }
+
+  const api = env.AEM_API?.replace(/\/$/, '');


why not ensure that env.AEM_API never has a trailing slash?

tripodsan · 2026-08-06T08:38:57Z

+  try {
+    response = await fetch(url, { headers, signal: AbortSignal.timeout(TIMEOUT_MS) });
+  } catch (e) {
+    return unknown(org, site, `${url} failed with ${e.name}: ${e.message}`);


why ignoring the exception?

tripodsan · 2026-08-06T08:39:38Z

+  try {
+    config = await response.json();
+  } catch (e) {
+    return unknown(org, site, `${url} did not answer json: ${e.message}`);


why ignoring an error?

tripodsan · 2026-08-06T08:41:16Z

+ * with the store base url, `legacy`, `unauthorized` with the status the API gave, or `unknown`
+ * with the reason it could not be answered
+ */
+export default async function resolveContentSource(env, daCtx) {


why is this needed at all? since you are not really interested in the actual content source, but just if it's source bus. which is already reflected via the upgrade header

it's needed because no upgrade header doesn't reliably mean "not on sourcebus" RN, see PR description.
that's also what makes the PR so broad (asking sidekick, and swallowing exceptions on the way, because for the question it's trying to answer they only mean 'we can't tell' [which content store] and 503 in that case)

the alternative I guess would be to make /ping more reliable, what this PR does is probably an over-complicated workaround for /ping answering 200 always (or we accept edge cases like: 1) config load blips, ping still says 200, no upgrade header, then 2) saves go to wrong store)

if that's the wrong approach, I can close this. turning to draft for now

tripodsan

I don't really like this PR.

too many places where exceptions are swallowed, leading to side effects
I do not understand why the sidekick config is loaded. this is indirectly handled via /ping
I do not understand why the stores have "write" related code

benpeter added 30 commits August 1, 2026 14:18

test: pin store urls and the binary post gate

72a14e0

covers daCtx.sourcePath, the /source/... url handed to env.daadmin.fetch across GET/HEAD/POST, and a 415 for a multipart File that is not text/html. red on main, where / reads /.html and /page.html reads /page.html.html. Relates to #256.

test: pin the post gate against charset and binary types

458bba7

4 red: text/html with a charset param is refused, octet-stream and text/plain are allowed through onto the key GET reads. the two case rows pass in node because undici normalizes File.type; they are workerd regression guards.

fix: accept only text/html on a file post, ignoring charset and case

acfa57e

workerd preserves a part's declared type verbatim and reports '' when the client declared none, so the check normalizes before comparing and no longer allows octet-stream or text/plain onto the key GET reads.

test: refuse a post that does not address an html document

9d5d4d0

3 red. the html serializer rewrites any body, so posting to /Media/Logo.PNG, a string part, or /sheet.json currently writes html onto the canonical key that GET reads. on main those went to a doubled key nothing reads.

fix: refuse a post that does not address an html document

532b67a

ext is html iff sourcePath ends .html, so this covers every non-html target and refuses nothing the universal editor produces. main sent those writes to a doubled key; sourcePath sends them to the key GET reads.

test: pin the 415 contract and drop the duplicated type matrices

48cb632

the non-form fallthrough asserted only instanceof Response, so a 200 passed. the part-type tests targeted a non-html path, where the path check now answers first. adds the urlencoded string-part path, which had no test.

refactor: read ue-service with searchParams.get

b66a96b

drops the intermediate object. on a duplicated param this takes the first value where fromEntries took the last; only 'local' is acted on, at ue/scaffold.js:36.

docs: drop the isHtmlPostType jsdoc

b0cfa3c

the workerd empty-type behaviour it described is in the acfa57e commit message.

refactor: build the da-admin source url in one place

1af5bcc

the four call sites had the same construction in three spellings. org, site and sourcePath drop out of three destructures as a result.

test: pin how the content source is resolved

be3a892

the sidekick config names the store and 404s when config resolution failed, so it can answer "unknown" where /ping reports legacy.

feat: resolve the content source from the sidekick config

e53df66

answers sourcebus with the store base url, legacy, or unknown. a config that could not be resolved lands on unknown rather than being reported as legacy.

test: pin each store's url shape and write body

065e201

the source bus keeps directory and extension case and reads a raw body; da-admin lowercases the whole path and reads a data form part.

feat: add the store adapter and the source-bus path case

6471fa0

each store builds its own url and its own write body, since neither parses the other's shape.

test: pin the stamp that links a read to its write

f8db1d1

a source-bus read carries its etag so the write is conditional; anything a stamp cannot be trusted to say is not trusted.

feat: add the source stamp

a80ee1d

carries the store and, on the source bus, the etag the read returned, so the write can be made conditional on it.

test: pin routed reads and the stamp they leave

3cfecf1

503 when the source is unresolved, only 404 means absent, and a UE read stamps the connection uri with the store and etag it read.

test: pin that an unusable admin host answers unknown

b1a8c90

a throw escapes into withCorsHeaders, which reads response.headers and throws again.

test: pin conditional writes and the store-moved refusal

0f90abb

a source-bus save carries the etag its read returned, so a wrong-store or stale write is refused by the store rather than landing. 409 when the read and the save find different stores.

test: pin that a store url must open the source, not appear inside it

6ff6120

the base is used verbatim as a store url and the author token goes with it. found by a mutation that swapped startsWith for includes and survived.

test: pin what a media read does when the source is unresolved

da9016f

an image falls through to the published copy on *.aem.page, since the handlers already race the two and an image cannot become a write. mp4 is not raced, so its refusal reaches the caller.

test: pin that a UE session can save more than once

02d19e4

the editor keeps the stamp it was served at page load and posts back to it for every edit, so a precondition pinned to a version lands the first save and refuses the rest.

test: pin reusing a lookup across a page's image reads

28146a5

a previewed page is one worker request per image, each looking the store up again at ~460ms. writes never reuse, and an unresolved answer is never kept.

test: pin that a write never reuses a stored lookup

5b098f4

a stale answer would let the 409 miss a site that moved stores inside the reuse window, which is the one case that overwrites a live page. found by a surviving mutation.

benpeter marked this pull request as ready for review August 3, 2026 11:26

benpeter requested review from andreituicu, kptdobe, rofe and tripodsan August 3, 2026 11:27

benpeter and others added 2 commits August 3, 2026 13:29

Merge branch 'main' into fix/dactx-source-path

d9def90

Merge branch 'fix/dactx-source-path' into feat/hlx6-source-bus

6467175

benpeter commented Aug 3, 2026

View reviewed changes

Base automatically changed from fix/dactx-source-path to main August 3, 2026 12:07

benpeter marked this pull request as draft August 3, 2026 12:09

Merge remote-tracking branch 'origin/main' into feat/hlx6-source-bus

acee189

# Conflicts: # src/responses/index.js # src/routes/da-admin.js # test/routes/da-admin.test.js

benpeter commented Aug 3, 2026

View reviewed changes

benpeter marked this pull request as ready for review August 3, 2026 12:14

tripodsan reviewed Aug 6, 2026

View reviewed changes

tripodsan requested changes Aug 6, 2026

View reviewed changes

benpeter marked this pull request as draft August 6, 2026 09:55

	['/Media/Holiday.PNG', '/Media/holiday.PNG', 'lowercases the stem, keeps directory and extension case'],
	['/Media/Holiday.PNG', '/media/holiday.png', 'lowercases the stem, keeps directory and extension case'],

	['/A/B/c.JSON', '/A/B/c.JSON', 'keeps every directory segment as requested'],
	['/A/B/c.JSON', '/a/b/c.json', 'keeps every directory segment as requested'],

	['/Sub-Folder/', '/Sub-Folder/index.html', 'keeps directory case on a directory index'],
	['/Sub-Folder/', '/sub-folder/index.html', 'keeps directory case on a directory index'],

		console.warn(`503 ${store.url}, the store could not be reached: ${e.name}: ${e.message}`);
		return undefined;

Uh oh!

Conversation

benpeter commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

before

now

Related Issue

How Has This Been Tested

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benpeter Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tripodsan Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tripodsan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

benpeter commented Aug 3, 2026 •

edited

Loading

benpeter Aug 3, 2026 •

edited

Loading

tripodsan Aug 6, 2026 •

edited

Loading