This is the implementation writeup for Auxilary Services Same Host. The plan it implements is Plan: Notify and Agents on the Site's Own Host.
Where the work is
Seed: branch feat/aux-services-same-host, PR seed#1238. It supersedes #1233 and keeps that PR's commit as its first commit.
SeedInfra: branch aux-services-same-host, PR infrastructure#14. Three commits, applied in order; the third must wait.
Why
Notify and agents each needed their own hostname (notify.<dns>, agents.<dns>, notify.seed.hyper.media, agentic.seed.hyper.media), while the vault already lived at /vault on the site. For anyone running their own node that meant extra DNS records, certificates and env wiring for each service. The self-hosting story was also unclear: run agents only, run three agent servers, or skip notify.
The decisions behind the change:
Notify has no reason to be hosted separately. It now always runs on a site's own origin, and the notify.seed.hyper.media hostname goes away. A site that doesn't run notify uses hyper.media's: https://hyper.media/hm/api/notify.
Agents don't have to share the host. A site may serve agents at /hm/api/agents, but a standalone server stays fully supported. The production agents server stays standalone at agentic.seed.hyper.media.
Links in emails sent before the switch are allowed to break.
Notify's frontend moves into the web app. Notify becomes a light headless Bun service, like the vault and agents.
The reverse proxy routes, not the web app. The web app's Node process is the part that saturates under crawler traffic, so it shouldn't also carry other services' requests.
#1233 had already put enterprise agents on the site, behind the members-only gate, with a service secret so the agents service can read a gated site. It mounted them at /agents/* and gave clients the bare site origin as the server URL. With that base the client posted messages to ${origin}/api/message, which the site routes to the web app's /api/*, not to agents. This change keeps #1233's gating and service secret and replaces the mount.
The new shape
flowchart LR
subgraph site["A site's origin (Caddy / Traefik)"]
proxy[reverse proxy]
web[web app]
notify["notify (Bun)"]
agents["agents (optional)"]
vault["vault /vault"]
daemon[daemon]
end
proxy -- "/hm/api/notify/* + legacy notify paths" --> notify
proxy -- "/hm/api/agents/* (members only on enterprise)" --> agents
proxy -- "/vault/*" --> vault
proxy -- "everything else, incl. /hm/email-notifications" --> web
notify -- gRPC-web --> daemon
agents -- "Seed API + service secret" --> web
standalone["agentic.seed.hyper.media (prod, standalone)"]URL contract
Both services are now named by a base URL:
canonical base | endpoints | legacy form (still accepted) | |
|---|---|---|---|
notify |
|
| origin only: |
agents |
|
| origin only: |
Clients build every URL through one helper per service: @shm/shared/notify-url and agentServerEndpoint in the agents client. A canonical base gets /<endpoint> appended. Anything else is treated as a legacy server.
The notify helper also rewrites the retired hosted hostnames: notify.seed.hyper.media → https://hyper.media/hm/api/notify, notify-dev.seed.hyper.media → https://dev.hyper.media/hm/api/notify. That takes care of every value already stored in desktop settings, the synced vault state and browser IndexedDB.
Discovery
A site's /hm/api/config now returns:
notifyServiceUrl: the canonical notify base, or null.
notifyServiceHost: the notify origin, kept for older apps, which append /hm/api/notifications. The proxy routes those legacy paths to notify on that origin.
agentServerUrl: the site's default agents server, or null. It is never a bare origin.
The vault's sign-in callback hands sites only the notify origin, for the same reason: web builds already deployed on self-hosted sites append legacy paths.
What changed, by area
Agents (agents/, frontend/packages/ui/src/agents/client.ts)
The server registers every endpoint under /hm/api/agents/... in addition to all legacy routes. WebSocket upgrades are accepted on both /hm/api/agents/ws and /agents/ws.
New webhooks get canonical, origin-relative paths. Old webhook URLs keep working.
One client helper, agentServerEndpoint(serverUrl, 'message'|'ws'|'health'|…), picks canonical vs legacy. normalizeAgentServerUrl keeps /hm/api/agents and still strips a bare /agents.
The desktop's local-server health probe understands canonical bases.
From #1233, unchanged: X-Seed-Service-Secret, SEED_SERVICE_SECRET on web, SEED_AGENTS_HM_SERVER_SECRET on agents. The secret is attached only to requests to the agents service's own site, and redirects are followed by hand so the header never leaves that origin.
Docs: hypermedia/agent/operations.md has a new section, "Where the service lives". Six other agent docs are updated.
Notify (frontend/apps/notify, frontend/apps/emails)
No hostname and no pages.
app/main.ts starts a Bun.serve route table (app/routes.ts) and the email loops.
Canonical routes live under /hm/api/notify/*, and the legacy /hm/api/<leaf> paths stay as aliases.
Every route answers the CORS preflight. A wrong method gets a CORS 405; anything else gets a CORS 404.
The three pages are gone.
read-redirect and verify-email are API endpoints that write, then redirect with a relative Location, so the redirect stays on whatever public origin the email pointed at. read-redirect keeps its http/https-only target check.
The settings page is the web app's now.
Email links are built in one place, frontend/apps/emails/notify-links.ts, from the origin of NOTIFY_SERVICE_HOST (falling back to SEED_BASE_URL):
The footer "Unsubscribe" and "Manage notifications" links open <origin>/hm/email-notifications?token=….
The List-Unsubscribe header points at <origin>/hm/api/notify/unsubscribe (RFC 8058 one-click POST). The one-click endpoint is header-only on purpose: link scanners (Outlook Safe Links, Gmail) prefetch GET links in an email's body, which would unsubscribe people silently.
Verify and read-redirect links point at the new endpoints.
Runtime:
better-sqlite3 → bun:sqlite, with the same file (${DATA_DIR}/web-db.sqlite), schema and migrations. A copy of a database the old code wrote (userversion 10, 3,784 inbox rows) opened with no migration, and every read worked.
@connectrpc/connect-node → connect-web, the transport the vault already uses against the daemon.
@sentry/remix → @sentry/bun, with the same env vars.
Removed: Remix, Vite, Tailwind, @shm/ui, react-dom, isbot, superjson, sharp, and the unused image processor.
Build and image:
bun build produces one self-contained build/main.js, which runs even with no node_modules; MJML rendering works inside the bundle.
The image runs that file on oven/bun:1.3.10, as the same uid 1001, on port 3000, with the /data volume. Swapping the image needs no data migration.
Tests:
Database-backed suites (*.bun.test.ts) run under bun test. One of them, routes.bun.test.ts, drives the real Bun.serve table against a temp database: aliases, preflight, 405, read-redirect, verify-email, unsubscribe, 404.
Module-mocking suites stay on vitest, because bun's mocks leak across files. pnpm --filter @shm/notify test runs both.
CI workflows install bun before the notify tests. The integration harness starts notify with bun and checks /hm/api/notify/health.
Docs: hypermedia/apps/notify.md is rewritten.
Clients (@shm/shared, web, desktop, vault, mobile)
@shm/shared/notify-url: normalizeNotifyServiceUrl, notifyEndpointUrl, notifyServiceOrigin, isCanonicalNotifyServiceUrl, NOTIFY_API_PREFIX, HOSTED_NOTIFY_SERVICE_URL. The shared notification service, the public subscribe box, web notifications, the vault, desktop and mobile all go through it.
Web: /hm/email-notifications. The settings page that notify used to render is now a web app route. It talks to <notify base>/email-notif-token.
It is open on members-only sites: whoever holds the email token may not be a member.
It sends Referrer-Policy: no-referrer.
The token is scrubbed from Sentry events, transactions and breadcrumbs (app/sentry-scrub.mjs).
Web: /hm/api/config gains notifyServiceUrl and agentServerUrl, as above.
Defaults are https://hyper.media/hm/api/notify everywhere: vault, the desktop release build, desktop dev builds (the dev twin), the desktop account-settings fallback, and mobile.
Local dev now matches production.
The web dev server (Vite server.proxy, dev only) forwards the notify paths to the notify process on :3060.
Every dev notify value is http://localhost:3000/hm/api/notify, so email links and the settings page share the web origin.
mprocs.yaml sets the notify pane's env.
Ops (ops/, templates, website_deployment.sh)
Enterprise Caddyfile, one site block, no extra hostnames:
@notify: /hm/api/notify, /hm/api/notify/* and the legacy notify paths go to seed-notify:3000. Open, before the sign-in gate; signed requests and email tokens are its credentials.
POST /hm/api/agents/webhooks/* goes to seed-agents:3050. Open, before the gate.
/hm/api/agents and /hm/api/agents/* go to seed-agents:3050 behind import gate (members only, API and WebSocket).
/hm/email-notifications* joins the open paths. It is mirrored by hand in the web app's require-sign-in.ts.
Site (non-enterprise) nodes can run agents now. There is a wizard question, an agents compose profile, a /hm/api/agents route (open, like the site), kvm, data dir, backup, and doctor support. Without agents, the web app keeps the hosted agents service.
Hosted notify is https://hyper.media/hm/api/notify in deploy.ts, all eight home-server templates and website_deployment.sh.
Tests (ops/deploy.test.ts, 276 pass) cover:
route order: open webhook before the gate, gated agents before the catch-all;
the open notify paths;
exactly one site block;
the standard Caddyfile's agents route only when agents is enabled;
site-profile env, dirs and backup.
SeedInfra (seed_infra/core_infra, seed_dns)
e6b36ac coreinfra: codify the hand-applied gateway protections.
Production Traefik had drifted from Terraform. The 2026-08-11 gw-inflight / gw-ratelimit middlewares, today's crawler-throttle router with its crawler-ratelimit middleware, and a newer update_traefik.sh (conditional middlewares on custom domains, certificates only for domains pointing at the box) existed only on the box.
Applying the notify change without this commit would have silently dropped them.
The rendered template now matches the live prod and dev configs, apart from blank lines and machine-generated custom-domain routers. It is prod-only where dev has none of it.
6f08a70 coreinfra: serve notify on the gateway's own origin.
A notifyapi router on hyper.media / dev.hyper.media sends /hm/api/notify[/*] and the legacy paths to the notify container already running on that box. Its priority is 2500: above crawler-throttle at 2000, below block-scanners at 3000. It has no web middlewares.
NOTIFY_SERVICE_HOST on web and notify, and the vault's default, switch to the canonical URL.
The old notify routers stay for now.
f765fc7 DO NOT APPLY until the notify-default desktop release is out: retire notify.seed.hyper.media. It removes the old routers, certificates and both DNS records. Note: the Terraform record keys were swapped; seed_notify_dev named the prod host.
Rollout order
Merge the Seed PR. Release web, notify, vault and agents images; the notify image is the new Bun one.
Apply SeedInfra commits 1 and 2 to dev. Check:
https://dev.hyper.media/hm/api/notify/health
a legacy path such as /hm/api/notifications (POST)
the email-settings page
a test email's links
Apply commits 1 and 2 to prod the same way. Do it only after the new images are live: older images would build wrong URLs from the new env values.
Ship a desktop release; it carries the new notify defaults and the retired-host rewrite.
Once that release has reached users, apply commit 3 (DNS and routers for notify.seed.hyper.media removed).
Enterprise nodes pick up the new Caddyfile on their next deploy. Their notify.<dns> / agents.<dns> DNS records can then be deleted.
Risks and how they're handled
Risk | What was done | What is left |
|---|---|---|
Applying SeedInfra would drop prod's hand-applied Traefik protections (crawler throttle, per-IP limits) | Commit 1 codifies them; the rendered config matches the live one | The systemd vs nohup difference for |
New env values with old images build wrong notify URLs | Rollout order: images first, then SeedInfra commit 2 | Follow the order |
Installed desktops and stored browser values still name | The client helper rewrites retired hosts. Sites advertise the notify origin for older apps, and the proxy routes the legacy paths | Keep the old host until the desktop release is out (commit 3 waits) |
Notify's runtime switch could corrupt or stall notification state | Same database file and migrations, tested against a copy of an old database, route tests against a real database | One first boot on a Docker anonymous volume stalled during database init and didn't reproduce (prod uses a bind mount). Watch the first boot on dev and on prod |
Crawler throttle catching notify traffic |
| None |
Members-only gate blocking desktop, webhooks or unsubscribe | Notify paths and the agents webhook route are open, before the gate. Agents API stays members-only (#1233's decision) | Desktop apps can't use a members-only site's agents without a session. That is by design for now |
Email-settings token leaking |
| None |
Agents microVMs competing with the site on a co-located node | Prod stays standalone. On site nodes agents are opt-in | Lower |
Old email links | Not preserved, by decision | Links sent before the switch return 404 |
Verification
Run on the final branch:
pnpm format:check (whole repo): pass.
pnpm typecheck: pass. agents and vault tsc: pass.
pnpm audit --audit-level=high: pass.
pnpm install --frozen-lockfile: pass.
Unit suites:
web: 635 passed
shared: 1420 passed
client: 708 passed
notify: vitest 23 + bun 82 passed
emails: pass
agents: 574 passed
ops: 276 passed
vault: 298 passed
desktop (subagent run): 1023 passed
Smoke tests: the notify source, the bundle and the PREBUILT=true image all served health, preflight and legacy paths with the daemon unreachable.
Not verified locally:
the notify PREBUILT=false Docker build (local Docker Desktop ran out of disk; CI uses PREBUILT=true)
the web dev proxy against a live notify (only its path matcher was tested)
terraform fmt/validate (terraform isn't installed here)
Next steps
Review and merge seed#1238. Watch CI, especially the notify image build and the integration job that now starts notify with bun.
Dev rollout: release images, then apply infrastructure#14 commits 1–2 to dev. Check:
https://dev.hyper.media/hm/api/notify/health
a desktop and a web inbox
the subscribe box
a test email's links, settings page and List-Unsubscribe
Prod rollout: the same checks, then watch notify's first boot on the new runtime.
Desktop release with the new notify default and the retired-host rewrite. After it has reached users, apply infrastructure#14 commit 3, which retires notify.seed.hyper.media.
Enterprise nodes: redeploy, then delete their notify.<dns> / agents.<dns> DNS records. ee1.botical.com is the one that motivated #1233.
Security follow-up: move the Mailgun SMTP password out of core_infra/main.tf into a variable or secret, and rotate it.
Later cleanup:
Once older apps are gone, remove the legacy notify path aliases and switch the vault's sign-in callback to hand out the canonical base instead of the origin.
Decide whether /hm/api/config.agentServerUrl or the home-document agentServerUrl metadata wins, and whether a site should advertise more than one agents server.
Add a notify profile to the standard site stack for operators who want their own email sending.
Add health checks for notify and agents to monitoring.
Rollout log, 2026-10-14
Landed.
Seed: #1238 (this change; supersedes #1233, which is closed with a note).
Seed fixes found on devnet:
Infra:
The retirement of notify.seed.hyper.media waits on branch retire-notify-host.
How infra applies. Terraform Cloud workspaces are VCS-driven from master: SHM-Core-DEV auto-applies on merge; SHM-Core-PROD plans and waits for a confirm.
What devnet testing found and fixed.
Domain-sync race (infra #15). The first dev apply silently lost the new notifyapi router: update_traefik.sh (running as the systemd unit) read config.yml, spent seconds on DNS checks, then copied its stale copy over Terraform's upload. It now skips a write when the file changed under it. The script also now ships whenever it changes, not only when the config does. Prod's apply must stop the unit first, because the old script is what runs during the upload.
Mail under Bun (seed #1240, #1242, infra #16). Notify mail failed with "Greeting never received". Measured from the dev host:
Bun's implicit TLS to Mailgun on 465 connected 1 of 5 times, and 4 of 5 with SNI.
Node connected 5 of 5.
Bun's STARTTLS on 587 connected 16 of 16.
Notify now uses STARTTLS on 587, like the vault. A welcome email went out from dev afterwards.
Open redirect (seed #1240). read-redirect redirected anyone to any http(s) URL from hyper.media's origin. It now requires a real email token.
Devnet state (dev.hyper.media, dev.agentic).
notify, web, vault, daemon and relay run main.
Notify answers at dev.hyper.media/hm/api/notify (health, CORS preflight, legacy aliases), and on the old notify-dev host until retirement.
Signed notify calls succeed through the canonical base, the bare origin and the old host.
Agents answer on both URL shapes, and both WebSocket paths open.
The email-settings page renders with Referrer-Policy: no-referrer.
Enterprise node following main: ee-main (https://ee-main.hyper.media, droplet 157.245.208.51).
Created from the hosting console with tag dev and main's ops URL, with agents and notify on.
Its auto-update cron is installed, and it already pulled newer :dev images on its own.
One site block, no notify./agents. hostnames.
A headless walkthrough passed: invite → password → profile → signed in and joined → tabs → new document published → comment → /hm/notifications (signed notify call 200) → /hm/agents (health and messages 200 through the gate; WebSocket on /hm/api/agents/ws).
Without a session, the agents API is 401 and webhooks are open. The agents service reads the gated site with the service secret.
Test member: cc-e2e-main-1@example.com (writer), one document and one comment.
Seen and not fixed.
The web client still calls notify on its legacy paths, because the vault's sign-in callback hands out the origin. This is by design until older apps are gone.
The agents client decodes a JSON 401 body as CBOR ("64-bit integer string lengths not supported") instead of showing "Sign in required".
A one-time node:module CORS console error appeared during the enterprise walkthrough. It isn't reproducible on public pages.
ee-test-1 still runs #1233's branch images. Configure it to dev or latest with main's ops URL.
One flaky integration suite on main (a8e0fc5, test daemon exited before ready); unrelated.
External content on enterprise nodes, seed#1245 (2026-10-15).
The problem. An enterprise daemon is isolated, so an embed of another space's document (the hosted Agent Guide) rendered as not found, and the agents service could not resolve it for its system prompt.
The fix. The web app now answers such reads from a mainnet node's public HTTP API: upstream_api_url, default https://hyper.media, "" to turn it off.
Resource, ResourceMetadata and Account fall back when not found, and Query when empty for a single space. Images and files fall back to the upstream node's /ipfs/<cid>.
Only accounts the daemon has never seen go upstream.
The agents service also gets the knowledge-base docs account, so hm://hyper.media/... links resolve.
Verified on ee-main after it updated itself:
A document with an Embed of the guide renders for a member.
The guide opens directly on the site.
The agents container fetches it through the site.
The system-prompt resolver yields the full guide (5,140 characters).
To production
Cut release 2026.10.5 from main. Watchtower on hyper.media then pulls the latest web, notify, vault and agents images on its own.
Right after the images land on hyper.media, apply SeedInfra to prod:
sudo systemctl stop traefik-domain-sync on hm (the old script would race the upload);
in Terraform Cloud, discard the queued SHM-Core-PROD runs for #14 and #15 and confirm the newest one (#16), which contains all three;
verify https://hyper.media/hm/api/notify/health, a legacy /hm/api/notifications call, the crawler throttle still present, and one welcome email.
Until this apply runs, prod notifications and notify mail are degraded: new web images rewrite the old notify host to hyper.media/hm/api/notify, which is not routed yet, and new notify on 465 is unreliable. So do steps 1 and 2 back to back.
Publish the desktop release (latest.json).
After the desktop update has spread, open retire-notify-host as a PR and apply it.
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime