Deploying to the box

The pipeline (2026-09-09)

Merge to main and it rolls. .github/workflows/deploy.yml runs on the box's own runner for this repo (slowdesk-postmortem, label postmortem, installed at /home/ubuntu/actions-runner-postmortem as a systemd service), builds the image natively from the merged checkout, syncs compose.yml into /opt/postmortem, points POSTMORTEM_IMAGE at the new tag, rolls api and worker, and proves it — /health must report the new build — printing the rollback command either way. The whole deploy is deploy/deploy.sh, the same script an operator runs by hand from a checkout on the box when Actions is down (./deploy/deploy.sh <sha>). Every PR runs .github/workflows/ci.yml (typecheck + the suite, hosted arm64, ~2 min); a direct push to main is refused by the deploy. The one freeze window is the nightly's, 00:15–01:00 ET.

The laptop-and-ECR path below is the break-glass: it still works, and compose.yml's image default still points at ECR, but the pipeline never uses a registry.

Where /opt/postmortem on the EC2 box, beside /opt/tradingdesk How docker compose up -d from that directory, the same image for api and worker Reach it ssh -i ~/.ssh/tradingdesk_ec2 ubuntu@100.61.104.169 — the Elastic IP; port 22 is open to Evan's address only

Evan, 2026-09-04: "where will we deploy this app, to /opt/postmortem". Yes. This mirrors how tradingdesk lives on the box, with less machinery: no GitHub Actions runner, no standby roll, no freeze window. This service has no live order path, so a restart costs a client a reconnect and nothing else.

What the box has (measured 2026-09-04)

disk    290 GB root, 219 GB free      (#569 filled it two days ago; watch it)
memory  30 GB, ~21 GB available
cpu     8 cores
docker  29.1.3, compose plugin
/opt    tradingdesk/  containerd/

Containers, not source

Evan, 2026-09-04: "would we need to push the source repo, wouldn't we just create containers?" Containers. The box holds three things — compose.yml, .env, and the image it pulls — and never the repo. Every command that runs there, minting keys and building tape included, runs inside the container with docker compose run.

The image lives in ECR: 366221175786.dkr.ecr.us-east-1.amazonaws.com/postmortem. The box is Graviton (arm64) and so is the laptop, so a plain docker build here runs there. No CI, no cross-build.

Build and push (laptop)

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 366221175786.dkr.ecr.us-east-1.amazonaws.com
SHA=$(git rev-parse --short HEAD)
docker build -t postmortem:$SHA .
docker tag postmortem:$SHA 366221175786.dkr.ecr.us-east-1.amazonaws.com/postmortem:$SHA
docker tag postmortem:$SHA 366221175786.dkr.ecr.us-east-1.amazonaws.com/postmortem:latest
docker push 366221175786.dkr.ecr.us-east-1.amazonaws.com/postmortem:$SHA
docker push 366221175786.dkr.ecr.us-east-1.amazonaws.com/postmortem:latest

npm run typecheck && npm test first; the image does not run them.

The box's role

The instance runs as the IAM role slowdesk-box, and containers can use it (the metadata hop limit is 2). So .env on the box carries no AWS keys. What the role could do before 2026-09-04 was the desk's own buckets — slowdesk-market-archive, where slowsocket's recorder writes its ticks, and the backups bucket. deploy/box-role-policy.json, attached inline as postmortem-perms, adds pull on the postmortem ECR repository and read/write under s3://slowdesk-io/market-history/. Nothing else.

First deploy (box)

sudo mkdir -p /opt/postmortem && sudo chown ubuntu:ubuntu /opt/postmortem
cd /opt/postmortem
# compose.yml: copy from the repo (scp); it is the only file besides .env
# .env: from .env.example, filled in —
#   DATABENTO_KEY        the same key as /opt/tradingdesk/.env (until the #9 cutover removes it)
#   TAPE_SOURCE          massive (default) | databento — which tape scores are judged against (#9); the Databento plan was cancelled 2026-09-24 (#14), so only a deliberate value reaches it
#   MASSIVE_API_KEY      the same key as /opt/tradingdesk/.env; required when TAPE_SOURCE=massive
#   LEDGER_DATABASE_URL  the RDS ledger, read-only role — reachable directly from the box, no tunnel
#   AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY   read/write on s3://slowdesk-io/market-history/, pull on ECR
#   PUBLIC_URL=https://api.slowdesk.io
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 366221175786.dkr.ecr.us-east-1.amazonaws.com
docker compose pull && docker compose up -d --wait
curl -s localhost:8470/health
docker compose run --rm api node bin/api-keys.ts create --name "evan"

A redeploy is a push from the laptop and docker compose pull && docker compose up -d --wait on the box. POSTMORTEM_IMAGE=…:<sha> in .env pins a version; latest follows the last push.

Ingress: api.slowdesk.io

Evan, 2026-09-04: "we'll need an ingres point, an api and api-key for users" · "we might need a subdomain too". The subdomain is api.slowdesk.io; the ingress is the Caddy already serving slowdesk.io on the box (active, verified 2026-09-04). Three steps, all the owner's:

  1. DNS — in Route 53, zone Z02869558GYM5EX9V5TX: A api.slowdesk.io → 100.61.104.169, TTL 300. That is the instance's Elastic IP and what slowdesk.io already resolves to; the 54.211.130.240 in the desk's docs is stale. Done 2026-09-04.
  2. Caddy — append deploy/Caddyfile to /etc/caddy/Caddyfile and sudo systemctl reload caddy. TLS is automatic. Ports 80 and 443 are already open in the security group for slowdesk.io. Done 2026-09-04.
  3. Keys — mint one per user: docker compose run --rm api node bin/api-keys.ts create --name "…".

Until the app is deployed, https://api.slowdesk.io answers 502 from Caddy: the front door is up and there is nobody home.

compose.yml keeps the API on 127.0.0.1:8470, which is exactly right once Caddy is the front door: the only public path is through TLS. Reading and replay are public (an anonymous replay ends after five minutes); anything that spends vendor time or disk needs a key. The desk's own containers on the same box can use https://api.slowdesk.io like anyone else, or the tailnet address if a second api.ports entry "100.61.104.169:8470:8470" is added.

PUBLIC_URL in .env should be https://api.slowdesk.io; the service names itself with it in GET / and in error messages.

The replay socket on the box

ws://100.61.104.169:8470/replay?symbols=…&day=…&from=09:25 is what a lane or a lab worker connects to. The first connection for a symbol-day pulls its object from S3 into ticks-cache (a named volume, so a redeploy keeps it); every later one plays from disk. POST /ticks/warm fills the cache ahead of time — the nightly worker is the natural caller, once the population is named. See the replay tape.

The cache grows by one object per symbol-day played, 17–40 MB each. A year of twelve names played end to end is ~50 GB; prune the volume or cap it before that is a problem on a disk that has already filled once.

Building the tape on the box

node bin/build-ticks.ts runs inside the image (docker compose run --rm worker node bin/build-ticks.ts …). It holds one symbol-day in typed arrays (a SPY day peaks under 2 GB) and one gzipped object on disk at a time, so it is safe on the root disk. It is also safe to run from a laptop, which is where the first year was built; the bucket is the same.

Deployed 2026-09-04

Build 4893dd4, pulled from ECR by the role, four containers healthy. https://api.slowdesk.io/health answers from the public internet; /me, the manifest, an object download and a cache warm all worked through the ingress with the first key (id 1, "evan", all three scopes). The warm proved the role reaches slowdesk-io/market-history/ from inside a container with no keys in .env.

Not done