The replay tape: what slowsocket would have recorded, for a year back

Spec tradingdesk#777, comment 5545323556 — "what the external tick store must deliver for the desk to replay a stock over a period" Built by bin/build-ticks.ts · checked by bin/verify-ticks.ts · shape pinned in src/ticks-day.test.ts Served from s3://slowdesk-io/market-history/ticks/… and GET /ticks/… on the API

What it is

The desk's replay (workers/slowsocket/ticks-load.ts → intoSimStream) consumes one shape: the gzipped NDJSON its own recorder writes to ticks/day=…/symbol=…/ticks.ndjson.gz. That archive started on 2026-09-02, holds only held names, and grows a session a day. Prereg K (the 20-second lane) needs a year of SPY, QQQ and ten liquid names it has never held.

This service already reads Databento's listing-venue feeds, free inside a rolling ~12-month window. So it writes the recorder's shape from that tape, and the desk's loader reads either archive with a different base URL and nothing else. Nothing here depends on a new loader on the desk side.

Where each channel comes from

Not tbbo, which the scorer uses. tbbo is one quote per trade; a 20-second lane prices its spread and its queue from the book between trades. Measured on AAPL, one session is 22,938 tbbo records against 528,669 for mbp-1. The spec's own estimate — 100k–500k quote rows per liquid day — is mbp-1.

Always the primary listing venue (XNAS.ITCH, XNYS.PILLAR, …), never EQUS.MINI or DBEQ.BASIC, whose partial books measured 68× too wide. The builder takes SYMBOL:VENUE and refuses anything else.

The row, and what a direct venue feed cannot say

The columns, in the spec's order, then two extras:

day, symbol, channel, ts, seq, source,
price, size, conditions, side, venue, trade_id, sip_seq,
bid, bid_size, ask, ask_size, bid_venue, ask_venue,
status, reason,
aggressor, feed_status          <- extras; nothing on the desk depends on them

The recorder's source is a SIP-shaped feed with condition codes and a consolidated venue per print. Databento's listing-venue feed is the exchange's own book, so four columns carry less than the recorder's would, and each is stated rather than faked:

source is databento:xnas.itch (the dataset, lower-cased), so a receipt built on this tape can say where it came from.

ts is Databento's ts_event — the matching engine's own stamp — at nanosecond precision (2026-09-02T13:30:00.123456789Z). Never ts_recv.

status is the spec's four words. Databento has no separate resume action, so resumed is a trading that follows a halted; reason is the feed's reason enum by name (regulatory_luld, regulatory_resume, …) or, when there is none, the action. feed_status keeps the raw enums so nothing is lost in the mapping.

A one-sided book is real: the missing side is null, never 0, and its venue is null too. A quote row is emitted on change of the top of book, so a burst of deeper activity that leaves the inside untouched produces no row.

The unit: one symbol-day, whole or absent

A day is 04:00–20:00 ET on its session date (both Nasdaq's and NYSE's system hours), resolved through the IANA zone so DST is right on both sides.

The builder writes the object to a temp file, runs the same checker the verifier runs on a delivered object — sorted, seq contiguous, columns in order, other channels' columns null, no 0 on a missing side — gzips and hashes it in one pass, uploads it, and only then writes the day's sidecar:

ticks/day=2026-09-02/symbol=AAPL/ticks.ndjson.gz    the tape
ticks/day=2026-09-02/symbol=AAPL/manifest.json      rows per channel, first/last ts, bytes, sha256
ticks/manifest.json                                 the coverage manifest, all symbols and days

A day is on file when its sidecar exists. An upload that dies between the two leaves an orphan the manifest never mentions; the next run replaces it. The coverage manifest is an aggregate of the sidecars, updated at the end of every run and rebuildable from them (--rebuild-manifest), so it is never the only record of a day.

Three channels or refuse. If mbp-1 or status cannot be fetched — not free, not licensed yet, not offered — the day is refused and the log names the channel. A day with trades and no quotes would let the desk's execution simulator price a spread from nothing, which is the lie Alpaca paper tells.

A session with no records (a holiday, a name not yet listed) is absent, not empty.

The cost gate, twice per day

mbp-1 is ~23× the bytes of tbbo, and ticks cost $6/GB outside the free window. Both schemas of every day are priced with isFree() before either byte is fetched, and an unknown price is a refusal, not a zero. See the no-spend guarantee.

Delivery

S3, the same keys as the recorder, under this service's lake root. (The recorder's own archive is a different bucket, slowdesk-market-archive, seen in the box role's policy 2026-09-04; the desk's loader is pointed at one or the other by its base URL.)

s3://slowdesk-io/market-history/ticks/day=YYYY-MM-DD/symbol=XXX/ticks.ndjson.gz
s3://slowdesk-io/market-history/ticks/manifest.json

Plain HTTPS, the same bytes, from the API — for a box without bucket credentials:

GET /ticks/manifest.json
GET /ticks/day=YYYY-MM-DD/symbol=XXX/ticks.ndjson.gz
GET /ticks/day=YYYY-MM-DD/symbol=XXX/manifest.json

The object is served only when its sidecar exists. No SDK on either side: the store shells out to aws exactly as the history puller does.

TICKS_OBJECT_MAX_BYTES, when set, refuses to deliver a day larger than the desk's loader will read into memory; bytes is in every sidecar and in the manifest regardless, so the loader can decide for itself.

Played back: the replay socket

The files are the archive. ws://<host>/replay?symbols=AAPL&day=2026-09-02&from=09:25 is the socket — the rows, one message each, at tape pace, which is what slowsocket's socket would have delivered that day. Parameters, frames and control are in the API.

What is preloaded, and what is not. Evan asked whether a connection needs data preloaded to play in real time. Only the object fetch. A liquid day is ~1M rows in ~17 MB gzipped; gunzip + parse runs at a few hundred thousand rows a second and real time needs a few hundred, so the reader is always far ahead of the clock and nothing is held in memory. The S3 fetch is the one wait — seconds for a liquid name — so objects are cached on local disk (TICKS_CACHE_DIR) after the first play, and POST /ticks/warm fills the cache ahead of a session. A connection says loading while it waits and ready when the clock is pinned.

Seeking, and the snapshot. A day starts at 04:00 ET. Seeking to from skips rows at decode speed, and everything skipped is folded into one snapshot per symbol — the quote, trade and status standing at that instant — sent before ready. A lane that joins at 09:25 starts with a book.

Memory, measured

A SPY day is 5.3M rows. Held as parsed JSON objects, it ran node out of heap on 2026-09-04, and so did NVDA. The builder now ingests records into typed-array columns (~60 bytes a record, off the JS heap), sorts an index, and yields rows straight into gzip:

SPY  2025-09-02   227,070 trades · 5,098,454 quotes · 3 status   81.3 MB   180 s   peak RSS 1.1 GB

Running it

node bin/build-ticks.ts --symbols AAPL:NASDAQ,SPY:ARCA,QQQ:NASDAQ --from 2025-09-08 --to 2026-09-03
node bin/build-ticks.ts --symbols AAPL:NASDAQ --from 2026-09-02 --dry-run        # price it, fetch nothing
node bin/build-ticks.ts --symbols AAPL:NASDAQ --from 2026-09-02 --out ./tape      # to local disk
node bin/build-ticks.ts --rebuild-manifest
node bin/verify-ticks.ts AAPL 2026-09-02
node bin/verify-ticks.ts --all
npm test

Days already on file are skipped; --force rebuilds them. The verifier re-reads the delivered bytes, recounts every channel, checks first and last ts and the sha256 against the sidecar, and exits 1 on the first day that fails. A day that fails is a day the desk must not replay.

What has been checked, and what has not

Checked, 2026-09-04, against a fake Databento serving synthetic mbp-1 and status days:

• 3 sessions × 1 symbol built, verified, rebuilt into a manifest; a second run found all three on file and fetched nothing.
• A symbol priced at $1.26 was refused before any fetch. A symbol with no records was absent, not empty. A symbol whose status schema returned 422 was refused with the channel named.
• Deleting one line from a delivered object made the verifier fail the day on the second line and every line after it.
• The API served the manifest, a sidecar and the object; the object's bytes and sha256 matched the sidecar; a day not on file returned 404.
• 10 unit tests pin the column order, the channel derivation, the one-sided book, the sort, the status mapping, the sentinel price, ns round-trip, and the checker's catches.

Checked, 2026-09-04, against the live API — AAPL, XNAS.ITCH, 2026-09-02:

mbp-1    priced $0.00      status   priced $0.00       (SPY on ARCX.PILLAR and QQQ: also $0)
rows     998,903           trade 117,815 · quote 881,086 · status 2
span     08:00:00.000104823Z -> 23:59:50.578714494Z   (04:00 -> 19:59 ET)
object   17.4 MB gzipped   built in 24 s              verified, sha256 matches

• The free window covers mbp-1 and status, not only tbbo.
• Status enums arrive as numbers (action: 1 pre-open at 04:00 ET, action: 7 trading at 09:30 ET). No halt that day, so halted/resumed has only been exercised by the fixture.
• The first quote of the day is one-sided — a bid with no ask — and lands as ask: null, ask_size: null, ask_venue: null, as the spec wants.
• aggressor is set on 88,769 of 117,815 trades (47,990 buy, 40,779 sell); the rest are the feed's "no side" — mostly crosses.
• A year of one liquid name is therefore ~250 × 17 MB ≈ 4 GB in the lake and ~2 hours of building. The twelve-name population is ~50 GB and a day.

Not yet checked:

• That status is offered on all five venue datasets. If a venue lacks it the day is refused and the log says so — it does not degrade to two channels.
• A real halt through the mapping. The fixture covers it; a LULD day will confirm feed_status.action = 8 maps to halted and the resume to resumed.

Still open

• Sizing before widening. Twelve names × ~250 sessions × tens of MB is a few tens of GB in the lake and one symbol-day in memory at a time on the builder. Fine on a laptop; on the desk's box, #569 filled the root disk two days ago — the builder never holds more than one day on disk, but a nightly build should still be sized.
• The nightly. Today's session is licensed at 04:00Z; bin/worker.ts could build the population's day right after it scores. Not wired — the population is not decided (open question 3 on the issue).
• Line streaming on the desk's loader is the spec's own stated follow-up if a liquid day exceeds its object cap. bytes is there so the desk can see it coming.