The lake layout

Bucket s3://slowdesk-io/ (us-east-1, public access blocked, SSE-S3 default) Prefix market-history/ Written by bin/pull-history.ts · read by DuckDB

Three stores, not one merged database

The tape, the desk's decisions and this service's output have incompatible shapes, and merging them would ruin all three.

The ledger is snapshotted into the lake, not joined live — which also satisfies the "everything recomputable from inputs" rule, since re-scoring a ride months later needs the ledger as it was.

Layout

s3://slowdesk-io/market-history/
  raw/     schema=tbbo/dataset=XNAS.ITCH/day=2026-09-02/data.csv.zst
  parquet/ schema=ohlcv-1d/dataset=XNAS.ITCH/2022-09-01_2026-09-03/data.parquet
           schema=ohlcv-1m/dataset=XNAS.ITCH/month=2024-03/data.parquet
           schema=tbbo/dataset=XNAS.ITCH/day=2026-09-02/data.parquet
  ticks/   day=2026-09-02/symbol=AAPL/ticks.ndjson.gz      ← the replay tape, for the desk
           day=2026-09-02/symbol=AAPL/manifest.json
           manifest.json

ticks/ is the odd one out: not Parquet, not for DuckDB, and partitioned by symbol. It is the desk's recorder's own layout, produced here so slowsocket's replay loader reads it unchanged — see the replay tape. Its consumer is one process reading one symbol-day, for which one object per symbol-day is exactly right.

Why raw/ exists for ticks only

Keeping the delivered bytes looks redundant when re-pulling is free — and for bars it is, so we don't. Ticks are different: the free window rolls. Today's tbbo costs $0; the same day costs $6/GB once it ages past ~12 months. If the Parquet schema later proves wrong we must be able to re-derive rather than re-buy. Insurance is kept exactly where re-buying would cost something.

Format: Parquet, not DBN

The Rust point is load-bearing: the trainer repo reads Parquet through Arrow with no vendor dependency, which keeps the boundary at the labels.

Two type decisions that are correctness, not taste

Prices stay BIGINT at Databento's native 1e-9 fixed point. The mechanical walk's entire job is "did price touch $19.27?", and float comparison at the boundary produces wrong door decisions — which become wrong deltas attached to a person's name. Scale at display time only (close/1e9).

Timestamps are TIMESTAMP_NS. Downcasting to milliseconds would collapse trades that share a millisecond, and order within that millisecond is exactly what decides which door was touched first.

Sort, don't partition by symbol

Rows are sorted (symbol, ts) inside each file. Partitioning by symbol would give 12,000 symbols × 1,000 days = millions of tiny objects, and S3 listing becomes the bottleneck. Sorting gives row-group statistics that prune to one symbol's row groups just as well, without the file explosion.

Chunk size follows density

Every chunk costs a cost-check, a fetch, a DuckDB pass and an S3 PUT — about six seconds regardless of size. The first run chunked four years of daily bars by month and spent 25 minutes of overhead on 1.4 MB.

ohlcv-1d   whole range   ~1.4 MB over 4 years for 125 symbols   -> 5 chunks
ohlcv-1m   month         ~20 MB each
tbbo       day           ~130 MB each

Reading it

INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE SECRET s3sec (TYPE s3, PROVIDER credential_chain, REGION 'us-east-1');

SELECT symbol, ts::DATE AS d, open/1e9 AS o, high/1e9 AS h,
       low/1e9 AS l, close/1e9 AS c, volume
FROM 's3://slowdesk-io/market-history/parquet/schema=ohlcv-1d/*/*/data.parquet'
WHERE symbol = 'LHAI' AND ts::DATE BETWEEN '2026-09-01' AND '2026-09-03'
ORDER BY ts;

Landed so far

ohlcv-1d   99,061 rows · 125 symbols · 2022-09-01 -> 2026-09-03 · 5 objects
ohlcv-1m   in progress
tbbo       not started (~37 GB for the traded set, free)

Three merge problems, none of them about storage

1. Symbol identity over four years. The ledger stores raw ticker strings, and tickers get recycled — symbol = 'XYZ' across four years can silently join two unrelated companies. The join key must be (symbol, date) resolved through Databento's definition schema, not symbol alone.
2. Timestamps. The ledger stores entered_at as TEXT. Databento is nanosecond integers. Normalise to timestamptz on the way in.
3. Corporate actions. An unadjusted series makes a split look like a −75% trade (already logged as trainer#50). Store bars unadjusted with a separate factors table and apply at read — adjustment conventions change, and a baked-in one cannot be revisited.