HomeMASTERT1T2T3T4T5T6T7T8T9Wireframes
Caverio trader companion, overnight programme 2026-09-26, T2

T2: Infrastructure and data architecture, with efficiency budgets

Caverio trader companion, overnight conceptual design, 2026-09-26. Worker T2 (Opus 5.5), started 00:37 Beirut.

Everything described in §1 is the experimentation stack, per Thomas's 00:28 decision. It is measured here so the build stack can be designed against real numbers, not so it can be kept by default.

How to read the numbers. Every number from our own machine comes from a script in scripts/T2-* with its output in data/T2-*, run between 00:37 and 00:55 on 2026-09-26 (grade A, ours). Web or vendor facts carry a grade: A primary source fetched and cited, B credible secondary, C my inference or an estimate. Estimates are labelled as estimates.

Script What it measured Output
scripts/T2-procs.sh process inventory (pid, uptime, RSS, lifetime CPU) data/T2-procs-0037.tsv
scripts/T2-unit-sample.py 300 CPU over a 300 s window per systemd unit (CPUUsageNSec delta) and per non-unit process; memory data/T2-unit-sample.tsv
scripts/T2-net.py 300 per-process network bytes from ss -tinp tcp_info counters over 302 s, loopback excluded; interface totals data/T2-net.tsv
scripts/T2-disk.sh bytes, files and newest mtime per entry of signal-room/data/ data/T2-disk-20260926-0038.tsv
scripts/T2-requests.py provider calls per day from broker.log, tape round failures by reason, tape state, holdings counters data/T2-requests.txt
scripts/T2-volumes.py rows per day: callers, Fomo events, tape trades, deep-paper observations, events.sqlite data/T2-volumes.txt
scripts/T2-llm-grep.sh, scripts/T2-llm-ledger.py every model call in the stack, and the one token ledger that exists data/T2-llm-grep.txt, data/T2-llm-ledger.txt
scripts/T2-units-cron.sh crontab, user units, timers (read-only) data/T2-units-cron.txt
scripts/T2-budget.py the 50 / 200 / 500 token budget model in §5, all constants named with their basis data/T2-budget.txt

Measurement caveat for tonight. Nine other overnight workers (claude -p) were running during the samples. They show up in the openclaw-gateway CPU (164% of one core) and in the interface totals (25.7 GB/day down, 11.9 GB/day up if extrapolated). They are not the stack. Per-process rows below exclude them. For a clean baseline, the 09-25 system audit's post-decommission reading is the reference: download 4.1–5.6 GB/day, upload 1.9–3.1 GB/day, load about 1.7 (projects/system-audit-2026-09-25/AUDIT.md).

1. Current stack, measured (the experimentation stack)

1.1 State of the box at 00:40

The crypto stack was cut to "option B, research-only" on 2026-09-25 at 13:10 (Thomas: "B, data is important"). Fifteen services were stopped and disabled, including the situations writer, patterns, telemetry, paper-tiered, case-builder, admin-status, thesis-enrich, gate and the cloudflared tunnel. Fifteen raw collectors kept running. signal-room-v2 (the server) was restarted at 17:09 because two kept collectors depend on it. So today's picture has three layers:

Host: 28 cores, 64 GB RAM, 915 GB disk with 567 GB free. signal-room/data/ is 15.6 GB (was 13 GB at the 09-25 00:45 audit).

1.2 The table

CPU is the share of one core averaged over the 300 s window. RSS is from /proc. Requests per day are from the workers' own counters where they exist. Disk growth is the 24 h delta against the 09-25 audit sizes (audit table du -h values against tonight's du -sh), so it is ±5% resolution. "Layer" is the product layer from the mandate: L1 money brief, L2 discovery and situations, L3 zones.

Component (unit) What it does Cadence RSS CPU Requests/day (source) Disk growth/day Layer Verdict and why
caverio-price-paths (workers/price_paths.py --feed) re-measures price paths for ~9,880 subjects 60 s 232 MB 52% 0 external; 1,440 local GETs of /api/feed price-paths.jsonl 6.4 → 6.7 GB (+0.3 GB) L3 research Rebuild. It re-parses its own 7.09 GB append-only ledger every cycle (samples_read()), so its CPU grows with its file. Latest-row-per-subject is ~46 MB. The product needs forward returns per situation, computed once from the tape and candles, not a re-scan.
signal-room-v2 (server.py :3045) legacy read model + API, kept only as a dependency on request, rebuilds room 502 MB 45% 0 external room.json 99.6 MB rewritten, feed.json 7.4 MB L2 (dead UI) Retire after price_paths, holdings and the BSC tape stop depending on feed.json. It spends half a core building a 100 MB room nobody views.
signal-room-v2-market (workers/market-refresh.mjs --loop 60) DexScreener price poll via broker; writes ticks, deep-paper observations, market-cache 60 s 105 MB 12% 3,594 DexScreener calls (batches of 30) deep-paper 314 → 435 MB (+0.12 GB), ticks 41 → 44 MB L2 price Keep with fix, then fold into the price collector. Fixes 1–3 of the audit (raw sidecar, source + providerAt, unchanged flag). 33% of its prints are exact repeats.
signal-room-v2-broker (workers/market-broker.mjs :3046) single client for every market provider, budget pacing per request 111 MB 0.0% DexScreener 3,594, DexPaprika 1,451, CoinGecko 1,179, GeckoTerminal 4 broker.log ~1 MB L1/L2 Keep. The one good pattern for provider budgets (credit pacing, 429 accounting). Becomes the fetch layer for all HTTP collectors.
caverio-chain-tape-bsc (workers/chain-tape.mjs --chain bsc) BSC swap tape per live pool via eth_getLogs on public RPC continuous 610 MB 1.3% ~128,000 failed rounds/day (99.5k "archive requests require a personal token", 28.9k "ranges over 10000 blocks") trades-bsc 998 → 1,022 MB (+24 MB, collapsing: 155k rows on 09-20, 22k on 09-25) L2 flow Rebuild the fetch strategy, keep the schema. 57 of 64 pools carry a lastError; pools that fell out of publicnode's head window never recover and retry forever. The log moves, so the stall watchdog sees it as healthy. See §6.
caverio-chain-tape-solana (--chain solana) Solana swap tape per pool via signatures + getTransaction 5 s live poll, 150 s round deadline 176 MB 0.4% Helius 20,000 / 20,000 used (state file); now public fallback trades-solana 72 → 107 MB (+35 MB) L2 flow Rebuild. Hung 8 days to 09-25 00:22, stalled again 07:40–12:09 (watchdog), and 280 rounds logged "N of M left unfetched at the 150 s deadline" (latest: 2,986 of 3,153). Per-transaction fetch cannot scale (§5). Download 848 MB/day for ~26k rows (32 KB per stored row).
signal-room-v2-onchain (lib/onchain-candles/ingest.mjs) Robinhood Chain swaps → trades + 1 m candles continuous 454 MB 2.3% public RH RPC; download 746 MB/day candles-onchain 107 → 121 MB (+14 MB) L2 flow (RH only) Keep with fix until the RH decision (D4). Merge into the one EVM tape collector.
signal-room-v2-holdings (workers/holdings.mjs --loop 120) early-buyer balance coverage per open situation 120 s 1,218 MB (heap 940 MB, cap 1,536) 1.3% ~2–3 RPC calls per pass via Multicall3 (~2,000/day) holdings 50 → 51 MB L2 holders Rebuild. Largest RSS in the stack for 23 situations; 46% of files are "chain has no tape" stubs (audit). Holder concentration is a derived field over the tape, not a service.
signal-room-v2-candles (workers/candles-service.mjs) OHLCV cache + outcome marks 300 s 110 MB 0.5% GeckoTerminal/CoinGecko via broker ohlcv ~6 MB, outcomes flat L2/L3 Keep with fix (providerAt on outcomes, audit fix 2). Becomes the candle fetcher behind the broker.
signal-room-v2-discovery (workers/discovery.mjs --loop 60) screens caller posts into discovery rows 60 s 66 MB 0.0% via broker discovery.jsonl +small L2 discovery Keep the logic, rebuild as a derivation over the event store.
Callers cron (watchtower/callers.mjs, */5) Telegram caller channels via public t.me/s/ previews, 5 channels 5 min n/a (cron) n/a ~288 page fetches/day callers.jsonl: 28–127 msgs/day captured L2 discovery Rebuild. Five channels, formatted calls only; the mandate asks for whole chats and many more callers. Needs a real Telegram client session (T4 decides the source).
Fomo watcher (watch-tracked-accounts.mjs, cron watchdog) Fomo websocket alerts for 119 traders streaming 90 MB 0.0% 1 websocket; 1,000–2,000 events/day raw-events-v2.jsonl ~1 MB/day L2 discovery Keep. Cheapest high-signal source we have. Move under a real unit with a heartbeat (§6).
crypto-experiment-a-collector / -marker raw Fomo collector + marker (research) loop 53 / 273 MB 0.0% / 7.8% Fomo REST (credits) research dir research Keep until its experiment reports, then retire. The marker costs 8% of a core.
caverio-social-sweep (social-sweep/collector.mjs --no-x) per live situation: X (dark) + public t.me previews 10 min 126 MB 0.1% X 0 (dark since 09-18); t.me ~91 MB/day down social-sweep 24 → 24 MB L2 social Retire into the new social collector (T4).
caverio-social-reader (--provider none) one counted-fact sentence per situation 10 min 98 MB 0.2% 0 LLM calls since 09-20 (template only) social-read ~6 MB L2 social Retire. Its ledger is our only measured LLM cost basis (§5): 430 input / 55 output tokens per rephrase call, 2,601 calls and $1.76 on its busiest day (09-17).
crypto-watchtower (watchtower.mjs) + smartmoney cron wallet watch, module A monitor + paper loop / 5 min 202 MB 0.6% download 208 MB/day, upload 608 MB/day watchtower data L2 wallets Keep the wallet list, rebuild the collector onto the tape (a wallet buy is a tape row with a known actor). The 608 MB/day upload needs a look; I did not attribute it.
signal-room-proof-daytime (+ proof_supervisor.py, onchain-bridge/proof-v1/run.mjs) funded-proof on-chain observer (research) 600 s bounded children 148 MB 1.9% download ~1 GB/day proof runtime research Retire at the end of the proof experiment; its useful output (linked receipt flow) is a tape derivation.
Timers: store-sync (60 s), catcher-publish (30 s), proof-detector (15 s), source-sentinel (5 min) JSONL → events.sqlite mirror; catcher annotations; detector; source cadence sentinel timers oneshot 2.3 s and 0.3 s CPU per run 0 external events.sqlite 372 → 412 MB (+40 MB) L2 store-sync: keep the pattern, replace the mirror with direct writes. Others: retire with the ladder.
Crash loops: crypto-fast-sources, crypto-upstream-paper research collectors restart every 15–30 s since at least 09-25 13:00 n/a n/a n/a journal spam none Retire now (one is status=200/CHDIR, a missing directory). Pure waste and journal noise.
Five wrangler pages deploy processes stuck Cloudflare deploys, 26–50 h old hung ~180 MB each, ~900 MB total 0% n/a n/a none Kill (Rook, not this lane). Memory leak by neglect.

Totals for the stack (excluding tonight's agents): about 1.3 cores of CPU, of which 0.97 is price_paths plus the server, i.e. two processes doing re-computation; about 4.3 GB RSS; download about 3.3 GB/day attributable per process (a floor: short-lived sockets are missed), matching the audit's 4.1–5.6 GB/day total; disk growth about 0.55 GB/day with the ladder writers stopped, of which 0.3 GB is the price-paths re-measurement ledger. External requests: about 6,200/day to market APIs through the broker, plus about 128,000/day of failing BSC RPC calls.

1.3 What the measurement says, in five lines

  1. CPU is spent on re-reading, not on collecting. 75% of the stack's CPU is price_paths re-parsing its 7 GB ledger and the server rebuilding a 100 MB read model for an audience of zero. The collectors themselves use 0.1 cores between them.
  2. The tapes are the product and they are both broken, silently. BSC coverage fell from 155k trades/day (09-20) to 22k (09-25) because publicnode stopped serving archive ranges; Solana spent its Helius allowance and misses most transactions at the round deadline. Neither shows up as "down".
  3. Liveness is measured as "the log moved". The BSC tape logs 1.5 failures a second, so it always looks alive. A heartbeat must count progress (blocks or slots covered per pool), not activity.
  4. Disk is not the constraint. 0.55 GB/day on a disk with 567 GB free is three years. Disk only matters for query speed (the 7 GB file) and for backup (the git backup includes data/).
  5. LLM spend in the product today is zero. The only model caller (social-reader) is on templates since 09-20, and X is dark. The companion's social layer will make LLM tokens the second-largest variable cost (§5), so it has to be designed with a budget from day one.

2. Target architecture

2.1 System context

flowchart LR
  subgraph Ext["External sources"]
    CH["Chains: Solana, BSC, Base, (Robinhood)"]
    MK["Market APIs: DexScreener, CoinGecko, DexPaprika, GeckoTerminal"]
    MAC["Macro: stablecoin supply, CEX/DEX volume, BTC dominance (T3)"]
    SOC["Social: Telegram callers + chats, X accounts (T4)"]
    FOMO["Fomo: trader alerts + theses"]
    LLM["LLM API: Haiku-class extraction, Sonnet-class stance"]
  end
  subgraph Box["Caverio backend (one box to start)"]
    COL["Collectors (supervised, heartbeat)"]
    STORE[("Event store + tape store + raw sidecar")]
    DER["Derivers (rebuildable)"]
    RM[("Read models per page")]
    API["API (read-only)"]
  end
  subgraph Users["Beta users"]
    WEB["Web app: market brief, discovery list, situation page"]
  end
  CH --> COL
  MK --> COL
  MAC --> COL
  SOC --> COL
  FOMO --> COL
  COL --> STORE --> DER --> RM --> API --> WEB
  DER <-->|budgeted calls| LLM
  OPS["Ops: supervisor, gap ledger, cost ledger"] -.-> COL
  OPS -.-> API

2.2 Data flow, source to page

flowchart TB
  subgraph Capture["1 Capture (raw, append-only, never rewritten)"]
    T1["tape collectors<br/>swap/trade rows"]
    P1["price collector<br/>raw provider payload + normalised print"]
    A1["attention collectors<br/>messages, posts, Fomo alerts"]
    M1["macro collector<br/>daily/hourly series"]
  end
  subgraph Norm["2 Normalise"]
    EV[("events: source, sourceId, raw sha256,<br/>receivedAt, availableAt, providerAt, quality")]
    TP[("tape: chain, block/slot, tx, logIndex,<br/>pool, side, amounts, usd, quoteSource")]
    PR[("prints: token, price, liq, mcap,<br/>provider, providerAt, unchanged, phantom")]
  end
  subgraph Derive["3 Derive (pure functions of 2, rebuildable)"]
    C1["1 m candles per pool (from tape)"]
    F1["derived fields per token per minute:<br/>net flow, buyers, holder concentration, volume vs band"]
    X1["mentions + stance per caller per token<br/>(LLM, versioned prompts)"]
    MB["market brief series<br/>(chain, band, BTC vs alts)"]
    GAP["gap ledger per collector per scope"]
  end
  subgraph Situ["4 Situations"]
    S1[("situation store: token, openedAt, trigger events,<br/>lights (categories lit), rank inputs")]
  end
  subgraph Read["5 Read models (one document per view)"]
    R1["brief.json per horizon"]
    R2["discovery list"]
    R3["situation page doc per token"]
  end
  T1 --> TP --> C1 --> F1
  P1 --> PR --> F1
  A1 --> EV --> X1 --> F1
  M1 --> EV --> MB --> R1
  F1 --> S1
  EV --> S1
  GAP --> R3
  S1 --> R2
  S1 --> R3
  F1 --> R3
  C1 --> R3
  R1 --> API2["API"]
  R2 --> API2
  R3 --> API2 --> PAGE["page"]

2.3 Modules

Collectors. One small process per source family, each doing only fetch, stamp, append. No collector computes anything a deriver could compute later. Four families: - Tape. One EVM tape collector (BSC, Base, optionally Robinhood) using one eth_getLogs per block window per chain with an address array for all tracked pools, not a request per pool. One Solana tape collector on a subscription (WSS logsSubscribe or a gRPC stream filtered to tracked pools), not per-transaction getTransaction. Both write the tape standard below. - Price. One price collector behind the broker, batched 30 tokens per DexScreener call, writing the raw payload to a sidecar and a normalised print with providerAt, unchanged and a phantom flag. CoinGecko/DexPaprika/GeckoTerminal only for candles and backfill. Our own tape candles become the primary price for tokens we tape; DexScreener becomes the fallback and cross-check. This also answers the rights critic: the displayed price is our tape, not a third-party print (T1 owns the rights verdict). - Attention. Telegram (callers and whole chats, T4), X (when a source exists), Fomo websocket and theses. Every message stored raw once, keyed by source and sourceId. - Macro. Low-frequency series for the market brief (T3 names them): stablecoin supply by chain, DEX volume by chain and platform, BTC dominance, market-cap band aggregates. Hourly or daily; a few hundred requests a day.

The event standard (applied everywhere). The events.sqlite pattern that already exists, made mandatory: source, sourceId, originalSourceId, raw blob (or a pointer into the raw sidecar), sha256 of raw, receivedAt (our clock), providerAt (their clock, null when they do not give one, never copied from ours), availableAt (when it became usable to a user, for honest replay), quality (source_observation, legacy_unverified, proof_recorded_not_reverified, plus new phantom, unchanged). Unique on (source, sourceId), content-addressed on sha256. Today availableAt is null on 91% of rows; in the build it is set at write time or the write fails.

The tape standard. The trades-bsc schema, which the audit rates full provenance: chain, txHash / signature, block / slot, logIndex / instruction index, blockTime, receivedAt, availableAt, pool, side, token amounts, usd, quoteUsdSource, tsBasis=block, schema. Keyed on (chain, txHash, logIndex). Add one thing it lacks: a coverage record per pool per window (from_block, to_block, status ok / gap / not_found / archive_unavailable). Without coverage rows a missing trade and a quiet minute look identical.

Derived-field store. Everything computed from capture: 1 m candles from tape, per-token per-minute fields (net buy USD, unique buyers, new holders, top-10 share, volume relative to peers in the same mcap band), attention fields (mentions per hour, distinct callers, stance balance, flips), brief series. Each row carries derivedFrom (input ranges and their gap status) and deriverVersion. Dropping and rebuilding any derived table from capture is a supported operation, and a nightly job proves it on one day.

Situation store. One row per token episode (situationId, token, openedAt, openedBy event ids, closedAt), plus an append-only table of light changes (category lit or dimmed, at, evidence event ids). The audit's rule becomes a constraint: no light change without evidence ids. T5 defines the categories and ranking; this module only stores them.

Read models. One pre-built JSON document per view: brief/{today,week,month}.json, discovery.json, situation/{token}.json. Rebuilt by a deriver on change, not on request. The API never computes; it serves documents. This replaces room.json (100 MB, rebuilt on request by the server) with small documents built once.

API. Read-only HTTP, auth by beta account, rate-limited per user, cacheable (ETag on each document). Also serves /status (collector heartbeats and gaps) and the internal query layer (§4.4).

Page. Static front end (Cloudflare Pages is already in use) reading the API. Wireframes are T8's.

2.4 Today's modules mapped onto the target

Today Target module Carry over Leave behind
events.sqlite, model/event_store.py, workers/store_sync.py event standard + event store schema, dedupe on sha256, quality classes the JSONL-then-mirror double write; legacy_unverified rows stay marked
chain-tape.mjs + lib/chain-tape/* (bsc, solana), lib/onchain-candles (RH) tape collectors row schema, pool resolution, quote sources per-pool polling, per-tx Solana fetch, public-archive dependence
market-refresh.mjs, market-broker.mjs, lib/candle-sources.mjs price collector + broker broker pacing and budget accounting, the provider ring ticks/ and deep-paper as separate writers of the same print
candles-service.mjs, ohlcv/, candles-onchain, candles-bsc derived candles onchain candle builder (rebuild.mjs) four candle stores with two schemas
watch-tracked-accounts.mjs, callers.mjs, social-sweep, thesis-enrich attention collectors Fomo websocket, roster files, thesis budget rule t.me preview scraping as the only Telegram path
holdings.mjs, telemetry.mjs, case-builder.mjs, lib/case-d8-fields.mjs derived fields field definitions worth keeping (T5 picks) services that hold 1.2 GB to compute a field
situations.py, model/states.json, alerts, patterns situation store evidence-id discipline the Now / Situation / People ladder (out, per Thomas)
server.py, room.json, feed.json, gate API + read models gate device-cookie idea read model built on request
price_paths.py, deep-paper, paper-tiered, strike research bench (off the product path) the replay tool and the forgiving-win measure (T6) re-measurement ledger as a live service
source_sentinel.py, chain-tape-stall-watchdog.sh supervisor + gap ledger per-source cadence policy, local outbox log-mtime liveness

3. Capture policy

Thomas's rule of 2026-09-19 governs: Caverio is first a database; keep any received data that could be useful, discard only if it hurts. "Hurts" has three meanings here: it costs more than it could be worth, we have no right to hold it, or it creates a privacy or legal exposure.

3.1 Capture raw and keep

Class What Why keep Retention
Tape every swap on every tracked pool, from pool open, with coverage records it is the product's own truth and cannot be bought back later at our price forever, compressed after day close
Raw market payloads the provider response behind each price print (sidecar) the only way to tell a provider's bad print from our handling (audit finding 2) 90 days raw, then keep only payloads referenced by a flagged print or a situation event
Messages every Telegram message from monitored callers and chats, every X post fetched, every Fomo alert and thesis, raw text + ids + author + timestamps stance, flips and caller track records are rebuilt from these; a message is cheap (0.5 KB) forever (text only; no media)
Macro series provider series as fetched brief history and backtests of "how flows moved markets" forever
Wallet labels and rosters every version of every list the track record of a list is part of the record forever, versioned
LLM calls prompt hash, model, prompt version, input ids, output, tokens, cost reproducibility and the cost ledger forever for outputs, 30 days for full prompts

3.2 Derive, do not store as truth (rebuildable)

Candles from tape, per-minute derived fields, holder concentration, band membership, attention aggregates, stance labels (the LLM output is stored, but a new prompt version re-derives), situations and lights, read models, forward-return marks. Retention: current version kept; previous versions kept 30 days then dropped, because they can be rebuilt. The rebuild must be tested (§2.3), or "rebuildable" is a claim.

3.3 Deliberately not captured

Not captured Why
Full-chain firehose (every swap on every pool of a chain) cost: at Solana meme volume it is tens of GB/day and a paid stream tier; we tape tracked pools plus a discovery ring, and use aggregate DEX volume APIs for the chain-level brief (C)
Media from Telegram / X (images, video) cost and rights; text and links are enough for mention and stance
Private user data of beta testers beyond account and feedback privacy; the feedback-for-access programme (T9) needs email, usage events and feedback, nothing else
Re-measurement ledgers (price-paths style) a re-measurement is a derivation; store the latest per key and rebuild on demand
Private Telegram groups we joined under a personal account, for display rights and trust; may be read for stance only if T4 and T1 clear it, never quoted raw on the page
Stale repeats as new observations stored with unchanged: true, never counted as an observation

3.4 Retention per class, summary

Capture: forever (compressed), except raw market payloads 90 days selective. Derived: current plus 30 days. Read models: current only. Logs: 14 days rotated (today they are not rotated at all). Backups: capture only; derived is rebuilt, which removes 85% of today's 15.6 GB from any backup.

4. Data organisation

4.1 Storage per class

Class Store Keys Partitioning Why
Tape rows Parquet (zstd) for closed days, append-only JSONL or SQLite for the open day (chain, tx, logIndex) tape/chain=<c>/day=<YYYY-MM-DD>/ columnar scans over millions of rows; 6–10x smaller (C); DuckDB reads it in place
Coverage records same partition as tape (chain, pool, from_block) by day and chain joins with the tape for gap accounting
Events (messages, alerts, prints, macro) SQLite with WAL, one file per month (source, sourceId), unique sha256 by month the existing standard; single-writer per file is enough at our volumes
Raw payload sidecar content-addressed zstd files sha256 raw/<source>/<day>/ dedupes naturally; pointer stored on the event
Derived fields, candles Parquet per day, or DuckDB tables (token, minute) by day rebuilt, so any format works; Parquet keeps it tool-neutral
Situations, lights, accounts, feedback Postgres or SQLite (see 4.3) situationId; (situationId, at) none small, relational, transactional, needed by the API
Read models JSON files (or KV) view + token none served as-is with ETags

4.2 Keys that tie it together

token = chain:address everywhere (the feed already uses it). pool = chain:poolAddress. Every derived row points to its inputs by (store, key range) and records the gap status of those inputs. A situation page is always addressable by token plus asOf, which is what makes an honest replay possible.

4.3 One database or a directory: the honest trade-off

The 40-entry data directory is the problem the audit named in fix 9: four candle stores in two schemas, two writers of the same print, a 7 GB file that one endpoint scans. There are three options:

Option Gains Costs Verdict
A. Postgres for everything (with TimescaleDB for tape) one query surface, transactions, concurrent writers, standard ops tooling, easy to host later tape at 500 tokens is ~1M rows/day; fine for Postgres but the box then needs vacuum/index tuning; raw blobs bloat it; another daemon to supervise and back up Overkill for capture on one box; right for the small relational part if we move off the box
B. DuckDB over Parquet + SQLite for events + small relational DB zero daemons; Parquet files are the backup and the interchange; DuckDB scans days of tape in milliseconds; SQLite is what events.sqlite already is DuckDB is single-writer; needs a clear writer per file and a compaction job at day close; less familiar for concurrent API reads (the API reads read models, so this is acceptable) Recommended to start. It matches the workload: append-heavy capture, analytical derivation, document-serving API
C. Keep the directory, add a query library (what Bolo's item 9 is doing) no migration keeps four candle schemas and double writers; the query library has to know every quirk Good as a bridge for the research bench, not the product

Recommendation: B for capture and derivation, SQLite (or a small Postgres) for situations, accounts and feedback, with the relational part kept behind one module so it can move to Postgres when there is more than one box. Decision D3.

4.4 The query layer that does not exist today

One Python (or Node) module, caverio.query, with DuckDB underneath, exposing the questions the product and research actually ask, for example: - tape(token, t1, t2) with coverage status; - candles(token, t1, t2, source=tape|provider); - fields(token, t1, t2, names); - mentions(token, t1, t2) with caller, stance, prompt version; - crossings(chain, band, t1, t2) with post-path at honest prices (audit fix 9's example); - gaps(collector, scope, t1, t2).

The API and the research bench both call it; nothing else opens the files. That is what lets storage change later without touching the page.

4.5 A situation page in one read

The situation deriver writes situation/{token}.json whenever an input for that token changes (debounced to 15–60 s). It contains the header numbers (price, mcap, liquidity, volume, flow), the chart series (1 m candles for 24 h, 5 m for 7 d), the lights with their evidence ids, the attention block (mentions, callers, stance balance and flips, peers in band), and the observation gaps for every input over the displayed window. The API serves that file with an ETag; the page does one GET and optionally subscribes to a small delta stream. Target size under 200 KB per token (C). At 500 tokens that is 100 MB of read models, rebuilt incrementally.

5. Efficiency budgets: 50, 200 and 500 tracked tokens

"Tracked" means a token with a live situation: its pools taped, price polled, attention read. The model is scripts/T2-budget.py; every constant is named there with its basis. Numbers marked MEASURED come from tonight's scripts; the rest are estimates (C).

5.1 Assumptions

Assumption Value Basis
Price poll every 60 s, 30 tokens per DexScreener call MEASURED: 3,594 calls/day for ~100 feed tokens today
Candle / backfill calls 26 per token per day MEASURED: (1,179 CoinGecko + 1,451 DexPaprika) / ~100 tokens
Trades per tracked token 2,000 per day average (heavy tail: hot memes 20k+) MEASURED: BSC ~82k rows/day over ~40 live pools on 09-22/23; Solana ~1,300/pool/day on 09-25
Stored bytes per trade row 1.2 KB MEASURED: 1,122 B BSC, 1,297 B Solana
Solana share of tracked tokens 50% C (meme flow; higher in the $1M+ band)
Solana wire bytes per trade 32 KB with today's per-tx fetch; ~3 KB on a filtered stream MEASURED 848 MB/day for ~26k rows; stream C
EVM tape requests one address-array eth_getLogs per 3 s per chain, 2 chains design target; flat in token count
Compression 8x zstd on JSONL tapes C
Telegram channels monitored 40 / 100 / 200 C, scaled with discovery breadth; T4 sets the list
Messages per channel per day 120 blended C: call channels 20–50/day, chats 300+/day; today's 5 channels give 28–127/day total
X posts per tracked token per day 30 from curated accounts C; X access is dark today
Extraction (Haiku-class) 20 messages per call; 1,500-token cached system prompt; 120 input + 40 output tokens per message C, consistent with the measured 430 in / 55 out per social-reader call for a single-token prompt
Stance (Sonnet-class) on the 20% of messages that name a tracked token; 1,000 input (last 5 messages by that caller + token card) + 120 output C
Model prices per 1M tokens Haiku 4.5 $1 in / $5 out / $0.10 cache read; Sonnet 5 $2 in / $10 out B: claude-api skill model table, cached 2026-06-24

5.2 The table

Metric 50 tokens 200 tokens 500 tokens
Requests/day: price (DexScreener, batched) 2,400 9,600 24,000
Requests/day: candles and backfill 1,300 5,200 13,000
Requests/day: EVM getLogs (flat) 57,600 57,600 57,600
Requests/day: Solana, today's per-tx design 55,000 220,000 550,000
Requests/day: Solana, stream design ~100 (reconnects) ~100 ~100
Messages/day ingested (Telegram + X) 6,300 18,000 39,000
Bandwidth down GB/day, stream design 0.4 1.5 3.8
Bandwidth down GB/day, today's design 1.9 7.3 18.3
Disk raw GB/day before compression 0.30 1.2 3.0
Disk GB/month capture, compressed 1.1 4.4 11
Disk GB/month derived (rebuildable) 1.3 5.2 13
CPU cores (estimate) 0.65 1.1 2.0
RAM GB (estimate) 1.8 2.7 4.5
LLM tokens/day, extraction (Haiku-class) 1.5 M 4.2 M 9.2 M
LLM tokens/day, stance (Sonnet-class) 1.4 M 4.0 M 8.7 M
LLM $/month, extraction $62 $177 $383
LLM $/month, stance $121 $346 $749

CPU and RAM assume the target design (collectors do only fetch and append; derivation incremental; API serves documents). Today's stack uses 1.3 cores for about 100 tokens, three quarters of it on re-reading, so the target is cheaper at 200 tokens than today is at 100. The box (28 cores, 64 GB, 567 GB free) carries 500 tokens with room; hardware is not a constraint at any of these sizes.

5.3 Money per month

Line 50 200 500 Basis
LLM (extraction + stance) $183 $523 $1,132 table above
Solana data $0–49 $49–99 $99–499 today's per-tx design needs 1.7 / 6.6 / 16.5 M calls a month: Helius Developer $49 (10M credits) covers 50 and 200, Business $499 needed at 500 (B, PROVIDERS.md 2026-09-16, Helius billing page). A filtered stream: Sodae free 5 GB/30 d covers 50; Chainstack $49/stream, Shyft ~$99, QuickNode $299 (B, same doc)
EVM RPC (BSC, Base) $0–49 $0–49 $49 today's public endpoints fail archive reads; NodeReal free key or one paid key (C)
Market data current plans current plans current plans CoinGecko Basic, DexScreener free; T1 owns the rights and price verdict
Fomo $49.99 $49.99 $49.99 Starter plan, expires 2026-10-10 (A, /v2/me, PROVIDERS.md)
Telegram / X access T4 / T1 T4 / T1 T4 / T1 X paid API tiers are the open line; not priced here
Hosting $0 marginal $0 $0 Bot-PC; a second box or VPS for redundancy is a D8 choice
Total excluding X ~$230–330 ~$620–720 ~$1,330–1,730

5.4 The three biggest cost drivers and how to cap each

  1. Solana trade capture. Per-transaction fetch grows linearly with trades and ran out of Helius credits in 8 days at ~20 pools. Cap: switch to a filtered subscription (logs or transactions for the tracked pool set), parse swaps from the stream, and use per-tx fetch only to repair gaps with a daily budget. Hard budget in the broker: credits per day per provider with a stop-and-mark-gap rule, never a silent exhaustion. Tape only pools above a liquidity floor and drop pools 72 h after a situation closes.
  2. LLM stance and extraction. Linear in messages, not tokens; whole-chat reading multiplies it. Caps: (a) regex and ticker/address prefilter so only messages naming a tracked or candidate token reach the model (cuts extraction by an estimated 60–80% in chat-heavy channels, C); (b) batch 20–50 messages per extraction call with a cached system prompt; (c) stance only when a caller's message names a tracked token, and only re-scored on change; (d) a per-day dollar cap per stage in the same ledger pattern lib/social-read/model.mjs already has, falling back to "stance unavailable" (printed, not guessed); (e) nightly batch re-scoring at the batch discount for anything not needed live.
  3. Re-computation on read. Today's largest CPU line: 0.97 cores re-reading a ledger and rebuilding an unread room. Cap: derivers are incremental and event-driven; read models are rebuilt only when an input changes; no endpoint scans a capture file; the research bench runs on its own schedule against Parquet, never inside the serving path.

6. Reliability design

6.1 What went wrong this week, and why nothing noticed

All five are the same failure: we measure whether a process runs, not whether data arrives.

6.2 Supervision model

  1. Every collector is a systemd user unit with Restart=always, RestartSec with backoff, MemoryMax and CPUQuota, WatchdogSec, and linger enabled so it survives reboot. No cron-launched long-runners, no transient units, no nohup.
  2. Progress heartbeat, not log heartbeat. Each collector writes heartbeat/<collector>.json every cycle: at, cursor (block, slot, message id, or print time per scope), rowsSinceLast, errorsSinceLast, budgetRemaining. It also calls sd_notify WATCHDOG=1 only when the cursor advanced, so a hung promise gets killed by systemd itself.
  3. Stall detection per scope. A supervisor (bash or Python, no LLM) compares each cursor with the source head: for a chain, head block minus the pool's covered block; for a price, now minus the latest providerAt; for a channel, the channel's own posting rate. Thresholds per class: tape lag over 5 min, price older than 3 polls, a Telegram channel silent for 3x its median gap. Breach state is written, then acted on.
  4. Restart policy. On stall: restart once; if stalled again within 30 min, stop restarting, mark the scope degraded, alert once, and switch to the fallback provider if one is configured. A failing scope (one pool, one channel) is quarantined without taking the collector down. Exponential backoff on provider errors, with the reason class stored (archive_unavailable, rate_limited, credits_exhausted, not_found).
  5. Dependency declared, not remembered. Units carry Requires=/After= for real dependencies, and the target design removes the ones that should not exist (collectors must not depend on the API server).
  6. Code-under-process guard. A service records the hash of its source at start; the supervisor flags any unit whose source changed since start, so a restart is a planned event with a test, not a surprise.

6.3 Gap accounting

Every collector writes coverage rows per scope: (collector, scope, from, to, status, reason). Gaps are first-class data, stored in the event store, and joined into every derived field and read model. The situation page prints them in place: "Observation gap 07:39 to 12:09: Solana tape stalled; flow and buyers for this window are missing." Derived numbers that span a gap carry partial: true and are never presented as complete. Backfill closes a gap and writes a new coverage row with status: backfilled and the backfill time; the original gap stays in the history, so replays remain honest about what a user could have seen at the time (the availableAt rule).

6.4 The reliability metric for the 14-day gate

The business plan's gate (ops critic, 09-25) should be measured from coverage rows, per collector class, daily:

Metric Definition Target
Observed-time share share of minutes in each tracked token's window covered by the tape with status ok ≥ 98% per chain per day
Max single gap longest gap per collector per day ≤ 15 min
Price freshness p95 of now minus latest providerAt at page-build time ≤ 3 min
Missed-event replay trades or crossings found by an independent daily refetch of 5 sampled pools that our tape missed ≤ 1%
Silent stall count stalls found by the replay or by hand that the supervisor did not flag 0
Credit runway days to exhaustion per paid provider at the 7-day burn rate ≥ 30 days

A day passes only if every row passes; the gate is 14 consecutive passing days, reset on failure. Today, by my reading of tonight's data, the BSC tape fails observed-time share and the Solana tape fails all six, so the gate clock has not started.

7. Build sequencing for the infra lane

One builder (Bolo), Vesper for specs and checks. Day estimates are honest working days for one builder including tests and a live soak, not calendar days. "Zero design input" means it does not wait for T5/T8.

# Item Depends on Days Starts
1 Stop the bleeding: retire the two crash-looping units, kill the five hung wrangler deploys, stop the BSC tape retrying out-of-window pools (mark archive_unavailable, stop), get a free NodeReal key for BSC logs none 0.5 tomorrow, zero design input
2 Supervisor and coverage rows: progress heartbeats, sd_notify watchdog, stall rules, coverage table, /status page; retrofit to the existing tapes, Fomo watcher and price poll none 2 tomorrow, zero design input
3 Event standard as a library (write path enforcing providerAt, availableAt, sha256, quality) and the raw sidecar; price collector moved onto it with unchanged and phantom flags (audit fixes 1–3) none; Bolo's staged items 1–3 are reusable 2 tomorrow, zero design input
4 EVM tape v2: one address-array getLogs per window per chain, coverage rows, BSC + Base, Robinhood on decision D4 2, 3 2 day 2–3
5 Solana tape v2: filtered subscription, swap parser for the main AMMs (Raydium, Pump AMM, Meteora, Orca), per-tx fetch only for gap repair with a credit budget 2, 3; provider decision D2 3–4 day 3
6 Storage layout + query layer: Parquet day close, DuckDB caverio.query, one candle schema from tape, migration of the tapes and events.sqlite (audit fix 9) 3, 4 2 day 4
7 Attention collectors: Telegram client session for callers and chats, message store, Fomo moved onto the event library 3; T4 source list 2 day 3, needs T4 only
8 Extraction and stance pipeline: prefilter, batched extraction, stance with versioned prompts, cost ledger and caps 7; T4 prompt spec 2–3 day 5
9 Derived fields v1: per-minute flow, buyers, holders, band membership and peer comparison, attention aggregates 6, 8; T5 field list 3 waits for T5
10 Situation store and lights 9; T5 rules 2 waits for T5
11 Macro collector + brief series 3; T3 metric list 2 waits for T3
12 Read models + API (documents per view, ETags, auth, per-user rate limits) 9, 10, 11; T8 wireframes 3 waits for T8
13 Retire the experimentation stack: server, price_paths (bench copy only), holdings, social-sweep/reader, proof services; archive frozen stores 12 live for 3 days 1 after 12
14 Gate instrumentation: daily replay against an independent refetch, the six metrics in §6.4 on /status 2, 6 1.5 day 5

Total: about 28–30 builder days for the whole lane; 8–9 days until items 1–6 give a supervised, gap-accounted tape and price store on the new standard, which is the foundation everything else sits on. This is consistent with the ops critic's 12–20 days for a much smaller scope, and it is not "a few days". What can honestly be done in a few days is items 1–3, and they should start tomorrow because they do not depend on any design choice.

8. Ten decisions for Thomas

  1. Label and freeze: confirm the whole of §1 is the experimentation stack, frozen except for the collectors listed "keep" and the tape rebuilds.
  2. Solana capture: move to a filtered stream (paid tier $49–299/month) instead of per-transaction fetch on Helius free.
  3. Storage: DuckDB over Parquet for tape and derived data, SQLite events, small relational store for situations and accounts (option B in §4.3), rather than one Postgres.
  4. Robinhood Chain: keep taping it, or drop it from the companion and cover Solana, BSC and Base only.
  5. Tracked-token ceiling for beta: 200 tracked tokens (about $620–720/month excluding X) as the design point, with 500 as the stretch.
  6. LLM caps: a daily dollar cap for extraction and for stance (suggest $10 and $15/day at 200 tokens), with "unavailable" shown when hit.
  7. Displayed price source: our own tape candles first, DexScreener as fallback and cross-check (also the rights answer T1 is checking).
  8. Redundancy: stay on Bot-PC for beta, or add a second small box for the collectors and the API.
  9. Retention: capture forever compressed, raw market payloads 90 days selective, derived 30 days (as in §3).
  10. Gate: adopt the six coverage metrics in §6.4 as the 14-day reliability gate, clock starting only after items 1–2 of §7 are live.

What I did not do

Status: DONE

HomeMASTERT1T2T3T4T5T6T7T8T9Wireframes