gordon-data

Purpose

gordon-data is the sole writer of the market_data.* schema. It ingests live market feeds from Binance (REST + WebSocket), Deribit, alternative.me, DefiLlama, and FRED; precomputes higher-timeframe candles (5m through 1w) from the canonical 1m source via pure SQL; and publishes KlineEvent payloads to the NATS JetStream bus via an atomic dual-write outbox. Every downstream consumer — bots, manager, risk, executor, console, lab — reads market data through either the REST surface or, in the case of gordon-risk and gordon-manager, named Postgres views. There is no consumer-facing WebSocket; that path was deleted in Wave 2 of the event-bus migration (2026-05-08).

Version + port + env var

Field	Value
Version	3.6.0
Port	8081
Env override	GORDON_DATA_BIND_ADDR
DB role	gordon_data_writer
Image	ghcr.io/dlepaux/gordon-data

HTTP endpoints

Health / ops

Method	Path	Purpose
GET	/healthz	Liveness probe
GET	/readyz	Readiness probe (DB + role probe must pass)
GET	/metrics	Prometheus metrics
GET	/config	Redacted config dump
GET	/openapi.json	OpenAPI spec

Business endpoints

Group	Example paths	Purpose
Klines	GET /klines, GET /perp_klines	Spot and perp OHLCV with timeframe + symbol params
Derivatives	GET /funding, GET /open_interest, GET /long_short_ratio	Derivatives data by symbol + range
Sentiment	GET /fear_greed, GET /stablecoin_supply, GET /metrics_data	Macro + sentiment indicators
GEX	GET /gex	Gamma exposure (Deribit)
Liquidations	GET /liquidations	Binance liquidation feed
Macro	GET /macro	FRED macro data
Warmup	POST /warmup	Bot warmup — returns strict-completeness candle slice
Inventory	GET /inventory	Symbol + timeframe availability
Sources health	GET /sources/health	Per-source freshness rows (source_freshness table)
Sources schedule	GET /sources/schedule	Scheduler run log
Backfill triggers	POST /backfill/{source}	Trigger backfill job per source (spot_klines, perp_klines, funding_rates, fear_greed, stablecoin_supply, gamma_exposure, metrics, precompute_klines, open_interest)
Backfill jobs	GET /backfill/jobs, GET /backfill/jobs/{job_id}, DELETE /backfill/jobs/{job_id}	List, inspect, cancel backfill jobs
Backfill reports	GET /backfill/{source}/report	Coverage report per source

NATS subjects

Pattern	Role	Direction
market.klines.binance.spot.{symbol_lc}.{tf}	Publishes KlineEvent via leader-elected outbox	Publisher
market.klines.binance.perp.{symbol_lc}.{tf}	Perp klines (if wired)	Publisher

Stream: gordon-bus, 168h retention. Dual-write: KlineEvent is written atomically inside the same sqlx transaction as the spot_klines INSERT via gordon_bus::nats::outbox_publisher. No durable consumers owned by gordon-data.

Database access

Action	Detail
Writer	market_data.* — sole writer, enforced via gordon_data_writer role
Role probe	Startup verifies positive + negative privileges before binding HTTP; connecting as gordon superuser aborts
Views used	None — data service owns the tables, others read via views

Prometheus metrics

• data_fetch_total
• data_fetch_duration_seconds
• data_rows_written_total
• gordon_data_source_quarantined_total
• gordon_data_source_unquarantined_total
• gordon_data_source_command_unknown_variant_total
• gordon_data_source_gap_seconds
• gordon_data_source_gap_seconds_by_symbol
• gordon_errors_total

Key env vars

Variable	Purpose
GORDON_DATA_BIND_ADDR	HTTP bind address (default :8081)
GORDON_DATABASE_URL	Postgres connection string (must use gordon_data_writer role)
GORDON_BUS_NATS_URL	NATS JetStream URL for outbox publisher
BINANCE_API_KEY / BINANCE_API_SECRET	Read-only Binance keys (startup probe rejects trading-capable keys)
FRED_API_KEY / FRED_API_KEY_ALIAS	FRED macro data fetch

Invariants

• Sole writer. market_data.* writes enforced at the DB layer. No other service may INSERT into this schema.
• Read-only Binance keys only. Startup probe queries /sapi/v1/account/apiRestrictions; aborts if any enable_* flag other than enable_reading is true. Key identity is separate from gordon-executor's trading keys.
• Fail loud on missing precomputed timeframe. Never returns silent empty arrays for an unmaterialized timeframe.
• Aggregation rules are a workspace contract. Rust and Python (lab) must produce bit-identical candles. See workspace CLAUDE.md klines pipeline section.
• Outbox atomicity. KlineEvent is dual-written (DB row + bus outbox) inside one transaction. A partial write is a bug.
• No /ws endpoint. The consumer-facing WebSocket was deleted in Wave 2 (2026-05-08). The live tail for bots runs via NATS JetStream.

Status

Phase 2 (Data): 16/16 complete (2026-04-19). Backend close-out audit passed (2026-04-22) — zero #[allow], clippy pedantic + nursery clean. Stable.

OBS-6 (2026-05-17): WS sources (BinanceKlines, BinancePerpKlines, BinanceLiquidations) now write source_freshness rows via a 1 Hz health-metrics-refresh task, closing the console blind spot for WS-continuous sources.

Related

• Architecture: event-bus-topology
• Architecture: GEX pipeline
• Reference: error codes
• gordon-bot — primary NATS consumer of market.klines.>
• gordon-risk — reads market_data.* via named views