E2E testing

When to use: validating the full Docker stack before pushing; catching regressions at the service-boundary level.

What the suite covers

Every Rust service image is built from source. The stack starts with migrations, SQL fixtures load, Playwright scenarios run against live ports, then the stack tears down.

Component	State	Notes
gordon-data	real	Ingests from mock-binance via spot REST + WS overrides
gordon-risk	real	Full portfolio risk engine, `/healthz` probed
gordon-manager	real	Reconciles bot containers via docker-socket-proxy
gordon-bot	real	Runs `GORDON_BOT_STRATEGY=manual` + scripted candle source for scenarios
gordon-executor	real	Full order-submit + reconcile + flatten
gordon-console	real	Next.js standalone bundle, three variants (operator/auth-off/no-token)
Binance exchange	mocked	`mock-binance` on ports 8181/8182 serves Binance-shaped REST + WS from scenario YAMLs

As of e2e-fidelity 03 (2026-04-20), SQL fixture seeding for trading.* rows has been removed. The real pipeline writes every trading.* row during the test run. No hand-authored fixture UUIDs in assertions.

Running the suite

Full run

bash

make e2e

Sequence: preflight checks + sequential image builds + stack start + fixtures + Playwright scenarios + teardown.

Re-run Playwright only (stack already up)

bash

cd gordon-console && GORDON_CONSOLE_URL=http://localhost:3000 pnpm playwright test --workers=1

Run a single spec file

bash

cd gordon-console && pnpm playwright test e2e/manager/bot-lifecycle.spec.ts --workers=1

Run by test title

bash

cd gordon-console && pnpm playwright test --grep "should transition to running" --workers=1

--workers=1 is mandatory. The executor and manager scenarios share mock-binance admin state and trading.order_intents rows. Parallel workers produce false failures. This constraint is not negotiable — it was the root cause of the 2026-04-30 postmortem.

Targeted-test loop (recommended for iteration)

Do not re-run make e2e on every spec change — rebuilding all images takes minutes. Instead:

Build images once:
bash
```
make e2e-build
```
Start the stack (one-time per session):
bash
```
make e2e-up
```
Load the shell env for Playwright:
bash
```
eval "$(make e2e-shell-env)"
```

Run only the specs you are working on:

bash

cd gordon-console
pnpm playwright test e2e/write/risk-flatten.spec.ts --workers=1

When finished:
bash
```
make e2e-down
```

This loop is equivalent to make e2e but avoids rebuilding unchanged images.

Soak mode

For multi-hour observation at realistic cadence (1 tick per real minute instead of 200 ms):

bash

make e2e-build       # build images once
make e2e-soak-up     # start with soak overlay (MOCK_BINANCE_KLINE_TICK_MS=60000)
make e2e-logs        # observe
make e2e-down        # tear down when done

Calibrate retention policies and alert thresholds against soak cadence numbers, not fast e2e numbers.

Mock-binance fixtures

Scenario YAMLs live in tests/common/mock-binance/scenarios/. The mock serves Binance-shaped REST and WS responses. Key scenarios:

happy_path.yaml — standard fills, mark price $50,000
quarantine_on_anomalies.yaml — 6 orphan orders for reconcile-quarantine testing

Binance conformance suite

Checks that mock-binance and real Binance testnet shapes are in sync:

bash

export BINANCE_TESTNET_API_KEY=...
export BINANCE_TESTNET_API_SECRET=...
make e2e-binance-conformance

# Mock-only (no credentials required)
make e2e-binance-conformance-mock-only

Run before releasing any gordon-exchange change that touches REST/WS shapes.

Targeted scenario scripts

Individual bash scenarios that run against a live stack:

bash

make e2e-money-path          # baseline: entry fills, SL attached, trade row lands
make e2e-emergency-flatten   # POST /risk/emergency-flatten + resume gate
make e2e-risk-drawdown       # DrawdownBreaker trip + halt-gate
make e2e-risk-connectivity   # ConnectivityBreaker
make e2e-risk-vpin           # VPINBreaker
make e2e-quarantine          # quarantine state machine
make e2e-executor-cluster    # all executor-cluster scenarios

Operator token

The emergency-flatten endpoint requires x-operator-token. In e2e:

test-token-13-11b

Hardcoded in docker-compose.yml as OPERATOR_TOKEN: test-token-13-11b. Production tokens are injected via Ansible vault and never committed.

Inspecting state after a failure

Service logs

bash

make e2e-logs                                          # all services
docker compose logs gordon-manager --tail=200
docker compose logs gordon-executor --tail=200
docker compose logs gordon-risk --tail=200
docker compose logs mock-binance --tail=200

Database state

bash

docker compose exec postgres psql -U gordon -d gordon
SET search_path = trading, market_data, public;

-- Bot configs
SELECT id, bot_name, desired_state, updated_at FROM bot_configs ORDER BY created_at;

-- Recent bot events
SELECT id, bot_id, event_type, created_at FROM bot_events ORDER BY created_at DESC LIMIT 20;

-- Orders
SELECT id, status, symbol, side, created_at FROM orders ORDER BY created_at DESC LIMIT 20;

Health check

bash

make e2e-status

Playwright trace

bash

cd gordon-console
npx playwright show-trace test-results/<scenario>/trace.zip

Common failure decision tree

BuildKit EOF / daemon died

io: read/write on closed pipe
ERROR: failed to build: ... EOF

Check for parallel builds: grep -n "compose.*--parallel" Makefile — expect no matches.
Check disk: make e2e-preflight.
Run make clean-workspace, then make e2e.

If the crash left orphaned containers: make e2e-recover.

kellnr_token secret missing

e2e preflight failed — KELLNR_TOKEN not set or empty

bash

make local-env-e2e
# re-source .env or open a new terminal
make e2e-preflight && make e2e

Port already in use

bash

lsof -i :5432
docker compose -f docker-compose.test.yml stop   # if gordon-postgres-test holds it
make e2e

Stack ports: 5432 (postgres), 8081–8085 (v7 services), 8181–8182 (mock-binance), 3000 (console).

Service /healthz never reaches 200

bash

make e2e-status
docker compose logs gordon-migrate --tail=50   # migration failures block everything
docker compose logs <service> --tail=100

Standard fix:

bash

docker compose down -v --remove-orphans
make e2e

Playwright assertion error

bash

cd gordon-console
npx playwright show-trace test-results/<scenario-dir>/trace.zip

Check for parallel-workers violation first: always run with --workers=1.

Disk management

make e2e auto-prunes build cache to 8 GB before building. If still hitting disk pressure:

bash

make e2e-cache-clean        # aggressive: removes ALL build cache + dangling images
make clean-workspace        # reclaims target/ + stale layers (20-100 GB first run)
DRY=1 make clean-workspace  # dry-run first

Never use docker system prune -a — it nukes postgres data volumes.

Make targets — full e2e target listing
Troubleshooting — service startup and NATS failures
Dev stack — host-only dev loop (no Docker required)

E2E testing ​

What the suite covers ​

Running the suite ​

Full run ​

Re-run Playwright only (stack already up) ​

Run a single spec file ​

Run by test title ​

Targeted-test loop (recommended for iteration) ​

Soak mode ​

Mock-binance fixtures ​

Binance conformance suite ​

Targeted scenario scripts ​

Operator token ​

Inspecting state after a failure ​

Service logs ​

Database state ​

Health check ​

Playwright trace ​

Common failure decision tree ​

BuildKit EOF / daemon died ​

kellnr_token secret missing ​

Port already in use ​

Service /healthz never reaches 200 ​

Playwright assertion error ​

Disk management ​

Related ​

E2E testing

What the suite covers

Running the suite

Full run

Re-run Playwright only (stack already up)

Run a single spec file

Run by test title

Targeted-test loop (recommended for iteration)

Soak mode

Mock-binance fixtures

Binance conformance suite

Targeted scenario scripts

Operator token

Inspecting state after a failure

Service logs

Database state

Health check

Playwright trace

Common failure decision tree

BuildKit EOF / daemon died

kellnr_token secret missing

Port already in use

Service /healthz never reaches 200

Playwright assertion error

Disk management

Related