The reference.
Every endpoint,
every layer.
Architecture diagram, every API endpoint with request and response shapes, the admin dashboard, the simulator (both flavors), the test isolation contract, and the retry pattern your downstream service should follow.
Architecture, top to bottom
The system is layered: HTTP at the top, core orchestration in the middle, two storage backends at the bottom. Click any layer to see what lives there.
Browser and downstream service both talk to one Hono app on port 3000 — the same process serves the admin dashboard, the public API, and the Server-Sent Event stream. The core layer is where the algorithm lives. Storage is split deliberately: SQLite for things that must survive a restart, Dragonfly for hot runtime state that the router recomputes anyway.
Click a box in the diagram. The architecture has four levels: clients (browser + downstream), HTTP layer (Hono routes), core orchestration (PaymentRouter + helpers), and storage (Dragonfly + SQLite).
The split between Dragonfly and SQLite
Two storage backends, on purpose:
| backend | what's there | survives restart? | recomputable? |
|---|---|---|---|
| SQLite (drizzle) | provider configs, algorithm params, audit log | yes | no (source of truth) |
| Dragonfly (native client) | weights, metrics, circuit state, Lua script SHAs | yes (snapshots) | yes, in ~10s |
The 10-second background payment-router.ts:407 updates all weights from the current Dragonfly metrics — so a Dragonfly wipe is recoverable as soon as new traffic starts flowing.
Selection lives in Dragonfly as Lua
The provider-selection algorithm is a Lua script loaded once into Dragonfly at process
start via SCRIPT LOAD. Every /api/route call invokes the
script by 40-byte SHA via EVALSHA — atomic, server-side, no app-process
race conditions. If Dragonfly flushes its script cache, the helper at
redis-pool.ts:18 reloads and retries on
NOSCRIPT transparently.
API reference
Eight endpoints. Click any card to expand: parameters, sample request, sample response, what runs internally.
Base URL: http://localhost:3000 in dev. All endpoints listed below are
under /api/*. The admin routes (/admin/*) are separate — see
chapter 03.
Routing
GET /api/route Get a provider recommendation ▾
NGN. Providers without this currency are
excluded.
{
"provider": "stripe",
"transactionId": "abc-123-def",
"wasExploration": false,
"weight": 0.847,
"alternates": [
{ "provider": "paystack", "weight": 0.652 },
{ "provider": "flutterwave", "weight": 0.421 }
]
}
wasExploration is true when this was a ε-greedy random
pick (5% of requests). Useful for diagnosing why the chosen provider doesn't have
the highest weight.
Returned when every configured provider is either excluded, currency-mismatched,
amount-out-of-range, or circuit-open. The body includes the
transactionId so you can correlate.
curl$ curl "http://localhost:3000/api/route?amount=5000¤cy=NGN&excludeProviders=paypal"
{
"provider": "stripe",
"transactionId": "1e4f1a30-3a7a-4f...",
"wasExploration": false,
"weight": 0.847,
"alternates": [{"provider":"paystack","weight":0.652}]
}
Filters eligible providers (currency, amount, circuit-state), builds the exclusion
list in the Dragonfly ZSET, then invokes the cached Lua selection script via
EVALSHA. Random values for ε-decision and weighted-pick are generated
client-side and passed as args so the script stays deterministic for replication.
Cost: one Dragonfly round-trip. p95 ~3 ms.
Source: payment-router.ts:164 · redis-pool.ts:18
Metrics
POST /api/metrics/report Report a transaction outcome ▾
needsWebhook: true.
timeout · rejected · error. Helps the
circuit breaker categorize failures.
true, the outcome is provisional and a webhook will
follow.
{
"received": true,
"provider": "stripe",
"newWeight": 0.871,
"circuitState": "closed"
}
newWeight is the recomputed weight after this report. Handy for debug
logs.
curl · success$ curl -X POST http://localhost:3000/api/metrics/report \
-H "Content-Type: application/json" \
-d '{"transactionId":"abc-123","provider":"stripe","success":true,"latencyMs":342}'
{"received":true,"provider":"stripe","newWeight":0.871,"circuitState":"closed"}
curl · failure$ curl -X POST http://localhost:3000/api/metrics/report \
-H "Content-Type: application/json" \
-d '{"transactionId":"abc-123","provider":"stripe","success":false,
"latencyMs":5000,"errorType":"timeout"}'
Three things happen: recordTransaction updates the Dragonfly metrics
hash, evaluateOnResult on the circuit breaker may flip
closed → open on the 5th consecutive failure, and
updateProviderWeight recomputes the weight and pushes it into the
Dragonfly ZSET via Bun auto-pipelined commands (ZADD + history
ZADD + ZREMRANGEBYRANK in one round-trip).
If needsWebhook is set, the circuit breaker is not evaluated
yet — the final outcome is still pending. The transaction goes into
pendingWebhooks Map and a 30-second sweep checks for timeouts.
Source: payment-router.ts:226
POST /api/metrics/webhook Final outcome for an async transaction ▾
/api/metrics/report with
needsWebhook: true.
true = the async payment succeeded,
false = it failed at the provider.
{
"received": true,
"provider": "flutterwave",
"newWeight": 0.723,
"circuitState": "closed"
}
{
"received": false,
"duplicate": true,
"message": "Webhook for ... was already processed or timed out"
}
curl$ curl -X POST http://localhost:3000/api/metrics/webhook \
-H "Content-Type: application/json" \
-d '{"transactionId":"abc-123","provider":"flutterwave","confirmed":true}'
Idempotent. The router looks up transactionId in
payment-router.ts:73 pendingWebhooks. If
absent → late or duplicate, returns { duplicate: true }. If present →
removes it, records the webhook outcome (incrementing
successesFinal or failuresFinal), evaluates the circuit
breaker, recomputes the weight.
The same internal path is invoked by the 30-second timeout sweep when
webhookTimeoutMs (default 120s) elapses — there, the input is
synthetic { confirmed: false }.
Source: payment-router.ts:275 · timeout sweep at payment-router.ts:455
Health
GET /api/health Health check + configured-provider list ▾
{
"status": "healthy",
"providers": ["stripe", "paystack", "flutterwave"],
"pendingWebhooks": 3
}
Pings Dragonfly and lists enabled providers from the config store. Suitable for
liveness probes — also returns 503 with
{ status: "unhealthy", error: "..." } if Dragonfly is down.
curl$ curl http://localhost:3000/api/health
{"status":"healthy","providers":["stripe","paystack","flutterwave"],"pendingWebhooks":0}
Provider stats & control
GET /api/providers/weights Current weights for all providers ▾
{ "stripe": 0.847, "paystack": 0.652, "flutterwave": 0.421 }
Single Dragonfly ZRANGE … WITHSCORES. Cheap.
curl$ curl http://localhost:3000/api/providers/weights | jq
GET /api/providers/:name/stats Full metric breakdown for one provider ▾
{
"metrics": {
"requestCount": 1523,
"successesFinal": 1456,
"failuresFinal": 67,
"timeoutCount": 12,
"webhookPending": 3,
"webhookConfirmed": 234,
"webhookFailed": 8,
"latencyHistogram": [...], // last 1000 latencies
"circuitState": "closed",
"consecutiveFailures": 0
},
"weight": 0.847,
"weightComponents": {
"successRate": 0.956,
"latencyPenalty": 0.91,
"circuitMultiplier": 1.0,
"businessMultiplier": 1.0,
"finalWeight": 0.847
},
"circuitBreaker": {
"state": "closed",
"consecutiveFailures": 0,
"timeSinceLastFailure": 184000,
"timeSinceLastSuccess": 2000
}
}
The weightComponents object is the debug view of the formula — useful
when a provider's weight is suspiciously low and you want to know which factor is
dragging it down.
curl$ curl http://localhost:3000/api/providers/stripe/stats | jq '.weightComponents'
POST /api/providers/:name/circuit/open Force-open a provider's circuit ▾
Use case: scheduled maintenance, manual intervention while a provider is having known issues, or kicking a provider out during incident response.
curl$ curl -X POST http://localhost:3000/api/providers/stripe/circuit/open \
-H "Content-Type: application/json" \
-d '{"reason":"Stripe quarterly maintenance window"}'
Sets circuitState = 'open' on the provider's metrics in Dragonfly.
Weight is recomputed (becomes 0 via the multiplier).
Does not auto-recover via the 30-second sweep — manually-opened circuits
stay open until explicitly closed. Different behavior from auto-tripped circuits,
on purpose.
Source: circuit-breaker.ts · openCircuit
POST /api/providers/:name/circuit/close Re-enable a provider's circuit ▾
curl$ curl -X POST http://localhost:3000/api/providers/stripe/circuit/close
Sets circuitState = 'closed' and resets
consecutiveFailures = 0. The weight will reflect actual metrics again
— if the success rate is still bad, it'll still be low; the circuit just won't
gate traffic.
Error responses
| code | meaning | typical cause |
|---|---|---|
| 200 | success | normal path |
| 400 | bad request | missing required param, invalid amount, malformed body |
| 503 | no eligible providers | everything excluded / circuit-open / currency-mismatched |
| 503 | unhealthy | Dragonfly ping failed |
| 500 | server error | uncaught exception — see logs |
Admin dashboard
HTMX + SSE web app at /admin. No SPA build step. Renders server-side, swaps
panels client-side, streams live metrics.
The admin UI is a Hono-rendered HTML application using
HTMX with hx-boost
for SPA-like navigation. The four panels — providers, algorithm, circuit breaker,
history — live at their own URLs. Live metrics arrive via
/admin/events (SSE).
What each panel does
Providers
CRUD for the rows in the SQLite provider_config table. Add a provider
(name, type, currencies, amount limits per currency, business multiplier),
enable/disable without deleting, force-open or close the circuit, delete with
confirmation. Per-row controls update Dragonfly weights instantly — no restart, no cache
wait.
Algorithm
Tune the runtime knobs: Beta prior α/β, p95 target, decay constant k, ε exploration
rate, weight bounds, webhook timeout. Saved to SQLite (algorithm_config
table), pushed to Dragonfly cache, applied on next weight recompute (within 10s).
Circuit breaker
Global thresholds: failures-to-open, open duration, half-open probe weight, exponential
backoff toggle, max backoff cap. Saved to the circuit_breaker_config table.
History
Audit log of every config change. Who, when, what was the diff. Pulled from the
config_history table — every setProvider /
setAlgorithmConfig insert here before they touch SQLite.
Form validation
All admin forms use valibot (via @hono/standard-validator)
for server-side validation with client-side error display. Field-level errors render
inline with red borders; a toast surfaces the summary; values are preserved on rejection
so the user can fix and re-submit.
Auth
Google OAuth via better-auth. Set GOOGLE_CLIENT_ID +
GOOGLE_CLIENT_SECRET in .env. Restrict the user pool by
setting ALLOWED_EMAILS to a comma-separated list — anyone outside that list
is rejected after OAuth succeeds. In dev, set BYPASS_AUTH=true to skip the
OAuth dance entirely.
Two simulators, two reasons
CLI for driving load against your real router. Web for visualizing the algorithm in isolation.
The CLI simulator · bun run simulate
Generates synthetic transactions against your actual router with your actual provider
configuration. Hits /api/route and /api/metrics/report over
HTTP. Use this when you want to load-test or watch how your real configured providers
behave.
| flag | default | what it does |
|---|---|---|
--tps |
2 | transactions per second |
--duration |
120 | seconds to run; 0 = forever |
--verbose |
false | log every transaction |
--dry-run |
false | don't call the router, just show what would happen |
--base-url |
http://localhost:3000 | where to send the requests |
The success/latency behavior of each provider is configured in
scripts/simulate.ts — edit the
SIMULATED_PROVIDERS dict to test how the algorithm reacts to specific
provider profiles.
The web simulator · bun run simulator
Self-contained app on port 3001 with mock providers and a live dashboard. Doesn't use
your real provider config — runs against its own MockConfigStore with 4
hardcoded mock providers. Use this when you want to see the algorithm work:
weight bars adjusting in real time, circuit breakers opening, traffic redistributing.
| mock provider | success rate | avg latency | profile |
|---|---|---|---|
| provider-alpha | 95% | 500 ms | fast and reliable |
| provider-beta | 90% | 800 ms | balanced |
| provider-gamma | 85% | 1000 ms | moderate |
| provider-delta | 75% | 1500 ms | slow, less reliable |
Predefined scenarios
Click a scenario in the web simulator's toolbar and it applies a new set of mock-provider parameters, flushes the Dragonfly DB it owns, and reinitializes the router. Useful for reproducing specific stress patterns:
- Baseline — all providers at their defaults. The router should converge with provider-alpha winning most traffic.
- Provider failure — provider-alpha drops to 10% success. Watch the circuit breaker open, watch traffic redistribute.
- High latency — all providers 3× slower. Weights decrease proportionally; throughput drops; no circuits open.
- Partial outage — multiple providers degraded simultaneously. Traffic shifts to the survivors.
- Recovery — providers gradually return to baseline. Watch weights climb back, circuits close.
Every algorithm path (weight calc, Lua selection, circuit transitions, pending-webhook
idempotency, EVALSHA caching) is identical to the production code path. The only
"fake" part is what would have been the provider API call — replaced by
Math.random() against the configured success rate. Watching the simulator
dashboard is a faithful preview of how the router behaves under similar real traffic.
Test suite
67 tests across 4 files. Hono's app.fetch means no HTTP server, no
fixtures, no flakiness.
What's covered
| file | tests | coverage |
|---|---|---|
| weight-calculator.test.ts | 8 | weight calc, latency penalty, circuit-breaker multiplier |
| redis-pool.test.ts | 16 | provider selection, weight updates, Lua scripts, history tracking |
| metrics-store.test.ts | 17 | transaction recording, webhooks, metrics persistence |
| api.test.ts | 26 | HTTP endpoints, validation, circuit-breaker control |
| total | 67 | all green |
Isolation
Tests run against the same Dragonfly and a different SQLite file, sharing zero state with development:
The setup is in test/preload.ts:
test/preload.tsprocess.env.NODE_ENV = 'test';
process.env.REDIS_DB = '1';
process.env.DATABASE_URL = './sqlite/test.db';
And declared in bunfig.toml via
preload = ["./test/preload.ts"] so it runs before any test file imports.
API tests use app.fetch directly
No HTTP server is ever started during tests.
test/api.test.ts imports the Hono app and calls
app.fetch(request) in-process. This is the same surface a real HTTP request
would hit — Hono's fetch is the actual request handler — but with no
socket, no ports, no flakiness. Faster than supertest, cleaner than mocks.
test/api.test.tsimport app, { redis } from '../src/index.tsx'; async function request(path, options?) { const req = new Request('http://test' + path, options); return await app.fetch(req); } it('returns a provider', async () => { const res = await request('/api/route?amount=5000¤cy=NGN'); expect(res.status).toBe(200); });
Run them
terminalbun test # all 67 bun test test/api.test.ts # one file bun test --coverage # with coverage bun test --watch # re-run on save
Retry patterns for your downstream service
The router doesn't retry — your service does. Here's the decision tree your code should follow.
Retryable vs not
| error type | retry? | why |
|---|---|---|
| timeout | yes | provider hung — probably transient |
| HTTP 5xx | yes | provider's problem |
| HTTP 429 | yes (backoff) | rate-limited — give it air |
| network error (ECONN…) | yes | infrastructure layer |
| HTTP 4xx (not 429) | no | request is bad — won't get better |
| card declined | no | user's card; won't be different elsewhere |
| insufficient funds | no | same |
| fraud flagged | no | same |
Sketch in code
processPaymentWithRetry.tsasync function processPaymentWithRetry(order, maxAttempts = 3) { const excludeProviders: string[] = []; for (let attempt = 1; attempt <= maxAttempts; attempt++) { // 1. Ask the router (excluding what already failed) const route = await fetch(buildRouteUrl(order, excludeProviders)).then(r => r.json()); if (!route?.provider) throw new Error('No available providers'); // 2. Execute with your existing integration const start = Date.now(); const result = await executeWithProvider(route.provider, order); const latencyMs = Date.now() - start; // 3. Report the outcome — even on failure await fetch('/api/metrics/report', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ transactionId: route.transactionId, provider: route.provider, success: result.success, latencyMs, errorType: result.errorType, needsWebhook: result.needsWebhook, }), }); if (result.success) return result; if (result.needsWebhook) return result; // async — don't retry yet if (!isRetryable(result)) throw new Error(result.error); // 4. Exclude the failed provider and loop excludeProviders.push(route.provider); await sleep(Math.pow(2, attempt) * 100); // optional backoff } throw new Error('All providers failed'); }
If you retry across providers, each attempt should use a unique idempotency key
(typically ${orderId}-${attempt}) when calling the provider — otherwise
you may double-charge the customer if a provider's "timeout" actually went through.
The router doesn't enforce this; it's your service's job.