Since v7.4.0 every provider proxy-router continuously verifies that the models it sells actually work, and publishes the result as a models array on its /healthcheck endpoint. This page is the operational reference for that capability: what runs, what it costs, how to tune it, and how to read the report when something is unhealthy. For the one-time post-setup walkthrough, see Verify your provider setup.
The httpStatus and modelName report fields, the rate_limited error kind, probe pacing (MODEL_HEALTH_CHECK_PROBE_DELAY), and the manual POST /healthcheck/models/refresh trigger were added in the release after v7.4.0. On v7.4.0 nodes you get the report without those features.
What actually runs
On a schedule (default hourly, plus one sweep immediately at startup), the proxy-router walks the union of:
- every model in
models-config.json, probed only if it has an active on-chain bid from your wallet, and
- every active bid whose model is missing from
models-config.json — capacity you are selling but cannot serve, reported as no_model_configured.
An LLM probe is a real, tiny chat completion sent directly to your backend apiUrl — a randomized arithmetic question (“What is 17+34? Reply with only the number.”), graded into promptCorrect. Embedding models get a one-input embeddings request. Other model types (TTS, image, video) are not probed and report skipped. Results are cached in memory between sweeps.
Two consequences worth internalizing:
- The probe validates your backend wiring, not your public path. It goes straight from the proxy-router to the
apiUrl in models-config.json — it does not traverse :3333, sessions, or the marketplace. A model can be healthy here while your node is unreachable from the internet (check that separately, see verify-setup).
- A configured model without a bid is not probed (
no_bid). A typo’d apiUrl stays invisible until you post a bid — probe-test new models by posting the bid first, or hit the backend manually.
Configuration and defaults
| Variable | Default | What it controls |
|---|
MODEL_HEALTH_CHECK_DISABLED | false | Set true to disable the loop entirely; the models field disappears from /healthcheck |
MODEL_HEALTH_CHECK_INTERVAL | 1h | How often the full sweep runs; results are cached between sweeps |
MODEL_HEALTH_CHECK_TIMEOUT | 60s | Per-model probe timeout; slower probes report errorKind: timeout |
MODEL_HEALTH_CHECK_PROBE_DELAY | 2s | Pause between consecutive probes within a sweep |
Impact on your node and your backend
The load on the proxy-router itself is negligible — one lightweight request at a time, sequentially. The cost that matters is downstream, on your backend:
- Every probe is a real completion against your upstream. With 65 models on an hourly sweep that is ~1,560 tiny (≈10-token) completions per day. Against a metered upstream this is real but usually negligible billing; raise
MODEL_HEALTH_CHECK_INTERVAL if you care.
- Probes share your upstream rate-limit budget with real consumer traffic. This is why pacing exists: many resale providers fan all models onto a single upstream account (for example a LiteLLM node in front of Venice), and an unpaced sweep of 60+ models fires several requests per second at it — enough to trip a per-minute rate limit, mark your own models
rate_limited, and throttle paying consumers at the same time. The default 2s delay keeps a 65-model sweep at ~0.5 req/s over roughly 3–5 minutes. If your upstream is particularly strict, raise the delay; you have a whole hour of headroom before it affects the schedule.
- A sweep takes about
models × (probe latency + probe delay). Keep interval comfortably larger than that.
Reading the report
modelName is the model’s public on-chain registered name — looked up from the model registry, never your private modelName from models-config.json (which, like apiUrl and apiKey, is never exposed).
| Status | Meaning |
|---|
healthy | Probe succeeded; for LLMs promptCorrect says whether the answer was right |
unhealthy | Probe failed — errorKind and httpStatus say why (see below) |
no_bid | Configured but no active bid; not probed |
no_model_configured | Active bid but no models-config.json entry — consumers can open sessions against this bid and get errors |
skipped | Model type not probeable (TTS, image, video) |
unhealthy models, the diagnosis is the combination of errorKind and httpStatus:
| errorKind | httpStatus | Almost always means |
|---|
connection | — | Backend apiUrl unreachable: wrong host/port, container down, DNS |
timeout | — | Backend reachable but slower than MODEL_HEALTH_CHECK_TIMEOUT |
rate_limited | 429 | Backend up and correctly configured, just throttled — often transient; if persistent, your upstream quota is too small for your traffic |
bad_response | 402 | Upstream account out of credits / payment required — a billing problem, not a node problem |
bad_response | 400 / 404 | Upstream rejected the request: wrong model name in models-config.json, model retired upstream, malformed base URL |
bad_response | 401 / 403 | Bad or expired apiKey |
bad_response | 5xx | Upstream itself is failing |
bad_response | (absent) | Backend returned 200 but an empty/invalid completion |
healthy model with promptCorrect: false deserves attention too: the backend responds, but the model can’t answer 2-digit addition — usually the apiUrl/modelName pair is wired to the wrong model.
Diagnosing locally — the tricky parts
The report is cached — trigger a re-check instead of waiting. lastChecked (unix epoch — date -r 1783536653 on macOS, date -d @1783536653 on Linux) tells you how stale each entry is. After fixing a backend, queue an immediate sweep:
The call returns 202 immediately and the sweep runs in the background — with probe pacing, a many-model sweep takes a few minutes, so poll GET /healthcheck and watch lastChecked move. Triggers don’t stack: while one manual sweep is queued or running, another POST answers "refresh already pending" — you cannot hammer your own backend through this endpoint. It requires basic auth (method permission model_health_refresh; the admin user’s * whitelist covers it). On older releases without the endpoint, restart the proxy-router instead — a sweep always runs at startup.
Timestamps carry the history. lastHealthy survives failures: an unhealthy model with a recent lastHealthy broke recently (billing, throttling); one with lastHealthy: 0 has never worked since the router started (misconfiguration).
The probe result can differ from what consumers experience. The probe hits the backend directly. If it’s healthy but consumers report failures, the problem is in front: :3333 reachability, on-chain endpoint registration, or session handling. Work through verify-setup steps 2–5.
Checking your node from the outside. /healthcheck lives on the :8082 admin API, which must not be exposed publicly. To read a provider’s self-report remotely, ask any C-Node to relay it over the Morpheus TCP protocol — the ping RPC returns the same models array:
Use the provider’s own registered wallet and endpoint exactly as they appear on-chain (/blockchain/providers) — a mismatched pair fails signature verification.
Quick triage one-liner — group your unhealthy models by cause:
If most rows share one errorKind/httpStatus (say bad_response/402), you have one upstream problem, not N model problems.
Turning it off
Set MODEL_HEALTH_CHECK_DISABLED=true only if you genuinely cannot afford the probe traffic. The report is how the network — and your own monitoring — can tell a billing failure from a dead node from a rate limit before consumers burn sessions discovering it for you. 
