Monitoring an NInfer-3090 setup

A drop-in monitoring bundle for the NInfer-3090 inference server: one Prometheus scrape job and one Grafana dashboard. No Nix, no provisioning tooling — plain config files.

What's in this repo

File	Purpose
ninfer-scrape.yml	Prometheus scrape jobs: ninfer /metrics, plus an (optional) NVIDIA GPU exporter
ninfer-dashboard.json	The Grafana dashboard — import it through the Grafana UI

How it fits together

ninfer serves /metrics on its normal API port — no extra flag, no extra port; it is just another route next to /v1/chat/completions. Prometheus scrapes that endpoint; Grafana queries Prometheus. Grafana never talks to ninfer directly, so the whole setup is three pieces:

1. a scrape job in Prometheus,
2. a datasource in Grafana pointing at that Prometheus,
3. the imported dashboard.

Setup

0. Sanity check

curl -s -H "Authorization: Bearer <api-key>" http://<ninfer-host>:<port>/metrics | head

You want ninfer_* lines back.

• 503 — the weights are still loading (up to ~2 min after a start; every route, /metrics included, answers 503 until the engine is ready). Retry later.
• 401/403 — missing or wrong API key.

1. Point Prometheus at ninfer

Put ninfer-scrape.yml next to your prometheus.yml and let the main config pick it up:

scrape_config_files:
  - "ninfer-scrape.yml"

(or just copy the job into your existing scrape_configs list). Then:

• Set targets to wherever ninfer listens. In docker-compose that is the service name if both containers share a network, or host.docker.internal:<port> if ninfer runs on the host.
• If you run ninfer with --api-key (you should): the key gates /metrics too. Write the key to a file readable by the prometheus user (docker: volume-mount it) and point credentials_file at it. Running without a key? Delete the authorization block.
• Reload without restarting: curl -X POST http://localhost:9090/-/reload (or restart the container).
• Verify: Prometheus UI → Status → Targets → ninfer is UP, or query up{job="ninfer"} and expect 1.

2. Make sure Grafana can see that Prometheus

Settings (gear) → Data sources. If a Prometheus datasource already exists, check its URL hits the Prometheus from step 1 and mark it default. Otherwise add one — URL http://localhost:9090, or the compose service URL such as http://prometheus:9090 — Save & test, then set it as default.

Verify in Explore: query ninfer_engine_ready and expect 1.

3. Import the dashboard

Dashboards → New → Import → paste the contents of ninfer-dashboard.json (or upload the file) → select your Prometheus datasource → Import.

4. (Optional) GPU panels

The four GPU panels (memory, utilisation, power, temperature) come from an nvidia-smi exporter (nixpkgs package: prometheus-nvidia-gpu-exporter), not from ninfer. It listens on :9835 by default; run it, then un-comment the nvidia-gpu job in ninfer-scrape.yml and reload. Without it, those panels simply show no data.

Worth running even on a single-card box: it's one small process, and the VRAM panel is the one that shows what else is squatting on the card beside ninfer (on WSL, a non-trivial share is held by Windows).

How to run it:

• Docker (with the nvidia container toolkit installed): docker run -d --name nvidia-gpu-exporter --gpus all -p 127.0.0.1:9835:9835 utkuozdemir/nvidia_gpu_exporter
• Package / binary: the releases ship .deb/.rpm and bare binaries (amd64/arm64). On normal Linux the exporter just uses the nvidia-smi that ships with your driver — keep it running however you keep other services up: nvidia_gpu_exporter --web.listen-address 0.0.0.0:9835
• WSL: here nvidia-smi is a Windows-side binary under /usr/lib/wsl/lib that Linux can't exec directly. Point the exporter's --nvidia-smi-command at a shim that runs it through glibc's dynamic loader, e.g. a script with: export LD_LIBRARY_PATH=/usr/lib/wsl/lib; exec /lib64/ld-linux-x86-64.so.2 /usr/lib/wsl/lib/nvidia-smi "$@" — and make sure the service may access /dev/dxg (WSL's GPU device node).

What to expect

• Every panel has a description (hover its title) explaining the metric.
• Engine stat: ready / loading percentage / not ready / down. The percentage during the ~2-minute weight-load window is an extra: the original setup publishes ninfer_weights_load_percent by parsing the engine's startup journal into node-exporter's textfile collector. Without that script the stat just reads down while loading — everything else still works.
• After every start (and whenever the host is off) the target legitimately reads down for the load window. The dashboard's annotation marks restarts.
• Metric names are pinned to the engine build the dashboard was written against (Qwen3.8-27B, DFlash2 profile). With a different build some counters may not exist; those panels read no data rather than error.
• Written for an RTX 3090, but card-agnostic except one panel: 3090 VRAM used filters on nvidia_smi_gpu_info{name=~".*3090.*"}. On a different card (say a 4090), edit that regex in the panel editor; everything else works as-is.
• Decode/prefill speeds are rates over execution time only: gaps between requests are idle time, not slowness.

Troubleshooting

Symptom	Likely cause
Every panel: No data	Prometheus isn't receiving metrics — is the target UP (step 1)?
Target DOWN but curl works	API key — credentials_file missing/unreadable, or wrong key
Dashboard imported, panels empty	Your engine build exports different metric names
Datasource errors in Grafana	Wrong URL (in compose use the service name), or Prometheus unreachable from Grafana