Your model does not crash when it goes wrong. It returns a 200 OK (the response code that means everything is fine), three times slower than last week, costing twice as much per answer, while your uptime check stays green the whole time.
That silence is the real problem with running ML in production. The standard fix is two tools splitting one job: Prometheus counts, Grafana shows.
Prometheus "scrapes" your running service, meaning it asks the service for numbers on a schedule you set and files each reading as a time series: a metric name, a timestamp, a value. Picture an ICU bedside monitor logging a patient's heart rate every few seconds, so you can read the trend instead of only the number on screen right now. Grafana is that screen. It draws the stored numbers as graphs and fires an alert when a line crosses a threshold you pick.
The part that makes this work for ML is labels: key-value tags attached to a metric, like model_version="v3". Same metric, sliced by tag. So instead of "response time went up", you get "response time went up on v3 and stayed flat on v2". That is the gap between knowing something is wrong and knowing what to roll back.
Worth counting once you are serving an LLM: requests per second, error rate, and time to first token, which is how long the user stares at nothing before any text appears.
Nobody fixes a cost spike or a quality drop they cannot see. Vital signs first, diagnosis second.
Quick check before you scroll: Why does Prometheus pull (scrape) metrics from your model instead of having your model push them in?
Full breakdown + the answer: frankduah.me/learnings/2026-08-26-prometheus-grafana-for-ml-model-observability
New here? I post a bite-size AI / ML concept like this every day - follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#Prometheus #AI #LLM #AIAgents #MachineLearning
The answer
Pulling lets Prometheus control the timing and frequency itself, and a failed scrape is an instant, built-in signal that a service is down - your model just has to expose a simple metrics endpoint, no extra logic needed to send data anywhere.