Prometheus

What is Prometheus?

Prometheus is the de-facto standard open-source metrics monitoring system. It works by scraping (pulling) numeric time-series data from your services’ /metrics endpoints at intervals, storing them in a purpose-built time-series database, and letting you query them with PromQL — a powerful language for slicing, aggregating, and computing rates over metrics. It’s a graduated CNCF project and the metrics backbone of most Kubernetes stacks. Crucially, Prometheus is for metrics — counters, gauges, latencies, error rates — the vital signs of a system, not its logs or request traces.

Pull model, and the three pillars

Two things define Prometheus in practice. First, the pull model: rather than services pushing metrics out, Prometheus reaches in and scrapes them, which simplifies service configuration and makes “is this target up?” a first-class signal (a target that can’t be scraped is visibly down). Push-based workflows (short-lived batch jobs) use a Pushgateway as a workaround. Second, its place among the three pillars of observability: metrics (Prometheus), logs (Loki, Elasticsearch), and traces (Jaeger, Tempo). Prometheus deliberately owns one pillar and does it well — a common mistake is expecting it to store logs or traces, which it doesn’t. It pairs almost universally with Grafana for dashboards and with Alertmanager for routing alerts.

Operational reality

Prometheus is powerful but has real edges. It’s fundamentally a single-node system with local storage — long-term retention and horizontal scale require add-ons like Thanos, Mimir, or Cortex that build a global, durable layer on top. Cardinality is the classic footgun: every unique combination of label values creates a new time series, so a label like user_id or request_id can explode the series count and overwhelm memory — high-cardinality labels are the number-one way teams blow up their Prometheus. Metric and label design discipline is what separates a healthy deployment from an unstable one.

What people get wrong

• Expecting logs or traces — Prometheus is metrics-only; pair it with Loki/Jaeger for the other pillars.
• High-cardinality labels: putting unbounded values (user IDs, request IDs) in labels explodes series count and crashes the server.
• Assuming it scales horizontally out of the box — long-term, highly-available Prometheus needs Thanos/Mimir/Cortex.

Primary source: Prometheus documentation

Prometheus’s Pull-Based Architecture

A distinguishing design choice in Prometheus is its pull-based metrics collection model: rather than applications pushing metrics to a central collector (the more traditional approach used by many earlier monitoring systems), Prometheus itself periodically scrapes metrics from HTTP endpoints exposed by monitored applications and services. This pull model makes it straightforward to verify a target is being monitored correctly (Prometheus’s own dashboard shows exactly which targets it’s successfully scraping and when), and it naturally handles the common case of ephemeral or dynamically scaled infrastructure through service discovery integrations, automatically detecting new instances to scrape as they come online in environments like Kubernetes without requiring each new instance to be manually registered with a central collector.