Prometheus

Prometheus is the CNCF graduated monitoring and alerting toolkit, featuring a multi-dimensional data model with time-series metrics, PromQL, and native OpenTelemetry ingestion.

When to Use

• Cloud-Native Time-Series Monitoring: Scraping, storing, and querying numerical metrics from infrastructure and applications.
• PromQL Quantitative Querying: Computing request rates, 99th percentile latencies, error ratios, and saturations.
• Automated Service Discovery: Dynamically discovering Kubernetes pods, EC2 instances, and Consul nodes to scrape.
• Alerting Rules & Alertmanager Integration: Evaluating threshold rules and dispatching alerts to on-call engineers.

Quick Start

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

Core Concepts

Prometheus Scrape Configuration (prometheus.yml)

Configuring scrape jobs with relabeling:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "api-microservices"
    scrape_interval: 10s
    metrics_path: "/metrics"
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ["production"]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        action: keep
        regex: "8080"

Production Alerting Rules (alert_rules.yml)

Defining SLO-based alert thresholds:

groups:
  - name: API_SLO_Alerts
    rules:
      - alert: HighHttpErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP 5xx error rate on {{ $labels.service }}"
          description: 'Service {{ $labels.service }} error rate is {{ $value | printf "%.2f" }}% (> 5% SLO threshold).'

Exposing Custom Metrics in Python

Instrumentation using official Prometheus Python client:

from prometheus_client import start_http_server, Counter, Histogram
import time

REQUEST_COUNTER = Counter(
    'http_requests_total',
    'Total count of HTTP requests processed',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Histogram of HTTP request latency in seconds',
    ['endpoint'],
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
)

@REQUEST_DURATION.labels(endpoint='/api/orders').time()
def process_order():
    time.sleep(0.08)
    REQUEST_COUNTER.labels(method='POST', endpoint='/api/orders', status='200').inc()

# Start metrics endpoint server on port 8000
start_http_server(8000)

Common Patterns

High-Throughput Scrape Config with Kubernetes Service Discovery

Problem: Static IP scraping configurations fail as pods scale dynamically in Kubernetes.

Solution: Use kubernetes_sd_configs with metric path annotations:

scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

Best Practices

Do:

• Adhere strictly to the Four Golden Signals (Latency, Traffic, Errors, Saturation) when building PromQL alert rules.
• Configure appropriate histogram buckets around your service SLO targets (e.g. 100ms, 200ms, 500ms).
• Use recording rules (record: job:metric:rate5m) for complex PromQL expressions queried by dashboards.
• Store long-term historical metrics using remote-write storage (Thanos, Cortex, or VictoriaMetrics).

Don't:

• Introduce high-cardinality label values (user IDs, email addresses, order IDs) into metric labels.
• Set scrape intervals too low (< 5s) across large fleets; it overloads target scrapers and storage.
• Alert on simple transient spikes; always use for: 2m or for: 5m to filter temporary noise.

Troubleshooting

References

• Prometheus Documentation