Appendix C · PromQL Recipes for CDN Monitoring

A cookbook of production-ready PromQL queries for CDN observability. Each query assumes the metric names from Lab 20. Adapt label names to your actual instrumentation.

Hit Ratio Queries

Request-level hit ratio (5-minute window)

rate(cdn_requests_total{cache="hit"}[5m])
/
rate(cdn_requests_total[5m])

Byte hit ratio (more meaningful for billing)

rate(cdn_bytes_served_total{cache="hit"}[5m])
/
rate(cdn_bytes_served_total[5m])

Hit ratio by PoP

rate(cdn_requests_total{cache="hit"}[5m]) by (pop)
/
rate(cdn_requests_total[5m]) by (pop)

Hit ratio trend (1-hour average over past 24 hours)

avg_over_time(
  (
    rate(cdn_requests_total{cache="hit"}[1h])
    /
    rate(cdn_requests_total[1h])
  )[24h:1h]
)

Latency Queries

p50, p95, p99 request latency (across all requests)

histogram_quantile(0.50, sum(rate(cdn_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(cdn_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(cdn_request_duration_seconds_bucket[5m])) by (le))

p99 latency, split by cache status

histogram_quantile(0.99,
  sum(rate(cdn_request_duration_seconds_bucket[5m])) by (le, cache)
)

This reveals the latency gap between cache hits and misses.

p99 origin latency (time spent fetching from origin)

histogram_quantile(0.99,
  sum(rate(cdn_origin_duration_seconds_bucket[5m])) by (le)
)

Latency heatmap (for Grafana heatmap panel)

sum(rate(cdn_request_duration_seconds_bucket[5m])) by (le)

Traffic Volume Queries

Requests per second

sum(rate(cdn_requests_total[1m]))

Requests per second by status code class

sum(rate(cdn_requests_total[1m])) by (status)

Bytes served per second

sum(rate(cdn_bytes_served_total[1m]))

Bandwidth in Mbps

sum(rate(cdn_bytes_served_total[1m])) * 8 / 1e6

Top 10 most-requested paths (requires path label — use sparingly)

topk(10, sum(rate(cdn_requests_total[5m])) by (path))

Warning: Only add path label if your URL space is bounded. Unbounded paths cause cardinality explosion.

Error Rate Queries

Overall error rate (5xx)

rate(cdn_requests_total{status=~"5.."}[5m])
/
rate(cdn_requests_total[5m])

Error rate by status code

sum(rate(cdn_requests_total{status=~"5.."}[5m])) by (status)

Origin error rate (errors returned by origin)

rate(cdn_origin_requests_total{status=~"5.."}[5m])
/
rate(cdn_origin_requests_total[5m])

Error rate that would violate 99.9% SLO

# If this is > 0, you're burning error budget
rate(cdn_requests_total{status=~"5.."}[5m])
/
rate(cdn_requests_total[5m])
> 0.001

SLO & Error Budget Queries

Error budget consumption rate (ratio to 30-day budget)

# Burn rate > 1 means you'll exhaust the monthly budget before the month ends
(
  rate(cdn_requests_total{status=~"5.."}[1h])
  /
  rate(cdn_requests_total[1h])
)
/ (1 - 0.999)

Remaining error budget (fraction, 30-day window)

1 - (
  sum(increase(cdn_requests_total{status=~"5.."}[30d]))
  /
  sum(increase(cdn_requests_total[30d]))
  /
  0.001  # error budget = 1 - SLO = 1 - 0.999
)

Multi-window burn rate (Google SRE approach)

# Fast burn: 1-hour window, threshold ~14.4× for 2-hour exhaustion alert
(
  rate(cdn_requests_total{status=~"5.."}[1h])
  /
  rate(cdn_requests_total[1h])
)
/
(1 - 0.999)
> 14.4

Cache Efficiency Queries

Cache entry count

cdn_cache_entries

Cache size in MB

cdn_cache_size_bytes / 1e6

Cache miss rate (requests going to origin)

rate(cdn_origin_requests_total[5m])
/
rate(cdn_requests_total[5m])

Average compression ratio

histogram_quantile(0.50,
  sum(rate(cdn_compression_ratio_bucket[5m])) by (le)
)

Alerting Rules

These are example Prometheus alerting rules (for prometheus.yml):

groups:
  - name: cdn_alerts
    rules:

      # Hit ratio dropped below 80% — caching problem
      - alert: CDNHitRatioLow
        expr: |
          rate(cdn_requests_total{cache="hit"}[5m])
          /
          rate(cdn_requests_total[5m])
          < 0.80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CDN hit ratio below 80% for 10 minutes"
          description: "Current hit ratio: {{ $value | humanizePercentage }}"

      # p99 latency above 1 second
      - alert: CDNHighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(cdn_request_duration_seconds_bucket[5m])) by (le)
          ) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CDN p99 latency above 1s"

      # Error rate above 1% (burning 99.9% SLO)
      - alert: CDNHighErrorRate
        expr: |
          rate(cdn_requests_total{status=~"5.."}[5m])
          /
          rate(cdn_requests_total[5m])
          > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "CDN error rate above 1%"
          description: "Current error rate: {{ $value | humanizePercentage }}"

      # Fast burn: exhausting monthly error budget in under 2 hours
      - alert: CDNFastBurn
        expr: |
          (
            rate(cdn_requests_total{status=~"5.."}[1h])
            /
            rate(cdn_requests_total[1h])
          )
          / (1 - 0.999) > 14.4
        for: 2m
        labels:
          severity: critical
          page: "true"
        annotations:
          summary: "CDN burning error budget 14.4× faster than sustainable rate"

Grafana Dashboard Layout

Recommended panel organization:

Row 1: Traffic Overview

• Total RPS (stat)
• Bandwidth Mbps (stat)
• Error rate % (stat)

Row 2: Cache Performance

• Hit ratio over time (graph)
• Byte hit ratio over time (graph)
• Cache size (graph)

Row 3: Latency

• p50/p95/p99 latency (graph)
• Latency heatmap (heatmap panel)
• Origin latency p99 (graph)

Row 4: Errors

• Error rate over time (graph)
• Error rate by status code (graph)
• SLO burn rate (graph with threshold annotation)

Row 5: Infrastructure

• Cache entry count (graph)
• Goroutine count (graph)
• Memory usage (graph)