Proper Kubernetes Monitoring

Kubernetes is very good at hiding problems.

A pod can be Running while the application inside it is completely broken. A node can be Ready while its disk is slowly filling up. A deployment can report all replicas as available while users are receiving HTTP 500 responses.

And somewhere in the middle of all of this, a container is probably writing 14 GB of logs because somebody forgot to disable debug mode.

This is why proper Kubernetes monitoring is more than installing Prometheus and Grafana.

A useful monitoring setup should answer four basic questions:

1. Is the cluster healthy?
2. Are the workloads healthy?
3. Can users actually reach the applications?
4. What happened when something went wrong?

This guide describes a practical monitoring architecture based around Prometheus, Grafana, Loki, Alertmanager, kube-state-metrics, and node-exporter.

Monitoring vs. Observability

The terms monitoring and observability are often used interchangeably, but they describe slightly different things.

Monitoring

Monitoring tells you whether something is currently behaving within expected limits.

Examples include:

• CPU usage
• Memory usage
• Disk usage
• Pod restarts
• Node availability
• HTTP error rate
• Request latency
• Database connections

Monitoring primarily answers:

Is something wrong?

Observability

Observability gives you enough information to understand why something is wrong.

It generally combines three major types of telemetry:

• Metrics
• Logs
• Traces

For example:

1
2
3
4
5
6
7
8
9
10
HTTP 500 rate increased
        |
        v
Application logs show database timeout
        |
        v
Database metrics show connection exhaustion
        |
        v
Node metrics show memory pressure

A good monitoring system should let you move through that chain without having to SSH into six servers and grep through log files.

The Monitoring Architecture

A practical Kubernetes monitoring architecture can look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
                         +----------------------+
                         |       Grafana        |
                         | Dashboards & Alerts  |
                         +----------+-----------+
                                    |
                     +--------------+--------------+
                     |                             |
              +------+-------+               +-----+------+
              |  Prometheus  |               |    Loki    |
              |   Metrics    |               |    Logs    |
              +------+-------+               +-----+------+
                     |                             |
       +-------------+-------------+               |
       |             |             |               |
       v             v             v               v
 node-exporter  kube-state-    Application     Log Collector
                 metrics         Metrics
       |             |
       +-------------+
              |
              v
       Kubernetes Cluster

Alerting should be handled separately:

1
2
3
4
5
6
7
8
9
10
Prometheus
    |
    v
Alertmanager
    |
    +-- Email
    +-- Matrix
    +-- Slack
    +-- PagerDuty
    +-- Other notification systems

The important principle is that Grafana should not be responsible for collecting your data.

Each component should have a clear responsibility:

Metrics with Prometheus

Prometheus should be the central metrics system for Kubernetes.

It collects time-series data from Kubernetes components, nodes, exporters, and applications.

Typical metrics include:

1
2
3
4
5
node_cpu_seconds_total
node_memory_MemAvailable_bytes
node_filesystem_avail_bytes
kube_pod_container_status_restarts_total
kube_deployment_status_replicas_available

Prometheus uses PromQL to query this data.

For example, CPU utilization can be calculated with:

100 * (
  1 -
  avg by (instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  )
)

Memory utilization:

100 * (
  1 -
  node_memory_MemAvailable_bytes
  /
  node_memory_MemTotal_bytes
)

Displaying raw CPU or memory values is not particularly useful.

A node having 8 GB of RAM does not tell you much.

A node using 93% of its available memory does.

Node Monitoring with node-exporter

Every Kubernetes node should expose operating system metrics.

The standard solution is node-exporter, usually deployed as a DaemonSet.

It provides metrics for:

• CPU
• Memory
• Swap
• Filesystems
• Disk I/O
• Network traffic
• Load average
• Kernel statistics
• System uptime

For example:

100 * (
  1 -
  node_filesystem_avail_bytes{
    fstype!~"tmpfs|overlay|squashfs"
  }
  /
  node_filesystem_size_bytes{
    fstype!~"tmpfs|overlay|squashfs"
  }
)

Filesystem monitoring requires some care.

You generally do not want alerts for every pseudo-filesystem Kubernetes exposes.

Common filesystems to filter include:

1
2
3
4
5
tmpfs
overlay
squashfs
proc
sysfs

Otherwise your monitoring system will eventually complain about a filesystem that was never intended to store anything.

Kubernetes State with kube-state-metrics

node-exporter tells you about the operating system.

It does not tell you enough about Kubernetes itself.

For that, you need kube-state-metrics.

It converts Kubernetes API objects into Prometheus metrics.

It provides information about:

• Pods
• Deployments
• StatefulSets
• DaemonSets
• Jobs
• CronJobs
• Nodes
• PersistentVolumes
• PersistentVolumeClaims
• Services
• ConfigMaps
• Secrets

For example, unavailable deployment replicas can be detected with:

kube_deployment_status_replicas_unavailable > 0

Pod restarts can be detected with:

increase(
  kube_pod_container_status_restarts_total[15m]
) > 0

The distinction is important:

1
2
3
4
5
6
7
node-exporter
    |
    +-- What is the machine doing?

kube-state-metrics
    |
    +-- What does Kubernetes think is happening?

You want both.

Container Metrics

Kubernetes workloads also need resource monitoring.

At minimum, monitor:

• CPU usage
• Memory usage
• CPU limits
• Memory limits
• CPU requests
• Memory requests
• OOM kills
• Container restarts

CPU usage:

sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total[5m])
)

Memory usage:

sum by (namespace, pod) (
  container_memory_working_set_bytes
)

OOM kills are particularly important.

A container that repeatedly gets killed because it exceeds its memory limit is not healthy simply because Kubernetes keeps restarting it.

Resource Requests and Limits

Monitoring resource usage without monitoring resource configuration is incomplete.

Suppose a container uses:

1
2
Memory usage: 700 MiB
Memory limit: 1 GiB

That is relatively informative.

But:

1
2
Memory usage: 700 MiB
Memory limit: 8 GiB

means something very different.

You should therefore monitor resource utilization relative to configured limits.

For example:

100 *
sum by (namespace, pod) (
  container_memory_working_set_bytes
)
/
sum by (namespace, pod) (
  kube_pod_container_resource_limits{
    resource="memory"
  }
)

Be careful with workloads that do not have memory limits configured.

Missing limits can result in missing or misleading series, which is an excellent way to make a dashboard look impressive while providing very little useful information.

Disk Monitoring

Disk usage is one of the easiest things to overlook.

Disks also have a remarkable tendency to become completely full at inconvenient times.

Monitor at least:

• Filesystem usage
• Filesystem available bytes
• Disk I/O
• Read/write throughput
• Disk latency
• Inode usage

A filesystem can have free space while being completely out of inodes.

Monitor both.

Inode utilization can be calculated with:

100 *
(
  1 -
  node_filesystem_files_free
  /
  node_filesystem_files
)

For Kubernetes specifically, pay particular attention to directories such as:

1
2
3
/var/lib/kubelet
/var/lib/containerd
/var/lib/longhorn

and any directories used for local application storage.

Storage problems can quickly become cluster-wide problems.

Kubernetes Storage Monitoring

Persistent storage deserves its own monitoring.

If you use Longhorn, monitor:

• Volume health
• Replica health
• Replica rebuilds
• Volume capacity
• Disk capacity
• Node availability
• Replica scheduling
• Read/write latency
• I/O errors

A storage system can be technically “up” while slowly running out of capacity.

That is not healthy.

A simple starting point might be:

1
2
3
80%  -> Warning
90%  -> Critical
95%  -> Emergency

The exact thresholds should depend on your environment and workload.

Network Monitoring

Network problems are frequently mistaken for application problems.

Monitor:

• Network throughput
• Packet errors
• Dropped packets
• TCP connections
• DNS failures
• Connection latency

For example, receive traffic:

rate(node_network_receive_bytes_total[5m])

Transmit traffic:

rate(node_network_transmit_bytes_total[5m])

Receive errors:

rate(node_network_receive_errs_total[5m])

Network monitoring becomes particularly important when running:

• Ingress controllers
• Load balancers
• Service meshes
• External databases
• Object storage
• Multi-node storage
• Container registries

Application Metrics

Infrastructure metrics are not enough.

Applications should expose their own metrics.

Useful application metrics include:

1
2
3
4
5
6
7
8
HTTP requests
HTTP errors
Request latency
Active connections
Database queries
Queue depth
Cache hit rate
Background jobs

A typical HTTP application might expose metrics such as:

1
2
http_requests_total
http_request_duration_seconds

The HTTP error rate can then be calculated using:

sum(
  rate(http_requests_total{
    status=~"5.."
  }[5m])
)
/
sum(
  rate(http_requests_total[5m])
)

This provides something infrastructure monitoring cannot:

Are users actually experiencing failures?

Black-Box Monitoring

Internal metrics can tell you that a service believes it is healthy.

That does not necessarily mean a user can reach it.

This is where black-box monitoring becomes useful.

Blackbox Exporter can test:

• HTTP
• HTTPS
• TCP
• ICMP
• DNS

A typical check looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
                    Internet
                        |
                        v
                Blackbox Exporter
                        |
             +----------+----------+
             |          |          |
             v          v          v
            DNS        TCP        HTTP
             |          |          |
             +----------+----------+
                        |
                        v
                     Result

This can detect problems such as:

1
2
3
4
Application pod: healthy
Service: healthy
Ingress: healthy
DNS: broken

From Kubernetes’ perspective, everything is fine.

From the user’s perspective, the website does not exist.

Logs with Loki

Metrics tell you that something happened.

Logs often tell you what happened.

Loki is a natural fit for Kubernetes because it stores log streams using labels rather than indexing every word in every log line.

A typical architecture is:

1
2
3
4
5
6
7
8
9
10
Kubernetes Pods
      |
      v
Log Collector
      |
      v
    Loki
      |
      v
   Grafana

The collector can be Grafana Alloy or another Kubernetes-compatible log agent.

Useful labels include:

1
2
3
4
5
6
cluster
namespace
pod
container
node
application

Avoid putting high-cardinality values into labels.

For example, this is a bad idea:

1
user_id

If your application has millions of users, that can produce an enormous number of unique label combinations.

Keep high-cardinality information inside the log body instead.

Structured Logging

Applications should preferably emit structured logs.

Instead of:

1
User login failed for user 123 because password was invalid

use structured JSON:

1
2
3
4
5
6
{
    "level": "warn",
    "event": "login_failed",
    "user_id": "123",
    "reason": "invalid_password"
}

Structured logging makes searching, parsing, and correlation considerably easier.

It also makes it easier to connect logs with metrics and traces.

Alerting with Alertmanager

A monitoring system that only displays dashboards is not really monitoring.

It is a very expensive screensaver.

Prometheus evaluates alert rules and sends firing alerts to Alertmanager.

Alertmanager handles:

• Grouping
• Deduplication
• Routing
• Silencing
• Inhibition
• Notification delivery

A useful severity model might look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
INFO
 |
 +-- Something changed

WARNING
 |
 +-- Something may require attention

CRITICAL
 |
 +-- Something is actively failing

EMERGENCY
 |
 +-- Immediate intervention required

Not every alert should wake somebody up.

If your monitoring system sends 47 notifications overnight, humans eventually learn to ignore all of them.

That is alert fatigue, and it defeats the entire purpose of alerting.

What Should Trigger Alerts?

A good rule is:

Alert on symptoms that require action, not every unusual metric.

Good alerts include:

1
2
3
4
5
6
7
8
9
10
11
Node is unreachable
Disk almost full
Filesystem became read-only
Pod repeatedly crashing
Deployment has unavailable replicas
Persistent volume unhealthy
High HTTP 5xx rate
High request latency
Certificate approaching expiration
Database unavailable
Ingress endpoint unreachable

Bad alerts include:

1
2
3
4
5
CPU > 70%
Memory > 50%
Pod restarted once
Network traffic increased
Disk usage changed

A high CPU percentage is not inherently a problem.

A high CPU percentage combined with increased request latency might be.

Context matters.

Alert on Trends

Thresholds are useful, but trends can be even more useful.

Suppose a filesystem looks like this:

1
2
3
4
5
Monday:    65%
Tuesday:   68%
Wednesday: 71%
Thursday:  74%
Friday:    77%

It is not currently critical.

But it is clearly heading somewhere unpleasant.

Prometheus provides functions such as predict_linear() that can be used to detect these trends.

For example:

predict_linear(
  node_filesystem_avail_bytes[6h],
  24 * 3600
) < 0

This can detect filesystems that are predicted to run out of space within the next day.

The exact query should be tuned to your environment, but the principle is valuable:

Do not wait for the fire alarm if you can see the smoke.

Kubernetes Dashboard Design

Do not create one enormous dashboard containing 400 graphs.

Nobody will use it.

Instead, create dashboards around specific responsibilities.

Cluster Overview

Show:

1
2
3
4
5
6
7
8
9
10
Cluster health
Node count
Ready nodes
CPU usage
Memory usage
Disk usage
Pod count
Failed pods
Restarting pods
Persistent volume health

Node Dashboard

Show:

1
2
3
4
5
6
7
8
9
CPU
Memory
Swap
Load
Disk
Filesystem
Network
Temperature
Uptime

Kubernetes Workloads

Show:

1
2
3
4
5
6
7
8
Namespaces
Deployments
Pods
Restarts
OOM kills
CPU
Memory
Replica availability

Storage

Show:

1
2
3
4
5
6
Volumes
Capacity
Replica health
IOPS
Latency
Errors

Applications

Show:

1
2
3
4
5
Requests
Errors
Latency
Throughput
Dependencies

Logs

Show:

1
2
3
4
5
6
Namespace
Pod
Container
Application
Search
Time range

The goal is to progressively narrow down the problem.

The Four Golden Signals

For applications, the four golden signals are extremely useful:

Latency

How long are requests taking?

Monitor:

1
2
3
p50
p95
p99

Traffic

How much traffic is the system receiving?

Examples:

1
2
3
requests/sec
connections
messages/sec

Errors

How many requests are failing?

Examples:

1
2
3
4
HTTP 4xx
HTTP 5xx
application errors
timeouts

Saturation

How close is the system to its limits?

Examples:

1
2
3
4
5
6
CPU
Memory
Disk
Connections
Queues
Workers

These four signals often tell you much more than a wall of infrastructure graphs.

Kubernetes Events

Do not ignore Kubernetes events.

They are incredibly useful when diagnosing scheduling and deployment problems.

Examples include:

1
2
3
4
5
6
7
8
FailedScheduling
FailedMount
FailedAttachVolume
BackOff
Unhealthy
FailedPull
ImagePullBackOff
Evicted

You can inspect recent events with:

1
kubectl get events -A --sort-by=.lastTimestamp

A good monitoring setup should make these events accessible alongside metrics and logs.

Monitor the Monitoring System

This is the part people conveniently forget.

You need to monitor:

• Prometheus
• Grafana
• Loki
• Alertmanager
• Exporters
• Log collectors

Otherwise you can end up with:

1
2
3
Application: DOWN
Prometheus:  DOWN
Alertmanager: DOWN

and confidently receive no alert.

Prometheus should therefore monitor itself.

Useful metrics include:

1
2
3
4
5
prometheus_tsdb_storage_blocks_bytes
prometheus_tsdb_head_series
prometheus_rule_evaluation_failures_total
prometheus_target_scrapes_exceeded_sample_limit_total
prometheus_tsdb_wal_corruptions_total

Also monitor whether targets are actually being scraped:

up == 0

If an exporter disappears, you should know about it.

High Availability

For a small homelab or development cluster, a single Prometheus instance may be perfectly reasonable.

For production infrastructure, monitoring itself becomes infrastructure.

A more resilient architecture might look like:

1
2
3
4
5
6
7
8
Prometheus A
Prometheus B
     |
     v
Alertmanager
     |
     v
Notification systems

Long-term metric storage can also be handled using systems such as:

• Thanos
• Mimir
• VictoriaMetrics

The correct architecture depends on scale and operational requirements.

Do not deploy a fourteen-component observability platform because somebody on the internet called it “production-grade.”

More components mean more things to operate.

Retention and Storage

Metrics consume storage.

Logs consume considerably more storage.

Before deploying monitoring, decide how much history you actually need.

For example:

1
2
Prometheus: 30 days
Loki:       14 days

might be perfectly reasonable for a smaller environment.

Larger environments may require considerably longer retention or dedicated long-term storage.

You should also monitor the storage consumed by the monitoring systems themselves.

Otherwise your observability stack can become the reason your cluster runs out of disk.

That would be a particularly embarrassing failure mode.

Recording Rules

PromQL queries can become expensive.

Instead of repeatedly calculating a complex query, you can create a recording rule.

For example:

1
2
3
4
5
6
7
8
9
10
groups:
    - name: kubernetes
      interval: 30s

      rules:
          - record: namespace:container_cpu_usage:rate5m
            expr: |
                sum by (namespace) (
                  rate(container_cpu_usage_seconds_total[5m])
                )

Dashboards can then query:

namespace:container_cpu_usage:rate5m

This reduces repeated computation and can make dashboards significantly faster.

Service-Level Objectives

For important applications, monitoring should eventually move beyond infrastructure.

Define service-level objectives such as:

1
2
3
Availability: 99.9%
HTTP 5xx rate: < 0.1%
p95 latency: < 500 ms

Then monitor whether the service is actually meeting those objectives.

This changes the question from:

Is the CPU okay?

to:

Is the service delivering what users expect?

That is a much more useful question.

A Practical Kubernetes Monitoring Stack

A sensible starting stack for a Kubernetes cluster is:

This provides a strong foundation without turning the cluster into a distributed systems thesis.

The Most Important Dashboards

If you only build a few dashboards, build these.

Cluster

1
2
3
4
5
6
7
8
Cluster health
Node health
CPU
Memory
Disk
Pods
Restarts
Storage

Nodes

1
2
3
4
5
6
7
CPU
RAM
Swap
Load
Disk I/O
Filesystem
Network

Kubernetes

1
2
3
4
5
6
7
8
Namespaces
Deployments
StatefulSets
DaemonSets
Pods
Restarts
OOM kills
Scheduling failures

Applications

1
2
3
4
5
Traffic
Errors
Latency
Saturation
Dependencies

Storage

1
2
3
4
5
6
Capacity
Volume health
Replica health
IOPS
Latency
Errors

Logs

1
2
3
4
5
6
Namespace
Pod
Container
Application
Search
Severity

A Useful Troubleshooting Workflow

When something breaks, do not randomly click dashboards.

Follow a consistent path.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
User reports problem
        |
        v
Is the endpoint reachable?
        |
        +-- NO --> DNS / network / ingress
        |
        v
Is the application returning errors?
        |
        +-- YES --> Application logs
        |
        v
Is the pod healthy?
        |
        +-- NO --> Kubernetes events
        |
        v
Is the node healthy?
        |
        +-- NO --> Node metrics
        |
        v
Is storage healthy?
        |
        +-- NO --> Storage metrics
        |
        v
Is a dependency failing?
        |
        +-- YES --> Database / cache / API
        |
        v
Investigate application behavior

This workflow turns monitoring from a collection of pretty graphs into an actual diagnostic system.

What Good Monitoring Looks Like

A good monitoring system has several characteristics.

It detects failures quickly

You should not learn about an outage from users 30 minutes after it started.

It provides context

An alert should tell you:

1
2
3
4
5
What happened?
Where?
When?
How severe?
What is affected?

It has low noise

If everything is an alert, nothing is an alert.

It preserves history

You should be able to answer:

What changed before this broke?

It is itself reliable

Your monitoring system should not disappear when the cluster has a bad day.

It helps humans troubleshoot

A dashboard should lead you toward the cause rather than simply displaying impressive numbers.

Final Architecture

A mature Kubernetes monitoring setup can ultimately look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
                         +------------------+
                         |     Grafana      |
                         | Dashboards/Logs  |
                         +--------+---------+
                                  |
                 +----------------+----------------+
                 |                                 |
        +--------v--------+                +-------v-------+
        |    Prometheus   |                |     Loki      |
        |     Metrics     |                |     Logs      |
        +--------+--------+                +-------^-------+
                 |                                 |
       +---------+-----------+                     |
       |         |           |                     |
       v         v           v                     |
 node-exporter kube-state- Blackbox                |
               metrics     Exporter                |
       |         |           |                     |
       +---------+-----------+                     |
                 |                                 |
                 v                                 |
          Kubernetes Cluster                       |
                 |                                 |
                 +---------------+-----------------+
                                 |
                                 v
                          Log Collector
                                 |
                                 v
                                Loki


                       Prometheus
                           |
                           v
                      Alertmanager
                           |
              +------------+------------+
              |            |            |
              v            v            v
            Email        Matrix       Pager

The important thing is not the number of components.

The important thing is that every layer answers a different question:

1
2
3
4
5
6
Metrics  -> What is happening?
Logs     -> What happened?
Events   -> What did Kubernetes do?
Traces   -> Where did the request spend time?
Alerts   -> Does someone need to act?
Grafana  -> How do humans understand all of this?

That is proper Kubernetes monitoring.

Not 400 graphs.

Not a giant Grafana dashboard nobody opens.

Not an alert every time CPU reaches 71%.

A good monitoring system is quiet when everything is healthy, loud when something actually matters, and useful when you need to figure out why the cluster has decided to ruin your afternoon.

• Previous
  The People Who Knew