Most Linux metrics are easy to understand.

CPU usage tells you how busy your processors are.

Memory usage tells you how much RAM is being used.

Disk usage tells you how full your storage is.

Then there's Load Average.

It's one of the most misunderstood metrics in Linux. Ask ten engineers what a Load Average of 10 means, and you'll probably hear ten different answers. Many assume it's simply another way of measuring CPU utilization.

It isn't.

Imagine opening your monitoring dashboard during a production incident and seeing this:

• CPU Usage: 18%
• Load Average: 12

At first glance, those numbers seem impossible. If the CPUs are mostly idle, why does the server appear heavily loaded?

The first instinct is usually to blame the processor. Teams begin looking for runaway processes, scaling nodes, or increasing CPU limits.

But in many production incidents, the CPU isn't the bottleneck at all.

The real culprit could be slow storage, blocked disk I/O, an unresponsive NFS mount, or processes waiting on the Linux kernel.

Understanding why this happens is one of those skills that separates experienced Linux and DevOps engineers from people who simply watch dashboards.

Welcome to Under the Hood, a DevOpsInside series that explores what really happens beneath the technologies we use every day. Every article starts with a common misconception, explains what's happening inside Linux, shows how it affects Docker and Kubernetes, and finishes with how production engineers debug it.

In this first article, we'll uncover what Load Average actually measures, why it doesn't represent CPU usage, and how production engineers investigate high-load systems without chasing the wrong problem.

The Common Misconception

Ask someone what Linux Load Average measures, and the answer is usually something like this:

"It's how busy the CPU is."

That's one of the oldest myths in Linux.

If it were true, this situation wouldn't be possible.

Yet systems like this appear in production every day.

The confusion comes from assuming that Load Average measures CPU utilization.

It doesn't.

CPU usage answers one question:

How much time are the CPUs actively executing instructions?

Load Average answers a completely different question:

How many processes are waiting for CPU time or waiting for the Linux kernel to finish something before they can continue?

That's an important difference.

A process doesn't have to be actively using the CPU to contribute to Load Average.

It only needs to be unable to continue.

Once you understand that distinction, many "mysterious" Linux performance problems suddenly start making sense.

What Linux Actually Measures

To understand Load Average, imagine a supermarket.

The cashier represents the CPU.

Customers represent Linux processes.

If one customer is paying while everyone else waits in line, the cashier isn't overloaded yet.

But if twenty customers are waiting, the queue becomes long even though only one person is actively checking out.

Linux works in a similar way.

Instead of measuring only the work currently being performed, it also measures the amount of work waiting to be performed.

Under the hood, Linux counts three types of processes.

Running Processes

These are processes currently executing on a CPU core.

Application
      │
      ▼
Running on CPU

They're actively consuming processor time.

Runnable Processes

These processes are ready to run but are waiting for an available CPU.

Ready to Run
      │
      ▼
CPU Queue
      │
      ▼
CPU

If every CPU core is busy, these processes must wait.

They increase the Load Average even before they begin executing.

Processes in Uninterruptible Sleep (D State)

This is where most people become surprised.

Some processes aren't waiting for CPU time at all.

They're waiting for the kernel to complete an operation, usually involving hardware.

Examples include:

• Reading data from a slow disk
• Waiting for an SSD
• Accessing an NFS share
• Waiting for SAN storage
• Blocked filesystem operations

These processes appear in the D (Uninterruptible Sleep) state.

Although they're barely using the CPU, Linux still counts them in the Load Average because they can't continue until the kernel finishes the requested operation.

That's why a server can show:

• CPU Usage: 20%
• Load Average: 15

The CPUs aren't busy.

The applications are simply waiting.

The Load Average Formula Isn't Really a Formula

One common misconception is that Load Average is calculated using CPU percentages.

It isn't.

Linux continuously samples the number of running and waiting processes and reports three rolling averages:

• 1 minute
• 5 minutes
• 15 minutes

When you run:

uptime

you'll see something similar to:

load average: 2.31, 1.87, 1.42

These numbers don't represent CPU utilization.

They represent the average number of processes that were either running or waiting during each time period.

This also explains why Load Average changes gradually.

A sudden traffic spike won't instantly produce a Load Average of 20 because Linux smooths the values over time.

Conversely, even after a problem has been resolved, the 15-minute average may remain elevated while the system recovers.

Why Disk I/O Can Make Load Average Explode

One of the biggest reasons engineers misinterpret Load Average is that they assume every process contributing to it is waiting for CPU time.

In reality, many are waiting for disk operations.

Imagine an application trying to read data from storage. It sends a request to the Linux kernel and waits for the result. During that time, the process isn't executing instructions, but it also can't continue until the storage device responds.

The flow looks like this:

Application
      │
      ▼
Read File
      │
      ▼
Linux Kernel
      │
      ▼
Storage Device
      │
      ▼
Waiting...
      │
      ▼
Process Enters D State

Notice what's missing.

The CPU.

While the storage device is busy responding, the processor may be almost completely idle. Yet the waiting process still contributes to the system's Load Average.

This explains a production scenario that confuses many engineers.

Looking only at CPU graphs would suggest the server has plenty of capacity.

Looking at disk metrics tells a completely different story.

The processors are waiting because the storage subsystem has become the bottleneck.

This situation commonly occurs with:

• Slow HDDs under heavy load
• Saturated SSDs
• Network-attached storage (NAS)
• Storage Area Networks (SAN)
• Large database reads
• Backup jobs competing for disk access

The CPU isn't overloaded.

It's simply waiting.

Why NFS Can Make a Healthy Server Look Broken

Network File System (NFS) introduces another layer of complexity.

Unlike a local SSD, an NFS mount depends on another server across the network. Every read or write operation involves network latency, remote storage, and another operating system.

If that remote server slows down, your application has no choice but to wait.

The request path looks like this:

Application
      │
      ▼
Linux Kernel
      │
      ▼
NFS Client
      │
      ▼
Network
      │
      ▼
Remote NFS Server
      │
      ▼
Waiting...

Again, notice that CPU utilization may remain low throughout the entire process.

Meanwhile, dozens or even hundreds of processes accumulate in the D (Uninterruptible Sleep) state, pushing the Load Average higher and higher.

This is why engineers sometimes restart applications, increase CPU resources, or even add more Kubernetes nodes without solving the problem.

The real issue isn't compute power.

It's storage latency.

A Real Production Investigation

Let's walk through a realistic production incident.

A Kubernetes cluster begins responding slowly.

API requests that normally complete in 50 milliseconds now take several seconds.

Grafana shows:

At first glance, nothing appears alarming except the Load Average.

Many engineers would immediately assume the CPUs are overloaded.

But a closer investigation tells a different story.

Running:

uptime

shows:

load average: 13.90, 11.72, 8.41

Next, checking CPU utilization with top reveals that most CPU cores are idle.

So where is the load coming from?

Running:

vmstat 1

shows a large number of processes blocked on I/O.

Finally:

iostat -x 1

reveals the real problem.

The storage device is running at nearly 100% utilization, and disk response times have increased dramatically.

Nothing is wrong with Kubernetes.

Nothing is wrong with the CPU.

Applications are simply waiting for storage.

The high Load Average was a symptom of slow I/O, not excessive processor usage.

This is why experienced production engineers rarely trust a single metric. They investigate the entire system before deciding where the bottleneck actually exists.

High Load Doesn't Always Mean High CPU

One of the biggest lessons to remember is this:

A server can have:

• High Load Average
• Low CPU usage
• Plenty of available memory

...and still perform terribly.

That's because Linux measures work waiting to be completed, not just work currently executing.

Once you start thinking about Load Average as a queue of waiting work rather than CPU utilization, many confusing production incidents become much easier to understand.

In the next section, we'll look at the exact commands experienced Linux engineers use to determine whether they're dealing with CPU contention, disk bottlenecks, or processes stuck waiting inside the kernel.

How Production Engineers Actually Investigate High Load

One of the biggest mistakes engineers make is opening top, seeing a high Load Average, and immediately assuming the CPU is overloaded.

Experienced Linux engineers take a different approach.

Instead of asking,

"Why is the Load Average high?"

They ask:

"What are processes waiting for?"

That's a much more useful question.

Let's look at the commands they use during a real production investigation.

Step 1: Check the Load Average

Start with the simplest command.

uptime

Example output:

15:42:31 up 24 days,  6:51,
load average: 8.42, 7.91, 6.75

This immediately tells you the system has been under sustained load.

What it doesn't tell you is why.

That's why uptime is only the starting point.

Step 2: Is the CPU Actually Busy?

Next, check CPU utilization.

top

Suppose you see:

This changes everything.

The CPUs are mostly idle.

If CPU isn't the bottleneck, the high Load Average must be coming from somewhere else.

Step 3: Check for Blocked Processes

Now investigate whether processes are waiting for I/O.

vmstat 1

Pay attention to the b column.

Example:

procs -----------memory---------- ---swap-- -----io----
 r  b
 2 15

The r column shows runnable processes waiting for CPU.

The b column shows processes blocked while waiting for I/O.

If b keeps increasing, Linux is telling you that applications are waiting for the kernel to finish disk operations rather than waiting for processor time.

Step 4: Check Storage Performance

Now verify whether storage is actually slow.

iostat -x 1

A busy device may look like this:

When disk utilization approaches 100%, applications begin waiting longer for reads and writes.

Those waiting processes increase the Load Average even though the CPU remains mostly idle.

Step 5: Find the Processes Causing the Load

Finally, identify which processes are responsible.

pidstat -d

Unlike top, which focuses mainly on CPU usage, pidstat can show which processes are generating heavy disk activity.

This often leads directly to the root cause.

It could be:

• A database performing large reads
• A backup job saturating storage
• A logging process writing excessively
• An application repeatedly scanning files
• An NFS-mounted filesystem responding slowly

By this stage, you're no longer guessing.

You're following evidence.

Common Mistakes

High Load Average often leads engineers toward the wrong solution.

Here are some of the most common mistakes.

Mistake 1: Treating Load Average as CPU Usage

This is by far the most common misunderstanding.

A high Load Average simply means processes are waiting.

It does not automatically mean the processors are overloaded.

Mistake 2: Scaling CPU Before Finding the Bottleneck

Seeing a high Load Average often triggers automatic scaling or larger VM sizes.

If the real problem is slow storage, adding more CPU cores won't improve performance.

The applications will continue waiting for disk operations to complete.

Mistake 3: Ignoring Processes in D State

Processes stuck in Uninterruptible Sleep (D) are often the real reason Load Average climbs.

Many dashboards don't highlight them, so engineers overlook one of Linux's most valuable troubleshooting clues.

Mistake 4: Looking at Only One Metric

CPU, memory, disk, and network all influence system performance.

Looking at CPU graphs alone rarely tells the complete story.

The best investigations combine multiple metrics before reaching a conclusion.

Key Takeaways

Before blaming the CPU, remember these principles:

• Load Average is not CPU utilization.
• Linux counts both running processes and processes waiting for resources.
• Slow storage and NFS issues can dramatically increase Load Average.
• Processes in D state contribute to Load Average even when CPU usage is low.
• Commands like uptime, top, vmstat, iostat, and pidstat work together to reveal the real bottleneck.
• The goal isn't to reduce Load Average—it's to understand why it's high.

Once you stop thinking of Load Average as a CPU metric and start treating it as a measure of waiting work, Linux performance becomes much easier to understand.

Conclusion

Load Average has confused Linux users for decades because its name suggests it measures processor utilization.

It doesn't.

Instead, it reflects how much work the operating system is trying to handle, including processes actively running and those waiting for CPU time, storage, or other kernel-managed resources.

That's why a server showing 20% CPU utilization can still feel painfully slow. The processors may be idle while applications wait on disk I/O, network storage, or other bottlenecks hidden beneath the surface.

The next time you encounter a high Load Average, resist the urge to immediately add more CPU. Start by asking a different question:

"What are my processes waiting for?"

More often than not, the answer lies somewhere other than the processor.

Next in the Under the Hood series: What Happens During Linux Boot:

From pressing the power button to the moment Docker, containerd, kubelet, and your applications begin running. Understanding the Linux boot process will make it much easier to troubleshoot node startup issues, systemd services, and Kubernetes worker nodes in production.