Under the Hood #6: Linux Memory Pressure Explained

It’s 2:00 AM on a Tuesday. Your monitoring system fires off a high-severity alert. You jump into Grafana, and the system dashboard greets you with a bright red panel:

Your immediate instinct? Someone pushed a memory leak to production.

You quickly SSH into the server and run the classic diagnostic command:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi        29Gi       1.0Gi       120Mi        14Gi        14Gi
Swap:         2.0Gi        0B         2.0Gi

It seems to confirm your worst fears: almost every gigabyte of RAM is consumed. In the incident channel, suggestions start flying:

• "Restart the application service immediately!"
• "We need to double the VM size in AWS right now!"

Meanwhile, deep inside the system, the Linux kernel is quietly thinking:

"Everything is working exactly as intended."

The reason for this disconnect is simple: Linux treats unused RAM as wasted RAM.

Once you truly understand that single architectural philosophy, you will understand why perfectly healthy production servers routinely sit at 90 - 95% memory usage without a single dropped request.

The Common Misconception

Most engineers mentalize RAM as a simple fuel tank:

Mental Model (Incorrect):
RAM Capacity
[ ■ ■ ■ ■ ■ ■ ■ ■ ■ □ ]  90% Used
  |_____________|   |
     Used = BAD   Free = GOOD

Under this mental model, "Used" memory is consumed memory, and "Free" memory is safety headroom.

Linux does not look at memory this way. The Linux kernel views RAM as a dynamic workspace:

Linux Kernel Model (Correct):
RAM Capacity
[ Applications | Page Cache | Buffers | Slab Cache ] [ Unused ]
  |________________________________________________|   |
              Active Workload = GOOD               Wasted Opportunity

If memory is not actively required by user-space applications, Linux intentionally claims it to accelerate system operations. Leaving gigabytes of RAM completely empty provides zero performance benefit; it is a wasted hardware investment. High memory usage often simply means Linux is doing its job effectively.

Used Memory Doesn't Mean Consumed Memory

When you run free -h, the output can be misleading if you don't know what the kernel is measuring:

              total        used        free      shared  buff/cache   available
Mem:           31Gi        16Gi       1.0Gi       120Mi        14Gi        14Gi

Look at the numbers above:

• Used: 16 GiB (or 31 GiB total system allocation)
• Free: 1.0 GiB
• Available: 14 GiB

Inexperienced engineers focus almost exclusively on Used and Free. They see 1.0 GiB Free and panic.

However, Linux wants you to look at Available:

• Free Memory: RAM that is literally holding nothing. No data, no zeroes, no code, completely unallocated bits.
• Available Memory: An estimate of how much memory is available for starting new applications without swapping. It includes free memory plus reclaimable caches.

The Available metric is your true operational health indicator.

Why Linux Loves Cache (Page Cache)

To understand why memory gets filled, you have to understand the Page Cache.

The Library Analogy

Imagine working at a massive reference library. Readers frequently walk up to your desk requesting specific books stored deep in the underground archive.

• Approach A: Every time a reader asks for Book A, you walk down to the basement, fetch Book A, hand it over, and when they return it, you walk all the way back down to put it away. When the next person asks for Book A 5 minutes later, you repeat the entire journey.
• Approach B: You fetch Book A from the basement. When the reader returns it, you leave it right on your desk. If anyone asks for Book A again today, you hand it over instantly. You only return books to the basement if your desk gets too cluttered to hold new ones.

Linux uses Approach B. Reading data from NVMe drives or SSDs is fast, but reading directly from RAM is orders of magnitude faster.

Without Page Cache (Slow):
[ Disk ] ═════════════════════════════════════► [ Application ]

With Page Cache (Fast):
[ Disk ] ──(First Read)──► [ Page Cache ] ──► [ Application ]
                                │
                        (Subsequent Reads)
                                ▼
                         [ Application ]

When an application reads a file from disk, Linux keeps a copy in spare RAM (the Page Cache). The next time any process reads that file, the read completes almost instantaneously from memory.

Buffers and Slab Memory

Page Cache isn't the only system subsystem borrowing your free RAM. Linux maintains two other essential memory caches:

1. Buffers

While the Page Cache stores actual file content, Buffers store raw disk block metadata. This includes filesystem overhead, directory location structures, and block mapping details. It ensures filesystem operations remain snappy.

2. Slab Memory

The Linux kernel frequently creates and destroys internal structures (like file descriptors, socket connections, and process tracking states).

Instead of allocating and freeing memory from scratch every time, Linux uses Slab Allocation. It retains pre-allocated memory pools for commonly used kernel objects:

• inode_cache: Holds metadata about individual files.
• dentry_cache: Holds directory name-to-inode mappings (making path resolution fast).

What Happens When Applications Need RAM?

A common fear among platform teams is: "If the kernel fills RAM with cache, what happens when my application spikes and needs memory?"

The short answer: Applications always win.

Application Request
        │
        ▼
Is Free RAM Available?
 ├── YES ──► Allocate immediately
 └── NO  ──► Linux immediately drops Page Cache / Reclaims Slab
                 │
                 ▼
          Hand freed RAM to Application

Page Cache is completely transient. The moment a process requests memory that exceeds the Free memory pool, Linux instantly discards unmanaged or clean page cache and hands the freed physical memory to the application.

There is zero overhead or penalty in dropping a clean cache page simply means removing an address reference.

Memory Pressure: What You Should Actually Monitor

If high memory percentage isn't inherently bad, what is the real metric?

The concept you need to monitor is Memory Pressure.

Memory Pressure occurs when the kernel is actively struggling to find reclaimable RAM to satisfy application allocations.

High memory usage merely means your system is heavily utilizing its resources. Memory pressure means the system is choking because it cannot free up memory fast enough to keep pace with demand.

Understanding Swap Correctly

Swap space is often misunderstood. Swap is not "fake RAM," nor is it inherently a sign of a failing server.

        ┌────────────────────────────────────────┐
        │               Physical RAM             │
        │  ┌──────────────────┬───────────────┐  │
        │  │ Active Pages     │ Inactive Pages│  │
        │  │ (Hot Application)│ (Idle/Sleep)  │  │
        │  └────────┬─────────┴───────┬───────┘  │
        └───────────┼─────────────────┼──────────┘
                    │                 │
                    ▼                 ▼
             Kept in RAM        Offloaded to Swap
                                (Freeing RAM for Cache)

Swap is an overflow mechanism on disk used to hold inactive pages.

If a background daemon allocates 200 MB of RAM on startup and then sits completely idle for three weeks, keeping those pages in fast physical RAM is wasteful. Linux will move those cold pages to Swap, freeing up high-speed RAM for hot application execution or active Page Cache.

• Good Swap Usage: Inactive pages moving to swap gradually over time.
• Bad Swap Usage (Thrashing): Active pages continuously bouncing between RAM and Swap because physical memory is exhausted.

Pressure Stall Information (PSI)

Modern Linux kernels (4.20+) introduced a breakthrough metric that solves the "Memory Usage vs. Memory Health" debate once and for all: Pressure Stall Information (PSI).

Instead of asking "What percentage of RAM is filled?", PSI asks:

"How much CPU time are applications losing because they are waiting on memory?"

You can inspect this directly on modern Linux systems:

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

• some: Percentage of time where at least some tasks were stalled waiting for memory reclaim/swap.
• full: Percentage of time where all non-idle tasks were completely blocked waiting for memory.

If your system sits at 98% Memory Usage but your PSI avg10 is 0.00, your system is flying high with zero memory issues. If PSI starts climbing above zero, you are experiencing genuine memory pressure.

The Out-Of-Memory (OOM) Killer

What happens when Linux runs out of options?

If the kernel has:

1. Reclaimed all dropable Page Cache,
2. Shrunk Slab caches to their minimum bounds,
3. Filled up available Swap space,

...and an application asks for one more byte of memory, Linux hits a hard wall. It cannot fulfill the request, so it triggers the OOM Killer (Out-Of-Memory Killer).

         Memory Allocation Request
                    │
                    ▼
      Can Linux reclaim Cache/Slab?
           ├── YES ──► Allocate Memory
           └── NO
                 │
                 ▼
         Can Linux use Swap?
           ├── YES ──► Write to Swap & Allocate
           └── NO
                 │
                 ▼
         [ TRIGGER OOM KILLER ]

The OOM Killer does not select processes randomly. It scans all running processes and calculates an oom_score based on memory consumption percentage and process flags (oom_score_adj). It selects the process with the highest score and sends signal 9 (SIGKILL) to protect the overall OS integrity.

In containerized environments like Kubernetes, this manifests as the infamous OOMKilled state (Exit Code 137), where a pod exceeds its cgroup memory limit and is terminated by the node kernel.

Real Production Investigation

Let's walk through two production scenarios to see the practical difference.

Scenario A: High Memory Usage (Healthy)

Grafana fires an alert: Memory Usage: 96%

You inspect the system:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           64Gi        61Gi       1.2Gi       200Mi        42Gi        43Gi

• Available Memory: 43 GiB
• PSI (/proc/pressure/memory): 0.00
• Swap Activity: Minimal / Static

Verdict: No action required. The system is operating at peak efficiency. 42 GB of disk reads are cached in RAM, making I/O ultra-fast.

Scenario B: High Memory Pressure (Unhealthy)

Grafana fires an alert: Memory Usage: 92%

You inspect the system:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           64Gi        58Gi       1.1Gi       200Mi       4.9Gi       300Mi

• Available Memory: 300 MiB
• PSI (/proc/pressure/memory): avg10=24.50 (Processes spending 24% of time blocked on memory)
• Swap Activity: Constantly increasing read/write I/O ops
• Logs: Kernel throwing OOM warning alerts

Verdict: Critical incident. The system has run out of cache to drop, applications are waiting on memory reclaims, and an OOM event is imminent.

5 Common Mistakes Engineers Make

1. Monitoring Used Memory instead of Available Memory. Setting up alert rules based purely on (used / total) * 100 guarantees false-positive alerts every time your node performs normal I/O caching.
2. Thinking Page Cache is "Wasted" RAM: Attempting to clear the cache manually using sysctl -w vm.drop_caches=3 in production strips away your I/O performance advantages and causes sudden disk latency spikes.
3. Disabling Swap entirely without understanding why: Disabling swap entirely deprives the kernel of offloading idle inactive pages, forcing active application code and page caches to compete for space earlier than necessary.
4. Restarting services purely because RAM looks "full" Restarting a service clears the local allocations, but if that RAM was actually serving page cache, the underlying system performance degrades immediately after the restart.
5. Ignoring PSI and tracking memory capacity alone. Capacity metrics tell you how much room is occupied; PSI tells you if your applications are actually suffering.

Key Takeaways

• Linux hates unused RAM: Empty RAM generates no value. Caching keeps systems fast.
• Cache exists to accelerate I/O: Page cache, buffers, and slab caches temporarily borrow unused physical memory.
• Applications always have priority: Discarding clean cache to provide RAM to applications happens dynamically with virtually zero latency.
• High Memory Usage $\neq$ Memory Pressure: Memory usage measures allocation capacity; memory pressure measures resource contention.
• Track the right metrics: Monitor Available Memory, PSI (/proc/pressure/memory), and OOM Kill events instead of raw Used percentages.

Conclusion

The next time you log into a production server, run free -h, and see 95% memory usage; pause before triggering an incident response.

Ask the right question:

Is Linux running out of memory, or is it simply using RAM to make my system faster?

More often than not, it isn't a runaway memory leak. It’s the Linux kernel doing exactly what it was designed to do.