Linux OOM Killer vs SSH: How to Stop Getting Locked Out of Your Own Server

When a Linux box runs out of memory, “sshd is running” stops meaning “you can log in.” The Linux OOM killer reclaims RAM by killing processes, and your login path is fair game. Even if sshd survives, the machine is thrashing swap so hard that a fresh login (itself a memory allocation) blocks or fails.

That happened to me last week. My DigitalOcean droplet exhausted its memory and I couldn’t SSH in. I recovered it through the web recovery console, the out-of-band terminal you reach for when nothing else works, one laggy keystroke at a time.

This post is the fix: four changes that keep the login path alive through the next memory storm, plus the socket-activation detour that nearly had me hardening the wrong systemd unit. There is also a post-mortem: the kernel spent half an hour killing tiny helper processes while the real culprit was never touched.

How the Linux OOM killer locks you out

A full server refuses your login for two separate reasons.

First, the kernel’s Out-Of-Memory (OOM) killer starts reclaiming RAM by force. It scores every process with a “badness” heuristic and kills the worst offender. sshd, its per-connection children, and systemd-logind (the process that actually grants you a session) are all candidates. Kill logind and new logins hang forever, even though the daemon listening on port 22 is technically still alive.

Second, even when sshd survives, a brand new login is itself an allocation event. The box has to fork() a shell, allocate a pseudo-terminal, run the PAM authentication stack, and load your shell profile. Under memory pressure every one of those steps can fail or block, while the kernel pages to disk so aggressively that the whole machine freezes for minutes at a time.

That second point is the one people miss when they search “can’t SSH into server out of memory” and get told to check their firewall. The connection refusal is memory pressure, not a network fault. My recovery console worked because it is out-of-band: it does not depend on the OS network stack, on free memory, or on a healthy sshd, but talks to the virtual serial console underneath all of that.

Know how to reach yours before you need it, whether that is the DigitalOcean recovery console, AWS serial console, IPMI, or whatever your provider calls it. By the time you actually need it, you cannot look it up on the box.

The fix is four layers, each a one-shot command

Four layers of defense in depth: keep the login path alive, guarantee a way back in, stop the freeze before it starts, and stop the thrash. Each is one copy-paste command on Ubuntu, and none depends on the others working.

Layer 1: make the login path unkillable

Tell the kernel never to pick sshd or systemd-logind as OOM victims. The OOMScoreAdjust value runs from -1000 (never kill me) to +1000 (kill me first); it biases that badness score the kernel computes for every process.

for s in ssh systemd-logind; do
  mkdir -p /etc/systemd/system/$s.service.d
  printf '[Service]nOOMScoreAdjust=-1000n' > /etc/systemd/system/$s.service.d/oom.conf
done
systemctl daemon-reload && systemctl restart ssh systemd-logind

This drop-in on ssh.service is the correct target, but on modern Ubuntu it comes with a socket-activation twist that nearly sent me hardening the wrong unit, plus a vendor drop-in that can quietly outrank your own.

Layer 2: reserve memory so root can always recover

This is the most direct answer to “I just want to SSH in and run kill.” The kernel keeps a small memory reserve that only root may dip into, governed by vm.admin_reserve_kbytes. The default is a stingy 8 MB, nowhere near enough to fork a login shell. Bump it so a root login can always get in:

printf 'vm.admin_reserve_kbytes=262144nvm.user_reserve_kbytes=131072n' 
  > /etc/sysctl.d/99-oom-recovery.conf && sysctl --system

256 MB in reserve is the difference between a recovery-console session and a normal SSH login, and of the four fixes it is the one I most wish I had set before the incident.

Layer 3: kill the hog before the freeze, with earlyoom

The kernel’s OOM killer fires too late. It waits for true exhaustion, by which point the swap-thrash has already locked you out for minutes. A userspace daemon fires earlier, acting on a memory-pressure percentage rather than total exhaustion. earlyoom gives you a dead-simple global safety net with --avoid and --prefer process steering:

apt-get install -y earlyoom
# /etc/default/earlyoom:
EARLYOOM_ARGS="-r 3600 -m 10 -s 100 
  --avoid '(^|/)(sshd|systemd|systemd-logind|init)$' 
  --prefer '(^|/)(python|python3|uvicorn|node|syncthing)'"
systemctl restart earlyoom

The -s 100 is deliberate, and it is the setting people get wrong. earlyoom fires only when its memory test and its swap test both pass. The common -m 10 -s 10 reads as “act when free RAM is under 10 percent and free swap is also under 10 percent.” On a box with a big, mostly idle swap file, that swap condition is almost never true. Mine has 8 GB of swap sitting nearly empty, so -s 10 watched RAM crater and did nothing, waiting on a swap drain that never came. It shipped in my first pass and cost me a later incident. Setting -s 100 makes the swap percentage always at or below 100, so the swap test always passes and earlyoom triggers on the RAM floor alone, which is the entire point of an early killer. --avoid doubles up Layer 1: even the early killer will not touch your way in. --prefer points it at the usual suspects first. Trim that regex to whatever runs on your box; mine reflects a fleet of Python and uvicorn workers.

Ubuntu now ships systemd-oomd by default, and it is the more correct long-term tool: it reacts to PSI pressure stalls instead of a raw free-memory percentage, and it is cgroup-aware. I reach for earlyoom here anyway because it is a single global net that does not care how your workload is organized, and stock oomd typically watches managed system slices rather than the ad-hoc user and agent processes that blew up my box. If you are willing to put your workloads into proper cgroups, systemd-oomd plus those cgroups is the better destination. This post hardens the login path first.

Layer 4: stop the swap thrash

A large disk swap is what stretches an outage from seconds into minutes at the console. Swappiness tunes the kernel’s balance between reclaiming anonymous pages (swapping out your processes’ memory) and reclaiming file-backed pages (dropping cached files). Lower it and the kernel leans harder on RAM and reaches the OOM decision sooner, instead of grinding swap for minutes first. It does not stop thrashing; it shortens the time to a decision.

printf 'vm.swappiness=10n' > /etc/sysctl.d/99-swappiness.conf && sysctl --system

The default on most distributions is 60. Dropping it to 10 does not disable swap; it just stops the kernel reaching for it so eagerly that the box becomes unresponsive long before anything gets killed. The kernel vm sysctl docs have the full parameter set.

The gotcha: socket activation sends you hardening the wrong unit

After applying Layer 1, I verified it the obvious way:

for p in sshd systemd-logind; do
  pid=$(pgrep -o -x $p)
  echo "$p -> $(cat /proc/$pid/oom_score_adj)"
done

systemd-logind read -1000. But the sshd that owns a login read -900, not the value I wrote. Still deep in “never kill” territory, but not what I set. A value you did not write means something else is winning, and you need to know what.

My first guess was the socket. Modern Ubuntu is socket-activated: the thing listening on port 22 is not a long-running sshd you started, it is systemd itself (PID 1) holding the port open via ssh.socket. So I assumed the process handling logins was spawned from the socket, that my ssh.service drop-in was decorating the wrong unit, and that the shield had to move to ssh.socket. I put -1000 on the socket. The login shell still read -900.

The socket was a dead end for a concrete reason. ssh.socket uses Accept=no. It does not spawn a fresh sshd per connection inetd-style. It activates ssh.service, which runs a single long-lived sshd -D, and that daemon forks a child for every login. The login child is a descendant of ssh.service, so it inherits ssh.service‘s OOMScoreAdjust, never the socket’s. I proved it live: with -1000 on ssh.socket and -900 on ssh.service, a fresh login’s sshd read -900. The socket’s value was irrelevant.

So the shield belongs on ssh.service, which is exactly where Layer 1 put it. That leaves one loose end: where did the -900 come from, when I had written -1000? The droplet image. DigitalOcean ships its own ssh.service configuration that sets -900 (its droplet agents run at that same score throughout my OOM logs), and that vendor value was outranking my drop-in. systemd merges drop-ins by filename, last one wins, so a stock file that sorts after your oom.conf quietly takes precedence. I cannot prove DigitalOcean’s exact filename and it does not change the fix: put the shield on ssh.service, and if a vendor drop-in is winning, name yours to sort last.

mkdir -p /etc/systemd/system/ssh.service.d
printf '[Service]nOOMScoreAdjust=-1000n' > /etc/systemd/system/ssh.service.d/99-oom-shield.conf
systemctl daemon-reload && systemctl restart ssh

Check whether your box is socket-activated. It does not change where the shield goes, but it explains the confusion if you go looking:

systemctl is-active ssh.socket    # "active" means socket-activated

Then open a fresh SSH session (your current one predates the change and did not inherit it) and confirm the sshd that owns your shell carries the value you meant. Read the live process, not the config file:

systemctl show ssh.service -p OOMScoreAdjust        # what systemd will hand new logins
cat /proc/$(ps -o ppid= -p $$)/oom_score_adj        # what your login actually inherited

A -900 from the droplet image is already deep enough to keep the login path off the kill list, so if that is what you read, you are safe. A -1000 drop-in that wins gets you the exact value and certainty about which unit is in control.

What actually happened: the real hog never died

The shape of the failure changed which fixes I trusted.

The machine is a 15 GB droplet with 8 GB of swap. When I started digging it had been up 18 days, carrying a one-minute load average above 11 and having peaked far higher during the event. The kernel journal goes back to March 4, and it holds not two OOM events but three: a small one on June 6 (three kills), the real storm on June 20 (eleven kills in roughly half an hour, six of them in a single second), and a third on June 22. The same trigger fired each time: one specific fault that kept coming back until I caged the thing causing it.

The June 20 log looked like a swarm. Six processes dropped in the same second, each a small Python or uvicorn worker:

Jun 20 10:29:20 kernel: Out of memory: Killed process 3346155 (uvicorn) ... UID:1000 oom_score_adj:200
Jun 20 10:29:20 kernel: Out of memory: Killed process 3346734 (python3) ... UID:1000 oom_score_adj:200
Jun 20 10:29:20 kernel: Out of memory: Killed process 3344391 (python)  ... UID:1002 oom_score_adj:200
Jun 20 10:29:20 kernel: Out of memory: Killed process 3346774 (python3) ... UID:1000 oom_score_adj:200
Jun 20 10:29:20 kernel: Out of memory: Killed process 3347555 (python3) ... UID:1000 oom_score_adj:200
Jun 20 10:29:20 kernel: Out of memory: Killed process 3347946 (python3) ... UID:1000 oom_score_adj:200

The obvious read is that a cluster of small processes collectively ate the box. The full victim list says otherwise.

When I pulled every OOM kill across all three events (one grep for Out of memory: Killed), the numbers did not describe a memory storm. Every process the kernel killed was tiny. The biggest single victim was syncthing at 43 MB; most were Python and uvicorn helpers in the 1 to 25 MB range, plus the systemd --user manager at roughly 1 MB on June 22. All fifteen kills add up to about 186 MB. The box has 15.6 GB of RAM, so during the June 20 event the kernel thrashed for half an hour and reclaimed roughly one percent of what was being eaten.

The real hog was never in the kill list. A separate cgroup probe clocked my per-user manager (user@1000.service) at 14.66 GB, with a single scope sitting at 12.5 GB on its own. That process, an 11 to 15 GB Python glutton from a runaway run, went untouched through every single OOM event.

Every victim carried oom_score_adj:200, the “kill me first” tag. My automation had marked its small, disposable helpers as expendable, which sounds responsible until you see what it does to the kernel’s badness math. The killer keeps picking those pre-tagged helpers, freeing 1 to 25 MB a shot, while the real glutton, carrying a normal score, never rises to the top. The tag did not just fail to save me; it pointed the killer at the disposable helpers and left the actual offender alone.

So this was not a swarm collectively eating the box. It was a single escapee doing the eating while a cloud of mis-tagged helpers got killed instead. The kills clustered under one account, my own login user (UID 1000, where the automation runs); the lone UID 1002 process was a near-idle service account the kernel simply reached first, not a cause. The collateral was real too: long-running services with low process IDs went down, a uvicorn at PID 1700 and a python3 at PID 1702, both up since boot. And on June 22 the kernel killed systemd --user itself, the manager behind user@1000.service, tearing down the entire user slice and every tmux session and user service under it in one shot.

This is why I stopped trusting per-process scoring as the fix. Tools that kill “the biggest process” barely help when the biggest process is effectively pre-shielded and the killer keeps reaping small helpers. Steering earlyoom --prefer at the families that fan out helps at the margin, but the durable fix cannot depend on oom_score_adj at all, since that is the exact knob that misfired here. That fix is the cgroup cap, below.

How the fan-out took down its own host

The runaway was a Claude Code agent workflow that fans work out across parallel subagents and the tools they each run. It ships with a safety default of roughly one worker per core, holding back a couple, so the real cap is cores-minus-2. This run ignored that cap and fanned out well past it. Each branch spun up its own Python and uvicorn helper, and one carried the heavyweight run that ballooned into the 11 to 15 GB glutton. Both the count and the footprint blew through what 15 GB of RAM could hold. It runs on the same $6/month DigitalOcean droplet whose memory setup I documented separately.

When a fan-out keeps launching new workers faster than the kernel can reap them, and tags every one oom_score_adj:200 on the way out, you do not have a fat-process problem you can score your way out of. You have a fork-storm that also misdirects the killer, exactly as the victim list showed. Stepping over the concurrency cap turned my own automation into a self-inflicted denial-of-service. I cover the same subagent fan-out from a security angle in a writeup on prompt-injection canaries.

A cap sized in cores is not automatically safe on a memory-light box. One worker per core assumes each worker is cheap. When each worker drags in its own interpreter and a web server, “cores minus two” can still be far more memory than the host has. Respect the cap, and size it against RAM, not just core count.

The other half of the fix: cap the run with a cgroup

Hardening the login path and capping the run answer two different questions, and you want both. Layers 1 through 4 are survivability: when something goes wrong, you can still get in and clean up. A cgroup cap is prevention: the run cannot take the whole box down in the first place. The shield holds even when the cap is misconfigured or missing; the cap depends on a limit you can still set wrong.

The trick is picking the right cgroup boundary. On a systemd box, every login user gets a user-<UID>.slice, and underneath it two things live side by side: user@<UID>.service (the per-user manager that owns long-running stuff like my automation and its tmux sessions) and the session-<N>.scope units that hold interactive logins. They are siblings, and that sibling split is what makes a safe cap possible. If you cap the entire user-<UID>.slice, you also cap the scope your own SSH session lands in, so a memory storm could throttle the very shell you logged in to fix it. Cap user@<UID>.service instead and the automation is boxed in while a fresh SSH login, living in a sibling scope, stays uncapped:

# /etc/systemd/system/user@1000.service.d/50-memory-cage.conf
# (1000 = the login user my agents run under; NOT the whole user-1000.slice)
[Service]
MemoryHigh=10G   # start reclaiming under pressure before the wall
MemoryMax=11G    # hard ceiling; kills happen INSIDE this cgroup, not on sshd

MemoryHigh is a soft throttle: cross it and the kernel aggressively reclaims memory from the cgroup, slowing it down. MemoryMax is the wall: cross it and the kernel OOM-kills processes inside that cgroup specifically, leaving everything outside it alone. On a 15 GB box, an 11 GB ceiling leaves headroom for the system services, the provider’s own agents, sshd, and a rescue shell, so an over-eager fan-out can starve itself without dragging down the login path or anything else.

A tmux attach pane is a child of user@<UID>.service, so it lives inside the cage. During an incident, SSH in and work in a fresh shell rather than reattaching to the tmux session you were running the job from, or you land right back inside the capped cgroup you are trying to escape.

This is now live. After the third OOM event on June 22, the one where the kernel killed my user-manager and cascaded the whole slice, I applied MemoryHigh=10G and MemoryMax=11G to user@1000.service with a drop-in plus systemctl set-property, and it survives a reboot. It goes on user@1000.service, not user-1000.slice, for the sibling-scope reason above. Layers 1 through 4 and the cage are all live and verified today. The cage is also the one fix here that does not lean on per-process oom_score_adj scoring, the mechanism that misfired in the post-mortem: MemoryMax keeps killing inside the cage until it drops back under the ceiling, so it eventually reaches the big process the badness heuristic never touched, and it never lets the pressure leak out to sshd or PID 1.

Levers I deliberately did not pull

A few knobs I left alone on purpose. vm.overcommit_memory is a footgun on a general-purpose box; turning off overcommit makes well-behaved processes fail allocations they would normally get away with, so I left it at the default. zram (a compressed RAM-backed swap device) is a real option if you genuinely cannot add memory, but it trades CPU for headroom and does not address a fork-storm. The cgroup cap is really a proxy for the actual fix: this workload wants more RAM than a 15 GB box has, and right-sizing the host is the honest answer the cap only approximates.

The short version

• “SSH is running” does not equal “I can log in.” A new login is a memory allocation, and allocations fail under pressure.
• Your out-of-band console (provider recovery console or serial console) is your most reliable backstop. Learn how to reach it before you need it.
• The fix is four layers: shield the login path (OOMScoreAdjust=-1000), reserve memory for root (vm.admin_reserve_kbytes), kill early in userspace (earlyoom), and stop thrashing (vm.swappiness=10).
• On modern Ubuntu, SSH is socket-activated, but ssh.socket uses Accept=no, so the login sshd inherits ssh.service‘s OOMScoreAdjust, not the socket’s. The shield belongs on ssh.service. Verify on a live connection, not the config file.
• Per-process badness scoring can misdirect the killer. Tagging disposable helpers “kill me first” made the kernel reap 1 to 25 MB victims (about 186 MB across fifteen kills) while an 11 to 15 GB hog was never touched. earlyoom --prefer helps at the margin, but the fix that does not depend on scoring is a cgroup cap.
• Agent orchestration can self-DoS its own host. Concurrency caps sized in cores are not safe on memory-light boxes. Respect the cap, and cap the run in a MemoryMax cgroup so it cannot take the machine with it.

FAQ

Why can’t I SSH into my Linux server when it’s out of memory?

A new SSH login forks a shell, allocates a pseudo-terminal, and runs the PAM authentication stack, all of which need memory. Under exhaustion those allocations block or fail. The OOM killer may also have already killed sshd or systemd-logind, the process that grants your session. The refused or hung connection is a symptom of memory pressure, not a firewall or network fault, which is why checking your firewall rules gets you nowhere.

What happens when Linux runs out of memory?

When Linux exhausts RAM and cannot reclaim enough through caching or swap, the kernel’s OOM killer frees memory by force. It scores every process with a “badness” heuristic and terminates the highest scorer, repeating until the system can continue. Before that, the machine usually thrashes swap hard enough to freeze for seconds or minutes. Critical processes like sshd and systemd-logind are fair game unless you have protected them, which is how a healthy server can refuse your login.

How do I set OOMScoreAdjust for SSH?

Add a systemd drop-in on ssh.service. Run mkdir -p /etc/systemd/system/ssh.service.d and write [Service] with OOMScoreAdjust=-1000 into a .conf file inside it, then systemctl daemon-reload && systemctl restart ssh. On socket-activated Ubuntu the shield still belongs on ssh.service, not ssh.socket (the ssh.socket FAQ below explains why). Do the same for systemd-logind, and verify on a fresh connection by reading /proc/<sshd-pid>/oom_score_adj.

What is the Linux OOM killer?

The Out-Of-Memory killer is a kernel mechanism that frees RAM by force when the system runs out and cannot reclaim enough through normal means. It assigns every process a “badness” score and terminates the highest-scoring one, repeating until there is enough memory to continue. It is a last resort that fires at true exhaustion, which is often too late to keep your machine responsive, since the swap-thrash leading up to it has already frozen things.

How do I protect sshd from the OOM killer?

Set OOMScoreAdjust=-1000 on ssh.service via a systemd drop-in, which tells the kernel to never select it as a victim. Put the shield on ssh.service, not ssh.socket (the ssh.socket FAQ below explains why). Do the same for systemd-logind. Then reserve memory for root with vm.admin_reserve_kbytes so even a fresh login has the headroom to start.

What is vm.admin_reserve_kbytes?

vm.admin_reserve_kbytes is a kernel parameter that reserves a slice of memory only the root user can allocate from. The default is around 8 MB, too little to fork a login shell during heavy pressure. Raising it to 256 MB (262144) guarantees that root can still SSH in and run recovery commands like kill even when the rest of the box is wedged. It is the single most direct fix for “I just want to log in and stop the bleeding.”

What is ssh.socket and does it help during an OOM event?

ssh.socket is the systemd unit that holds port 22 open under socket activation, the default on modern Ubuntu. Instead of a long-running sshd you start, systemd (PID 1) listens and hands the socket to ssh.service. Because ssh.socket uses Accept=no, it does not spawn a fresh sshd per connection; it activates one sshd -D under ssh.service that forks a child per login. That child inherits ssh.service‘s OOMScoreAdjust, not the socket’s, so your hardening belongs on ssh.service. Run systemctl is-active ssh.socket to confirm your box is socket-activated.