Every WordPress site on my web server, including this one, returned HTTP 500 one evening. MySQL had been killed by the kernel’s out-of-memory killer, and the obvious conclusion was that MySQL had a memory problem. It did not. It was the victim, and the real cause was arithmetic I should have done years ago. Here is the post-mortem, with the numbers.
Symptoms that mislead
Apache was running. systemctl status apache2 said active, pages loaded, and they loaded a database error. That is why the first ten minutes went on the wrong thing. The journal told the real story:
journalctl -k | grep -i oom
# apache2 invoked oom-killer ...
# Out of memory: Killed process 1234 (mysqld)
Apache triggered the OOM killer, and the kernel chose mysqld because it was the largest single process. The restart counter on the MySQL unit was already at eight, so it had been happening for days and recovering on its own.
Why MySQL would not start
After the final kill, systemctl start mysql was refused. Repeated restarts had tripped systemd’s start rate limit and latched the unit as failed. You have to clear that first:
systemctl reset-failed mysql
systemctl start mysql
The arithmetic
The server is a VM with 5.8 GB of RAM running Apache with mod_php in prefork mode. The configuration had MaxRequestWorkers 150 and MaxConnectionsPerChild 0, which means up to 150 PHP-carrying children that are never recycled.
I measured the children rather than guessing: average resident size 146 MB, maximum 182 MB. 150 workers at 146 MB is nearly 22 GB. On a 5.8 GB machine. The only reason it had survived so long is that nothing had ever pushed Apache to its limit.
What pushed it
The access logs showed around 9,500 POST requests to xmlrpc.php in one day from a spread of addresses, all claiming to be Jetpack. It was a pingback and brute-force amplification attempt, and each request spins up a full PHP worker. Enough concurrent requests, enough workers, and MySQL gets evicted.
The fixes
- Size prefork to the RAM you have.
MaxRequestWorkerswent from 150 to 20. 20 workers at 182 MB worst case is 3.6 GB, which leaves room for MySQL.MaxConnectionsPerChildis now 1000 so leaked memory gets returned. - Block xmlrpc.php in an Apache conf snippet, with an exception for the one site that actually uses Jetpack. I then added the same block as a Cloudflare rule so the requests never reach the server at all.
- Protect MySQL from the OOM killer with a systemd drop-in:
OOMScoreAdjust=-500andRestartSec=5. If something must die, it should be an Apache child. - Restart, not reload. MPM settings and PHP ini changes need a full
systemctl restart apache2. A graceful reload keeps the old limits.
The trap in wp-config.php
While tuning PHP’s memory_limit I discovered it was being ignored. WordPress sets its own limit with WP_MEMORY_LIMIT, and that overrides php.ini for WordPress requests. If you are doing memory arithmetic, set it deliberately in each site’s wp-config.php rather than assuming php.ini applies.
What I took from it
The process that gets killed is rarely the process that caused the problem. Measure the real per-worker memory, multiply by the worker limit, and compare it with the RAM in the box. If the answer is bigger than the box, you have an outage waiting for a traffic spike, and bots will supply the spike for free.