~ / troubleshooting / php bus error

A PHP bus error under load

A shop platform started crashing its PHP workers with a bus error, and the more visitors it had, the more often it fell over. The cause was not exotic. It was a cache file being rewritten while another process was reading it.

A shop is only one of several hosted services here, and none of the application code was written by the person chasing the crash.

The symptoms

• Random crashes of PHP workers, each one killing a request.
• Because a worker died mid flight, parts of the site appeared to hang.
• The problem grew sharply with the number of concurrent visitors.

Capture the crash

A bus error produces a core dump. Set the pattern so cores land somewhere useful, allow the service to write large cores, and wait for the fault.

echo "/tmp/core-%e.%p" > /proc/sys/kernel/core_pattern
# add 'ulimit -c unlimited' to the PHP service init
# and restart the service

Open the core with the debugger and the matching debug package. In this case the crash was inside the PHP parser, at the scanner stage, while reading a generated file under the application cache directory.

Why a cache file crashes a parser

The file is PHP code that gets included on requests. The parser was reading it exactly while another worker was rewriting it in place. A parser that sees a half written file can read past the end of a buffer and raise a bus error. The file itself was not corrupt on disk afterwards; the corruption existed only in the moment between two processes.

The root cause

The writing routine used a plain write to the destination file. That is not atomic: the file is opened and truncated first, then filled. Between those two moments the file is incomplete, and any other process that includes it is reading a lie. The writing code tried to cover this with a retry loop and a check of the final bytes, which treats the symptom and not the cause.

The fix

Publish the file atomically. Write the new content to a temporary file in the same directory, then rename it over the target. Rename is atomic on a local filesystem, so a reader sees either the old file or the new one, never a partial one.

$tmp = $filename . '.tmp.' . getmypid();
if (file_put_contents($tmp, $content, LOCK_EX) !== false) {
    rename($tmp, $filename);
} else {
    error_log('could not write ' . $tmp);
}

The change is small, but it removes the race entirely. The lesson generalises: whenever one process can read a file while another writes it, the write has to be atomic. A lock is not enough on its own, because not every reader respects locks, and the language runtime is one of those readers.

An entry in the PHP bug tracker describes a related crash when the parser reads a file that another process is still writing.