~ / troubleshooting / apache segfault

Debugging an Apache segfault on a live server

A web server that crashed every now and then, a vulnerability scanner that made it worse, and a trail that ended at a null pointer inside a small module. This is the walk through, written up so the method can be reused.

Written while still learning C debugging and the internals of a web server, so corrections are welcome.

The symptoms

• A Debian based host running Apache in the prefork model, sitting behind a reverse proxy.
• A module that rewrites the client address from a forwarding header.
• Occasional crashes, strongly correlated with security scanning.

Step one: get a core dump

A crash without a core dump is a rumour. Turn core dumps on for the server, raise the size limit before it starts, and give it a place to write.

CoreDumpDirectory /var/cache/apache2
ulimit -c unlimited

Install the debugger and the matching debug symbol packages for the server and its libraries.

apt install gdb apache2-dbg libapr1-dbg libaprutil1-dbg

Step two: read the backtrace

Open the program and the core together, then print the whole stack.

gdb /usr/sbin/apache2 /var/cache/apache2/core
bt full

In this case the fault was in the code that builds the environment for scripts, at the line that reads the remote address. Printing the two relevant pointers told the story: the connection object was valid, but its remote address was the null pointer.

(gdb) print c
$1 = (conn_rec *) 0x7f...
(gdb) print c->remote_addr
$2 = (apr_sockaddr_t *) 0x0

Step three: find who nulled it

The backtrace alone did not show the culprit, because the server runs through modules and hooks. Listing the loaded modules pointed straight at the address rewriting module, and its source showed why.

• It copies the client address from the forwarding header.
• It clears the old address structure and sets the pointer to null.
• It then asks the library to build a new address.

If that library call fails, the pointer is left null, and a later write dereferences it. There was no check on the return value. The forwarding header can carry junk, which is exactly what a scanner produces.

The lesson and the fix

The right habit is to check the return value of any call that can fail and to never leave a pointer dangling. If the address cannot be rebuilt, keep the original one and carry on. The practical workaround for a system that cannot be patched immediately is to strip extra forwarding headers at the reverse proxy before requests reach the back end.

This turned out to be a real advisory rather than a curiosity, and it was assigned CVE 2012 3526.