Mysterious daily huge load spikes by apache...?
I have an Ensim 3.1.1 system - P4 1.7GHz, 1GB RAM, 40GB WD 7200rpm drive. I've not modified much on the system, mainly only installed Bastille/psad, MRTG, and LogSentry, upgraded glibc, and did a few custom bug fixes (i.e. Mod_jk fix).
For some reason, several times a day, one or several seemingly harmless apache processes begin consuming massive amounts of CPU... it's always a single process that consume 60-90% cpu, or a few separate ones, each taking 15-20%. I note the PIDs and track them down to the actual responsible files in "server-status" output, and they always seem to be completely harmless files, such as a simple .GIF file or something.
Here's one example out of a tracked-down PID in server-status from last night:
31-0 9755 0/24/11286 _ 0.04 219 0 0.0 0.04 19.98 211.28.96.8 www.domain.com GET /general/s/ra_o.gif HTTP/1.0
Kill -9 on the processes only works sometimes.... other times it takes "2" stop/starts on apache to rid the system of these things. When it's happening, the load is as much as 15, and the instant I kill the pid(s) or restart apache (twice) the load plummets back down to around 0.30 (and this is a busy server hehe)... so I know it has nothing to do with the "actual load" on the server. I don't have enough stats recorded in MRTG yet to see if it happens daily at a specific time (this is a pretty new box). Also "so far" I've only seen and tracked down PIDs to files that belong to 2 certain sites on the server, and no others - but always like I said, completely harmless files (usually .gif).
Anyhow I'm hoping someone can advise me on what to do.
Thanks in advance for your help
