Outragous Load with no traffic puzzle

specs: Intel 3.6, 2 GB RAM, CentOS 4.1, Cpanel
21 low to moderate traffic sites

I just got a new server from LT and in the past 3 days, load has spiked upwards of 200 (highest 338) bringing the server to it's knees. Fortunately, I've been logged in via SSH during two of these and found nothing suspicious when I run top (though it takes 10-13 mins for top to dispay output). I can't even restart Apache without rebooting the entire box.

My first thought was that I was hacked though I had paid a very reputable admin co. to secure it immediately after it was brought online. Apache had been optimized also. I checked /tmp, /var/tmp and /dev/shm and found nothing. Cpanel logs don't how anything except cpsrvd and named failing dozens of times daily.

I did find this in /var/log/mesages but it doen's point to the problem that caused it.
Code: 
 Nov 20 10:52:40 server kernel: Free pages: 1032kB (112kB HighMem)
Nov 20 10:52:40 server kernel: Active:149692 inactive:92249 dirty:1 writeback:0 unstable:0 free:258 slab:5334 mapped:245314 pagetables:3129
Nov 20 10:52:40 server kernel: DMA free:16kB min:16kB low:32kB high:48kB active:6552kB inactive:5184kB present:16384kB pages_scanned:93027 all_unreclaimable?
yes
Nov 20 10:52:40 server kernel: protections[]: 0 0 0
Nov 20 10:52:40 server kernel: Normal free:904kB min:936kB low:1872kB high:2808kB active:510120kB inactive:329768kB present:901120kB pages_scanned:4888913 all
_unreclaimable? yes
Nov 20 10:52:40 server kernel: protections[]: 0 0 0
Nov 20 10:52:40 server kernel: HighMem free:112kB min:128kB low:256kB high:384kB active:82096kB inactive:34044kB present:122684kB pages_scanned:634953 all_unr
eclaimable? yes
Nov 20 10:52:40 server kernel: protections[]: 0 0 0
Nov 20 10:52:40 server kernel: DMA: 0*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16kB
Nov 20 10:52:40 server kernel: Normal: 14*4kB 74*8kB 12*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
Nov 20 10:52:40 server kernel: HighMem: 8*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 20 10:52:40 server kernel: Swap cache: add 6241673, delete 6241315, find 2964983/3792546, race 5+194
Nov 20 10:52:40 server kernel: Free swap: 0kB
Nov 20 10:52:40 server kernel: 260047 pages of RAM
Nov 20 10:52:40 server kernel: 30671 pages of HIGHMEM
Nov 20 10:52:40 server kernel: 3147 reserved pages
Nov 20 10:52:40 server kernel: 81993 pages shared
Nov 20 10:52:40 server kernel: 358 pages swap cached
Nov 20 10:52:40 server kernel: Out of Memory: Killed process 19531 (httpd).
Any ideas on what could be causing this?

 

 

 

 

Top