Today I noticed that my server is not responding to httpd, so I started to ivestigate. First of all I tried to ssh into the box - it was difficult, because the load was very high. After some minutes I was able to ssh, here is what I get from /var/log/messages (it is CentOs 4.7 box)


Oct 3 20:22:35 host-1 kernel: oom-killer: gfp_mask=0xd0
Oct 3 20:24:10 host-1 kernel: Mem-info:
Oct 3 20:24:10 host-1 kernel: DMA per-cpu:
Oct 3 20:25:02 host-1 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct 3 20:25:02 host-1 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct 3 20:27:39 host-1 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct 3 20:29:57 host-1 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct 3 20:35:05 host-1 kernel: Normal per-cpu:
Oct 3 20:39:46 host-1 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct 3 20:39:46 host-1 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct 3 20:44:40 host-1 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct 3 20:45:51 host-1 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct 3 20:46:41 host-1 kernel: HighMem per-cpu:
Oct 3 20:46:59 host-1 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct 3 20:46:59 host-1 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct 3 20:47:09 host-1 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct 3 20:47:21 host-1 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct 3 20:47:21 host-1 kernel:
Oct 3 20:47:26 host-1 kernel: Free pages: 16920kB (704kB HighMem)
Oct 3 20:47:28 host-1 kernel: Active:129408 inactive:342648 dirty:180711 writeback:4315 unstable:0 free:4230 slab:33396 mapped:100166 pagetables:4750
Oct 3 20:47:30 host-1 kernel: DMA free:12520kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
Oct 3 20:47:32 host-1 kernel: protections[]: 0 0 0
Oct 3 20:47:35 host-1 kernel: Normal free:3696kB min:3728kB low:7456kB high:11184kB active:264kB inactive:726944kB present:901120kB pages_scanned:1003794 all_unreclaimable? yes
Oct 3 20:47:36 host-1 kernel: protections[]: 0 0 0
Oct 3 20:47:37 host-1 kernel: HighMem free:704kB min:512kB low:1024kB high:1536kB active:517368kB inactive:643648kB present:1178560kB pages_scanned:0 all_unreclaimable? no
Oct 3 20:47:39 host-1 kernel: protections[]: 0 0 0
Oct 3 20:47:39 host-1 kernel: DMA: 4*4kB 3*8kB 4*16kB 2*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12520kB
Oct 3 20:47:40 host-1 kernel: Normal: 4*4kB 118*8kB 5*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3696kB
Oct 3 20:47:44 host-1 kernel: HighMem: 44*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 704kB
Oct 3 20:47:44 host-1 kernel: Swap cache: add 61304, delete 50672, find 161479/164254, race 0+32
Oct 3 20:47:45 host-1 kernel: 0 bounce buffer pages
Oct 3 20:47:47 host-1 kernel: Free swap: 1884632kB
Oct 3 20:47:47 host-1 kernel: 524016 pages of RAM
Oct 3 20:47:47 host-1 kernel: 294640 pages of HIGHMEM
Oct 3 20:47:47 host-1 kernel: 5581 reserved pages
Oct 3 20:47:47 host-1 kernel: 577377 pages shared
Oct 3 20:47:47 host-1 kernel: 10632 pages swap cached
Oct 3 20:47:48 host-1 kernel: Out of Memory: Killed process 27945 (httpd).


Could anyone help me how to track what caused this problem? I know that httpd was killed, but I have RMemLimit in Apache configuration and I did never have had such problems before. Also, the server is working well after this very high load.