single apache process hogging cpu - file leak problem??
This thing is driving me nuts. Every few hours or so, a single apache process goes awol and hogs the cpu at 90-100%. I just cannot figure out why. It doesn't seem to be a rogue script, since it's only recently started. If I kill the child, then all goes back to normal, for a while.
I can't be sure, but it seems to be linked to the apache 1.3.33->1.3.34 upgrade I did on Jan 14 (since, like I say it's only fairly recent and no no new scripts/websites are running (I control all the websites, so I know what's there and not).
iostat shows nothing grinding away:
Code:
tty twed0 cpu tin tout KB/t tps MB/s us ni sy in id 0 8 30.94 2 0.05 8 0 1 0 91
Code:
last pid: 95249; load averages: 1.01, 1.05, 1.07 up 66+07:50:51 18:39:18 64 processes: 2 running, 61 sleeping, 1 stopped CPU states: 99.2% user, 0.0% nice, 0.8% system, 0.0% interrupt, 0.0% idle Mem: 288M Active, 929M Inact, 187M Wired, 48M Cache, 112M Buf, 31M Free Swap: 2048M Total, 108K Used, 2048M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 64709 nobody 132 0 110M 104M RUN 502:36 94.68% 94.68% httpd 431 root 96 0 3360K 1844K select 3:08 0.00% 0.00% sshd 14442 root 8 0 2924K 2148K nanslp 3:06 0.00% 0.00% perl5.8.7 20633 root 8 0 9468K 8340K nanslp 2:41 0.00% 0.00% perl5.8.7 366 _pflogd -58 0 1548K 1024K bpf 2:19 0.00% 0.00% pflogd 649 root 8 0 1236K 604K nanslp 1:40 0.00% 0.00% svscan 88823 skinlist 116 20 15572K 14868K STOP 1:39 0.00% 0.00% perl5.8.7 960 root -58 0 3668K 1640K bpf 1:24 0.00% 0.00% tcpdump 28134 root -58 0 3608K 1372K bpf 1:16 0.00% 0.00% tcpdump 20340 root 8 20 16704K 15188K wait 0:50 0.00% 0.00% perl5.8.7 287 root 96 0 1312K 772K select 0:43 0.00% 0.00% syslogd 64507 root 96 0 17028K 10400K select 0:27 0.00% 0.00% httpd 451 root 8 0 1356K 908K nanslp 0:21 0.00% 0.00% cron 20358 mailnull 8 0 5592K 4560K nanslp 0:16 0.00% 0.00% perl5.8.7 90569 nobody 4 0 31292K 24972K accept 0:16 0.00% 0.00% httpd 90334 nobody 4 0 29352K 23008K accept 0:16 0.00% 0.00% httpd 90314 nobody 4 0 18656K 12352K accept 0:15 0.00% 0.00% httpd 89866 nobody 4 0 18068K 11764K accept 0:15 0.00% 0.00% httpd 90436 nobody 4 0 29880K 23544K accept 0:15 0.00% 0.00% httpd 90144 nobody 4 0 18436K 12128K accept 0:15 0.00% 0.00% httpd 90435 nobody 4 0 18940K 12672K accept 0:14 0.00% 0.00% httpd 90915 nobody 4 0 18744K 12476K accept 0:14 0.00% 0.00% httpd 91504 nobody 4 0 17840K 11536K accept 0:11 0.00% 0.00% httpd 14433 mailnull 96 0 5860K 2448K select 0:09 0.00% 0.00% exim-4.60-0 754 root 96 0 2724K 1740K select 0:08 0.00% 0.00% pure-ftpd 763 root 96 0 1436K 1052K select 0:06 0.00% 0.00% inetd 20356 root 96 0 9268K 5808K select 0:02 0.00% 0.00% cppop 68127 cpanel 20 0 3528K 2424K kserel 0:02 0.00% 0.00% stunnel-4.04local 68141 root 100 0 11128K 7296K select 0:01 0.00% 0.00% cpsrvd 94720 nobody 4 0 17816K 11228K accept 0:01 0.00% 0.00% httpd 88822 skinlist 8 20 940K 476K nanslp 0:01 0.00% 0.00% logrunner 95214 mysql 20 0 57332K 26072K kserel 0:00 0.00% 0.00% mysqld
lsof -p 64709:
Code:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 64709 nobody cwd VDIR 4,19 5632 4192772 /home/skinlist/public_html/ct/includes httpd 64709 nobody rtd VDIR 4,12 1024 2 / httpd 64709 nobody txt VREG 4,18 616739 1345062 /usr/local/apache/bin/httpd httpd 64709 nobody txt VREG 4,12 141604 32936 /libexec/ld-elf.so.1 httpd 64709 nobody txt VREG 4,12 28644 13 /lib/libcrypt.so.2 httpd 64709 nobody txt VREG 4,18 188080 259353 /usr/lib/libssl.so.3 httpd 64709 nobody txt VREG 4,12 1005832 36 /lib/libcrypto.so.3 httpd 64709 nobody txt VREG 4,18 248574 259500 /usr/lib/libexpat.so.4 httpd 64709 nobody txt VREG 4,12 882780 24 /lib/libc.so.5 httpd 64709 nobody txt VREG 4,18 59239 1345093 /usr/local/apache/libexec/mod_rewrite.so httpd 64709 nobody txt VREG 4,18 9531 1345094 /usr/local/apache/libexec/mod_expires.so httpd 64709 nobody txt VREG 4,18 7655 1349973 /usr/local/apache/libexec/mod_bwlimited.so httpd 64709 nobody txt VREG 4,18 7929 1349974 /usr/local/apache/libexec/mod_log_bytes.so httpd 64709 nobody txt VREG 4,18 7749 1349975 /usr/local/apache/libexec/mod_auth_passthrough.so httpd 64709 nobody txt VREG 4,18 76439 1350233 /usr/local/apache/libexec/mod_gzip.so httpd 64709 nobody txt VREG 4,18 72470 1350234 /usr/local/apache/libexec/mod_security.so httpd 64709 nobody txt VREG 4,18 12971 1350232 /usr/local/apache/libexec/mod_geoip.so httpd 64709 nobody txt VREG 4,18 27382 969496 /usr/local/lib/libGeoIP.so.4 httpd 64709 nobody txt VREG 4,18 7800337 1345098 /usr/local/apache/libexec/libphp5.so httpd 64709 nobody txt VREG 4,18 834196 259319 /usr/lib/libstdc++.so.4 httpd 64709 nobody txt VREG 4,18 774875 967572 /usr/local/lib/libc-client4.so.8 httpd 64709 nobody txt VREG 4,18 374566 1107993 /usr/local/lib/mysql/libmysqlclient.so.14 httpd 64709 nobody txt VREG 4,18 179411 259486 /usr/lib/libmhash.so.2 httpd 64709 nobody txt VREG 4,18 318139 259485 /usr/lib/libmcrypt.so.8 httpd 64709 nobody txt VREG 4,18 31949 968041 /usr/local/lib/libltdl.so.4 httpd 64709 nobody txt VREG 4,18 25800 259252 /usr/lib/libpam.so.2 httpd 64709 nobody txt VREG 4,18 42049 970863 /usr/local/lib/libintl.so.6 httpd 64709 nobody txt VREG 4,18 469562 965954 /usr/local/lib/libfreetype.so.9 httpd 64709 nobody txt VREG 4,18 881458 1579813 /usr/X11R6/lib/libX11.so.6 httpd 64709 nobody txt VREG 4,18 64658 1579869 /usr/X11R6/lib/libXpm.so.4 httpd 64709 nobody txt VREG 4,18 140812 967082 /usr/local/lib/libpng.so.5 httpd 64709 nobody txt VREG 4,12 61836 34 /lib/libz.so.2 httpd 64709 nobody txt VREG 4,18 131801 970756 /usr/local/lib/libjpeg.so.9 httpd 64709 nobody txt VREG 4,12 114504 15 /lib/libm.so.3 httpd 64709 nobody txt VREG 4,18 1007038 967947 /usr/local/lib/libiconv.so.3 httpd 64709 nobody txt VREG 4,18 257680 969741 /usr/local/lib/libssl.so.4 httpd 64709 nobody txt VREG 4,18 1307722 969733 /usr/local/lib/libcrypto.so.4 httpd 64709 nobody txt VREG 4,18 1057736 259495 /usr/lib/libcurl.so.3 httpd 64709 nobody txt VREG 4,18 1927143 259491 /usr/lib/libxml2.so.8 httpd 64709 nobody txt VREG 4,18 63244 1579845 /usr/X11R6/lib/libXext.so.6 httpd 64709 nobody txt VREG 4,18 133164 259262 /usr/lib/libpthread.so.1 httpd 64709 nobody txt VREG 4,18 15207 1342476 /usr/local/apache/libexec/mod_evasive.so httpd 64709 nobody txt VREG 4,18 8708 1207053 /usr/local/Zend/lib/ZendExtensionManager.so httpd 64709 nobody txt VREG 4,18 466932 1486901 /usr/local/Zend/lib/Optimizer-2.5.10/php-5.0.x/ZendOptimizer.so httpd 64709 nobody 0r VCHR 2,2 0t0 16 /dev/null httpd 64709 nobody 1w VCHR 2,2 0t0 16 /dev/null httpd 64709 nobody 2w VREG 4,18 36983168 1345868 /usr/local/apache/logs/error_log httpd 64709 nobody 3u VREG 4,16 0 91 /tmp (/dev/twed0s1e) httpd 64709 nobody 4u IPv4 0t0 TCP no PCB, CANTSENDMORE, CANTRCVMORE httpd 64709 nobody 5r VREG 4,18 832246 1130635 /usr/local/share/GeoIP/GeoIP.dat httpd 64709 nobody 6u VREG 4,16 1003628 101 /tmp (/dev/twed0s1e) httpd 64709 nobody 7u unix 0xc3468654 0t0 ->0xc7435a20 httpd 64709 nobody 8r VREG 4,19 43 4192404 /home/skinlist/public_html/ct/1pixel.gif httpd 64709 nobody 15w VREG 4,18 1248495 1350251 /usr/local/apache/logs/audit_log httpd 64709 nobody 16w VREG 4,18 36983168 1345868 /usr/local/apache/logs/error_log httpd 64709 nobody 17u IPv4 0xc745f380 0t0 TCP *:http (LISTEN) ...logfiles
httpd 64709 nobody 4u IPv4 0t0 TCP no PCB, CANTSENDMORE, CANTRCVMORE
what is this?? I searched and saw some apache bug for file leaking or something....
any clues?
httpd.conf:
RLimitMEM 202365269
RLimitCPU 240
Alias /bandwidth/ /usr/local/bandmin/htdocs/
ServerType standalone
ServerRoot "/usr/local/apache"
PidFile /usr/local/apache/logs/httpd.pid
ScoreBoardFile /usr/local/apache/logs/httpd.scoreboard
Timeout 240
KeepAlive On
MaxKeepAliveRequests 50
KeepAliveTimeout 5
MinSpareServers 5
MaxSpareServers 10
StartServers 10
MaxClients 350
MaxRequestsPerChild 1000
FreeBSD 5.3-RELEASE with WHM 10.8.0 cPanel 10.8.1-S114
