High loads of 500 on linux Web Servers
I have a site that has a lot of traffic with loads at around 2. Just recently we found out that every morning at around 7-9am the load on all our web servers (3 in total with a load balancer) have loads of about 100. One time I have noticed in an afternoon between 2 - 3:30pm there were loads of even up to 500.
We restarted apache and it would bring down the loads gradually at times, but today it didn't go down. It stayed around 20 - 45. I wrote a script yesterday that would restart apache when loads above 50 are reached.
During the time of high loads, the following information were recorded on all 3 servers:
-CPU: stayed around 4%
-swap: didn't move for "used" and "free": Swap: 979924k total, 34720k used, 945204k free, 1163888k cached
-Scripts: none were running in the cron job during this time.
-Top file - tasks: were at about 200
-process: a lot of-> 25860 www-data 17 0 6868 6864 5996 S 1.3 0.3 0:00.14 apache
-pache error logs: I don't see any hints there.
-Number of people on our web site: normal compared to other times, and should be less because it is in the morning.
We also have separate database servers (3 in total with a load balancer as well). The loads look normal over there but the ipvsadm from the load balancer show abnormal high connections during this time. Our php applications from the web servers talk to these DB servers.
After 10pm today, everything went back to normal. During this time there were about 6000 people on the site. I have concluded the following, perhaps I could be wrong:
1) The problem is not the number people online.
2) It is not the databases.
3) It is not the cron job scripts.
4) I strongly don't think it is the php application script for the site.
5) It has to do with something internally on all 3 web servers (by the way, they are all running Linux - Debian).
Any help would be greatly appreciated. I am currently very puzzled and don't know where else to look.
Thank you,
Donny
