Server Randomly Lagging
I have a server which has randomly started to lag and have problems and then be perfectly fine at other times.
The server is a Dual Xeon 3.2 Ghz, 2GB RAM, RAID5 WD Raptor drive array. It has nothing but mySQL running on it with a modified my-huge.cnf config file.
On the average day, the load on this server would sit at around 2.2 never really varing much either way. This would hold true even when we would be at our highest traffic times on the website associated with this database server.
That is no longer the case. Starting Friday afternoon, the server became very unstable and highly volatile to even slight traffic increases. Our normal site traffic has become too much for this server to handle and has resulted in loads shooting up into the mid teens and lower 20s.
The problems actually began a week ago when an up2date hung and crashed the server. However, the lag wasn't much and was very brief. Up until Friday night, we had been ignoring it.
Here is all that I have done to check the server:
Ran rkhunter to check for root kits
Checked partition free space
Tweaked the hell out of the my.cnf file
Ran memtest86
Ran mprime to test the CPU, RAM, etc
Checked the RAID array status for errors
Replaced old kernel with latest CentOS kernel
Nothing has caused this to be fixed. But what is even more strange is how the problem has presented itself.
Friday night -> Almost unusable due to incredibly high lag
Saturday -> Appears fine with random patches of lag
Saturday night -> Returns to a state of extreme lag
Sunday morning -> Extreme lag, but normal loads. Turns out that RAM was being used and not released. Of the 2.0GB RAM, 1.5gb became locked and could not be freed. Rebooting the server had no effect.
Sunday afternoon -> RAM issue has vanished. Load issue has returned to the upper teens.
Current -> Server is nearly unusable due to the lag
Right now, the traffic hitting the site is half that of our normal traffic which this server could handle without any problem.
I have no idea what to do. Everything I have tried has failed. All hardware tests have returned with zero errors. I can't figure this out.
If anyone can help, please do so.
Thank you.
