Load, load and again LOAD

Hi

For the last weeks I've been trying everything possible as far as my knowledge goes to lower the load on this server. I failed Load, load and again LOAD

Here are some facts, maybe you could spend some minutes looking over them and give me some pretious advise:

1. TOP

a) normal

16:54:26 up 9 days, 4:13, 3 users, load average: 7.64, 6.74, 7.10
121 processes: 111 sleeping, 7 running, 3 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 64.0% 0.0% 35.4% 0.4% 0.0% 0.0% 0.0%
cpu00 66.9% 0.0% 32.0% 0.9% 0.0% 0.0% 0.0%
cpu01 61.1% 0.0% 38.8% 0.0% 0.0% 0.0% 0.0%

11712 apache 17 0 17548 17M 6052 S 42.1 1.7 0:03 0 httpd
11829 mysql 21 0 24000 14M 1764 R 14.0 1.4 0:01 1 mysqld
11854 mysql 19 0 24000 14M 1764 R 10.1 1.4 0:00 1 mysqld
11850 mysql 21 0 24000 14M 1764 R 6.7 1.4 0:00 1 mysqld
11625 apache 15 0 18352 17M 6124 S 5.3 1.7 0:02 1 httpd
11855 mysql 23 0 24000 14M 1764 R 4.8 1.4 0:00 1 mysqld
11851 mysql 20 0 24000 14M 1764 S 4.3 1.4 0:00 1 mysqld
11833 apache 17 0 17508 17M 6004 S 3.8 1.7 0:01 1 httpd
11298 mysql 25 0 24000 14M 1764 R 2.4 1.4 1:03 0 mysqld
10115 apache 24 0 17500 17M 5888 S 1.4 1.7 2:58 0 httpd
10218 apache 24 0 17224 16M 5812 R 1.4 1.6 2:32 0 httpd
11863 mysql 21 0 24000 14M 1764 S 0.9 1.4 0:00 0 mysqld
11626 apache 15 0 17092 16M 5964 S 0.4 1.6 0:09 0 httpd
11862 root 19 0 1152 1152 892 R 0.4 0.1 0:00 1 top


b) sort by MEM


2006 qmailq 19 0 133M 100M 1628 S 0.0 9.9 0:39 0 spamd
25279 qmailq 17 0 135M 97M 1584 S 0.0 9.7 1:57 1 spamd
4936 qmailq 22 0 133M 95M 1628 S 0.0 9.5 0:29 0 spamd
25277 qmailq 21 0 134M 84M 1480 S 0.0 8.4 2:01 1 spamd
25276 qmailq 21 0 134M 70M 1284 S 0.0 7.0 1:58 0 spamd
11625 apache 15 0 18312 17M 6092 S 0.0 1.7 0:07 0 httpd
11249 apache 15 0 18000 17M 5660 S 0.4 1.7 0:02 1 httpd
11602 apache 15 0 17820 17M 6136 S 0.0 1.7 0:14 0 httpd
11444 apache 15 0 17580 17M 6020 S 1.6 1.7 0:07 0 httpd
11712 apache 15 0 17508 17M 6020 S 0.2 1.7 0:05 1 httpd
10115 apache 25 0 17500 17M 5888 R 18.1 1.7 3:14 0 httpd
11833 apache 15 0 17488 17M 6012 S 0.1 1.7 0:05 1 httpd
11846 apache 15 0 17292 16M 6012 S 0.5 1.6 0:00 0 httpd
11226 apache 15 0 17280 16M 5976 S 0.0 1.6 0:17 0 httpd
11248 apache 15 0 17220 16M 6076 S 0.8 1.6 0:15 1 httpd
10218 apache 25 0 17188 16M 5784 R 16.7 1.6 2:51 1 httpd
11828 apache 15 0 17168 16M 6048 S 0.9 1.6 0:05 1 httpd
11839 apache 15 0 16492 16M 5924 S 0.0 1.6 0:04 0 httpd
11704 apache 15 0 16456 16M 5944 S 7.6 1.6 0:03 1 httpd
3171 mysql 19 0 23272 13M 1764 S 0.0 1.3 0:00 1 mysqld
3172 mysql 25 0 23272 13M 1764 S 0.0 1.3 0:00 1 mysqld
3173 mysql 25 0 23272 13M 1764 S 0.0 1.3 0:00 1 mysqld
3174 mysql 25 0 23272 13M 1764 S 0.0 1.3 0:00 1 mysqld
3175 mysql 25 0 23272 13M 1764 S 0.0 1.3 0:00 1 mysqld
3176 mysql 15 0 23272 13M 1764 S 0.0 1.3 0:01 1 mysqld
3177 mysql 15 0 23272 13M 1764 S 0.0 1.3 0:00 1 mysqld
3178 mysql 24 0 23272 13M 1764 S 0.0 1.3 0:00 0 mysqld
3179 mysql 15 0 23272 13M 1764 S 0.0 1.3 0:00 0 mysqld
11298 mysql 25 0 23272 13M 1764 R 21.5 1.3 1:23 1 mysqld
12014 mysql 21 0 23272 13M 1764 S 0.0 1.3 0:00 0 mysqld
9999 root 15 0 12768 12M 4596 S 0.0 1.2 0:01 0 httpd
10007 root 25 0 12568 12M 4400 S 0.0 1.2 0:00 1 httpd
1380 qmailq 15 0 11740 6404 588 S 0.0 0.6 0:09 1 clamd
7557 psaadm 15 0 7384 5312 548 S 0.0 0.5 0:03 1 httpsd
....

c) by CPU

11828 apache 18 0 17172 16M 6052 S 21.8 1.6 0:10 0 httpd
11298 mysql 25 0 23332 13M 1764 R 13.4 1.3 1:41 0 mysqld
11833 apache 16 0 17492 17M 6016 S 11.1 1.7 0:08 1 httpd
11704 apache 15 0 16460 16M 5948 S 7.2 1.6 0:13 0 httpd
10218 apache 25 0 17228 16M 5816 R 3.6 1.6 2:59 1 httpd
10115 apache 24 0 17500 17M 5888 R 2.2 1.7 3:23 1 httpd
11226 apache 15 0 17280 16M 5976 S 1.1 1.6 0:21 0 httpd
11846 apache 15 0 17308 16M 6024 S 1.1 1.6 0:02 1 httpd
11625 apache 15 0 18312 17M 6092 S 0.5 1.7 0:09 0 httpd
11839 apache 15 0 16532 16M 5964 S 0.5 1.6 0:07 1 httpd
11249 apache 15 0 18000 17M 5660 S 0.2 1.7 0:02 0 httpd
11444 apache 15 0 17588 17M 6028 S 0.2 1.7 0:09 1 httpd
11248 apache 15 0 17624 17M 6080 S 0.1 1.7 0:16 1 httpd
11602 apache 15 0 17840 17M 6156 S 0.1 1.7 0:15 0 httpd
12098 root 15 0 1152 1152 896 R 0.1 0.1 0:00 1 top
12119 mysql 22 0 23332 13M 1764 S 0.1 1.3 0:00 1 mysqld
1 root 15 0 116 84 56 S 0.0 0.0 0:14 1 init
.....


2. Sever hardware details:

a) MEM

[root@wise conf]# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 1050005504 1033625600 16379904 0 17227776 688738304
Swap: 2097434624 474648576 1622786048
MemTotal: 1025396 kB
MemFree: 15996 kB
MemShared: 0 kB
Buffers: 16824 kB
Cached: 366096 kB
SwapCached: 306500 kB
Active: 760540 kB
ActiveAnon: 543740 kB
ActiveCache: 216800 kB
Inact_dirty: 143728 kB
Inact_laundry: 28768 kB
Inact_clean: 14288 kB
Inact_target: 189464 kB
HighTotal: 131008 kB
HighFree: 1116 kB
LowTotal: 894388 kB
LowFree: 14880 kB
SwapTotal: 2048276 kB
SwapFree: 1584752 kB
Committed_AS: 1368292 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

b) CPU

model : 2
model name : Intel(R) Xeon(TM) CPU 2.00GHz
model : 2
model name : Intel(R) Xeon(TM) CPU 2.00GHz

3. CONFIG FILES

a) httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3

<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 1000
</IfModule>


<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
ThreadGuardArea off
</IfModule>

b) my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
thread_concurrency=4
max_connections=500
key_buffer=150M
myisam_sort_buffer_size=64M
join_buffer_size=1M
read_buffer_size=1M
sort_buffer_size=1M
table_cache = 1500
thread_cache_size = 128
wait_timeout = 14400
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
skip-innodb

[mysql.server]
user=mysql

[mysqld_safe]
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[mysqlhotcopy]
interactive-timeout

c) sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Disables packet forwarding
net.ipv4.ip_forward=1

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.eth0.log_martians = 0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Disables the magic-sysrq key
kernel.sysrq = 0

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
net.ipv4.tcp_sack = 0

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

# Allowed local port range
net.ipv4.ip_local_port_range = 16384 65536

4. OTHER

a)apache status: htp://hosting-wise.com/status

b) netstat:
[root@wise root]# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
1 CLOSING
17 ESTABLISHED
3 FIN_WAIT2
22 LISTEN
1 SYN_SENT
76 TIME_WAIT

c) apache 2, php 4, mysql 4, spamassassin 3 + qmail-scan + clamav

d) free -m

[root@wise etc]# free -m
total used free shared buffers cached
Mem: 1001 985 15 0 21 365
-/+ buffers/cache: 598 402
Swap: 2000 455 1544


Any other information will be provided on request.

Thank you!

 

 

 

 

Top