Server Performance Issues - Advice Needed
I am having server performance problems and wanted to get some advice from the experts here. I am running a Dual Xeon 3.0 HyperThreaded server with 4GB of ram. I run a relatively high traffic Invision Board site (pyp/mySQL) with about 16,000+ members, 600,000+ posts, and usually between 300-500 members online at a given time. My server has 4 other domains on it that are very very low traffic (less than 100MB Bandwidth per month).
Here are the symptoms. Currently the server is running very slow and often stops responding, requiring a reboot. I have tried to tweak my mySQL and Apache settings in an attempt to get better results. Last week I had the server running very fast, but with no real warning the server would crash. At the time, my memory usage was at about 95% with 50% of that cached and the rest used by apps, etc. It has not crashed in a few days, but it is running very slowly. I have included all the important config files and some output from various commands, such as Top, etc.
Any tips would be greatly appreciated.
Thanks,
-Craig
Server Info:
Kernel Version 2.4.22-1.2199.nptlsmp (SMP) Distro Name Fedora Core release 1 (Yarrow)
Processors 4 Model Intel(R) Xeon(TM) CPU 3.06GHz CPU Speed 3.07 GHz Cache Size 512 KB
httpd.conf settings
Timeout 300
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 10
MinSpareServers 30
MaxSpareServers 60
StartServers 30
MaxClients 600
my.cnf settings
[mysqld]
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_connections=500
max_user_connections=30
interactive_timeout=20
wait_timeout=20
connect_timeout=10
thread_cache_size=64
key_buffer=96M
join_buffer=1M
max_allowed_packet=16M
table_cache=1536
record_buffer=1M
sort_buffer_size=1M
read_buffer_size=1M
max_connect_errors=10
thread_concurrency=4
myisam_sort_buffer_size=64M
[mysql.server]
user=mysql
basedir=/var/lib
[safe_mysqld]
err-log=/var/log/mysqld.log
open_files_limit=8192
[mysqldump]
quick
max_allowed_packet=16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer=96M
sort_buffer=4M
read_buffer=4M
write_buffer=4M
[myisamchk]
key_buffer=96M
sort_buffer=4M
read_buffer=4M
write_buffer=4M
[mysqlhotcopy]
interactive-timeout
TOP Output:
22:55:49 up 3:08, 2 users, load average: 1.33, 1.47, 1.72
22:58:31 up 3:11, 2 users, load average: 1.22, 1.47, 1.68
23:00:13 up 3:13, 2 users, load average: 1.38, 1.46, 1.65
136 processes: 132 sleeping, 1 running, 2 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 129.6% 0.0% 41.2% 0.0% 0.0% 0.0% 228.4%
cpu00 27.3% 0.0% 8.2% 0.0% 0.0% 0.0% 64.3%
cpu01 32.4% 0.0% 12.5% 0.0% 0.0% 0.0% 54.9%
cpu02 33.0% 0.0% 9.2% 0.0% 0.0% 0.0% 57.6%
cpu03 37.0% 0.0% 11.2% 0.0% 0.0% 0.0% 51.7%
Mem: 4059648k av, 1685220k used, 2374428k free, 0k shrd, 149332k buff
779476k active, 800584k inactive
Swap: 4192924k av, 0k used, 4192924k free 1254924k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1040 mysql 15 0 61264 59M 1536 S 70.1 1.5 48:07 0 mysqld
19480 root 25 0 4200 4200 4076 S 24.6 0.1 0:47 1 httpd
14681 nobody 15 0 0 0 0 Z 4.5 0.0 0:00 2 httpd <defunc
14686 nobody 25 0 0 0 0 Z 3.5 0.0 0:00 0 httpd <defunc
13491 root 16 0 1304 1304 896 R 1.7 0.0 0:00 3 top
1361 mailman 16 0 4744 4740 1936 S 0.3 0.1 0:02 1 python2
140 root 15 0 0 0 0 SW 0.1 0.0 0:03 0 kjournald
1359 mailman 16 0 4644 4640 1896 S 0.1 0.1 0:02 0 python2
1 root 16 0 428 428 372 S 0.0 0.0 0:06 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 swapper
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 swapper
NetStat Output:
netstat -nape | grep 80 | wc -l
728
mySQL Show Status Output:
Variable_name Value
Aborted_clients 547
Aborted_connects 155
Bytes_received 91807447
Bytes_sent 1678138740
Com_admin_commands 0
Com_alter_table 0
Com_analyze 0
Com_backup_table 0
Com_begin 0
Com_change_db 37631
Com_change_master 0
Com_check 0
Com_commit 0
Com_create_db 0
Com_create_function 0
Com_create_index 0
Com_create_table 0
Com_delete 5651
Com_delete_multi 0
Com_drop_db 0
Com_drop_function 0
Com_drop_index 0
Com_drop_table 0
Com_flush 0
Com_grant 0
Com_ha_close 0
Com_ha_open 0
Com_ha_read 0
Com_insert 9906
Com_insert_select 0
Com_kill 0
Com_load 0
Com_load_master_data 0
Com_load_master_table 0
Com_lock_tables 2288
Com_optimize 0
Com_purge 0
Com_rename_table 0
Com_repair 0
Com_replace 10228
Com_replace_select 0
Com_reset 0
Com_restore_table 0
Com_revoke 0
Com_rollback 0
Com_savepoint 0
Com_select 177136
Com_set_option 223
Com_show_binlog_events 0
Com_show_binlogs 0
Com_show_create 0
Com_show_databases 8
Com_show_fields 21
Com_show_grants 2
Com_show_keys 2
Com_show_logs 0
Com_show_master_status 0
Com_show_new_master 0
Com_show_open_tables 0
Com_show_processlist 48
Com_show_slave_hosts 0
Com_show_slave_status 0
Com_show_status 10
Com_show_innodb_status 0
Com_show_tables 36
Com_show_variables 1
Com_slave_start 0
Com_slave_stop 0
Com_truncate 0
Com_unlock_tables 2288
Com_update 53934
Connections 38103
Created_tmp_disk_tables 34
Created_tmp_tables 382
Created_tmp_files 0
Delayed_insert_threads 0
Delayed_writes 0
Delayed_errors 0
Flush_commands 1
Handler_commit 0
Handler_delete 29923
Handler_read_first 574
Handler_read_key 5576341
Handler_read_next 273750078
Handler_read_prev 42451
Handler_read_rnd 956425
Handler_read_rnd_next 30089517
Handler_rollback 0
Handler_update 215193
Handler_write 305866
Key_blocks_used 41704
Key_read_requests 52489842
Key_reads 41537
Key_write_requests 136913
Key_writes 110184
Max_used_connections 32
Not_flushed_key_blocks 0
Not_flushed_delayed_rows 0
Open_tables 192
Open_files 267
Variable_name Value
Open_streams 0
Opened_tables 198
Questions 426741
Qcache_queries_in_cache 2725
Qcache_inserts 163959
Qcache_hits 89661
Qcache_lowmem_prunes 0
Qcache_not_cached 13171
Qcache_free_memory 28779992
Qcache_free_blocks 1640
Qcache_total_blocks 7159
Rpl_status NULL
Select_full_join 4
Select_full_range_join 0
Select_range 38720
Select_range_check 0
Select_scan 15887
Slave_open_temp_tables 0
Slave_running OFF
Slow_launch_threads 0
Slow_queries 17
Sort_merge_passes 0
Sort_range 11566
Sort_rows 978698
Sort_scan 10112
Table_locks_immediate 301238
Table_locks_waited 5887
Threads_cached 28
Threads_created 33
Threads_connected 5
Threads_running 1
Uptime 11614
