Ultra high IoWAIT

Dual Xeon 2.8, 2 GB RAM, SATA drives:

Code:
top - 20:37:11 up 19:49,  2 users,  load average: 16.41, 18.05, 16.04
Tasks: 257 total,   1 running, 255 sleeping,   0 stopped,   1 zombie
Cpu(s): 21.9% us,  7.0% sy,  0.0% ni, 21.7% id, 48.2% wa,  1.2% hi,  0.0% si
Mem:   2074424k total,  1316712k used,   757712k free,    32068k buffers
Swap:  1052248k total,     2492k used,  1049756k free,   668112k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18986 nobody    15   0 62216  24m  10m S 16.7  1.2   0:02.11 httpd
19451 nobody    16   0 68832  30m  10m S 13.8  1.5   0:02.06 httpd
19450 nobody    15   0 64116  24m 9292 S 12.5  1.2   0:01.07 httpd
18437 mysql     16   0  245m 105m 2492 S  9.3  5.2   0:02.47 mysqld
18214 mysql     16   0  245m 105m 2492 S  5.1  5.2   0:03.81 mysqld
19443 nobody    15   0 61812  22m 9104 S  5.1  1.1   0:00.81 httpd
18289 mysql     16   0  245m 105m 2492 S  4.2  5.2   0:02.35 mysqld
19433 nobody    15   0 63772  25m 9.8m S  3.9  1.2   0:02.00 httpd
19490 nobody    16   0 60700  20m 7712 S  3.5  1.0   0:00.28 httpd
19439 nobody    15   0 61948  22m 8876 S  3.2  1.1   0:00.51 httpd
19426 nobody    15   0 63884  25m 9.8m S  2.9  1.3   0:01.32 httpd
19494 nobody    16   0 63000  21m 7180 S  2.6  1.1   0:00.51 httpd
18207 mysql     16   0  245m 105m 2492 S  2.2  5.2   0:01.35 mysqld
19428 nobody    16   0 61160  19m 6888 S  2.2  1.0   0:00.62 httpd
19453 nobody    16   0 62532  23m 9.8m S  1.9  1.2   0:00.95 httpd
19554 root      18   0 11840 7488 2988 D  1.9  0.4   0:00.67 rpm
19498 nobody    15   0 64072  26m  10m S  1.6  1.3   0:01.14 httpd
18196 mysql     16   0  245m 105m 2492 S  1.3  5.2   0:00.46 mysqld
19425 nobody    15   0 63340  24m 9752 S  1.3  1.2   0:01.15 httpd
19448 nobody    16   0 64052  25m 9.9m S  1.3  1.3   0:01.29 httpd
19499 nobody    16   0 60720  20m 8548 S  1.3  1.0   0:00.41 httpd
18424 mysql     16   0  245m 105m 2492 S  1.0  5.2   0:01.30 mysqld
18427 mysql     16   0  245m 105m 2492 S  1.0  5.2   0:02.68 mysqld
    6 root      RT   0     0    0    0 S  0.6  0.0   2:46.69 migration/2
 3011 root       0 -20     0    0    0 D  0.6  0.0   2:56.40 loop0
18238 mysql     15   0  245m 105m 2492 S  0.6  5.2   0:03.12 mysqld
18362 mysql     16   0  245m 105m 2492 S  0.6  5.2   0:02.55 mysqld
19424 nobody    15   0     0    0    0 Z  0.6  0.0   0:01.16 httpd <defunct>
19429 nobody    15   0 61248  22m 9800 D  0.6  1.1   0:00.96 httpd
19430 nobody    16   0 60596  20m 8252 S  0.6  1.0   0:00.58 httpd
19437 nobody    16   0 64172  22m 7208 S  0.6  1.1   0:00.99 httpd
19438 nobody    16   0 61992  22m 8940 D  0.6  1.1   0:00.50 httpd
19476 nobody    16   0 60812  20m 8472 S  0.6  1.0   0:00.53 httpd
19495 nobody    16   0 63852  21m 5924 S  0.6  1.1   0:00.77 httpd
19497 nobody    15   0 61156  19m 6888 S  0.6  1.0   0:00.39 httpd
19500 nobody    15   0 60360  19m 7480 S  0.6  1.0   0:00.41 httpd
    8 root      RT   0     0    0    0 S  0.3  0.0   1:10.49 migration/3
  215 root      15   0     0    0    0 S  0.3  0.0   8:27.27 kjournald
 2772 mailman   16   0  9464 5276 2004 S  0.3  0.3   0:07.33 python
 2774 mailman   16   0  9512 6228 2088 S  0.3  0.3   0:09.66 python
17726 root      16   0 58464  14m 4072 S  0.3  0.7   0:03.10 httpd
18172 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:00.27 mysqld
18173 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:00.15 mysqld
18180 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:00.88 mysqld
18215 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:03.87 mysqld
18320 mysql     15   0  245m 105m 2492 S  0.3  5.2   0:05.21 mysqld
18322 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:01.20 mysqld
18356 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:03.80 mysqld
18417 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:01.68 mysqld
18419 mysql     16   0  245m 105m 2492 S  0.3  5.2   0:01.34 mysqld
19411 root      16   0  2296 1072  744 R  0.3  0.1   0:00.63 top
19412 nobody    15   0 60900  21m 8768 S  0.3  1.0   0:00.78 httpd
19431 nobody    16   0 62240  24m  10m S  0.3  1.2   0:00.93 httpd
iowait is falling from 40 to 90%. System is Centos 4. I test both 2.6.9-22.0.2.ELsmp and 2.6.9-11.0.2.ELsmp kernels and the effect is the same..

I'm going crazy with this server.. Any ideas what's wrong?

 

 

 

 

Top