Do I need more memory?

Hi:
Recently my server has beening slowing down it seems. Server load is always <1 like before, but RAM usage hovers around 70%-90% now, and swap, at 10% sometimes. Here's the results of "top" and "free -m" right now:

Top:

Code: 
top - 19:45:28 up 124 days, 11:08,  1 user,  load average: 0.01, 0.05, 0.04
Tasks: 263 total,   1 running, 261 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.7% us,  0.5% sy,  0.0% ni, 98.5% id,  0.2% wa,  0.0% hi,  0.2% si
Mem:   1002260k total,   832380k used,   169880k free,    42764k buffers
Swap:  5116692k total,   480644k used,  4636048k free,   180380k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4176 root      15   0  1636  464  372 S  0.3  0.0 163:03.18 portsentry
 5356 nobody    15   0 26236 8708 2948 S  0.3  0.9   0:38.61 httpd
27238 bobuser   16   0  3956 1012  728 R  0.3  0.1   0:00.03 top
    1 root      16   0  2904  324  300 S  0.0  0.0   0:16.27 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:04.61 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:08.13 ksoftirqd/0
    4 root      RT   0     0    0    0 S  0.0  0.0   0:04.61 migration/1
    5 root      34  19     0    0    0 S  0.0  0.0   0:04.46 ksoftirqd/1
    6 root       5 -10     0    0    0 S  0.0  0.0   5:42.74 events/0
    7 root       5 -10     0    0    0 S  0.0  0.0   9:07.74 events/1
    8 root      10 -10     0    0    0 S  0.0  0.0   0:00.01 khelper
   21 root      15 -10     0    0    0 S  0.0  0.0   0:00.00 kacpid
   88 root       5 -10     0    0    0 S  0.0  0.0   0:07.22 kblockd/0
   89 root       5 -10     0    0    0 S  0.0  0.0   1:48.00 kblockd/1
   97 root      15   0     0    0    0 S  0.0  0.0   0:00.00 khubd
  158 root      14 -10     0    0    0 S  0.0  0.0   0:00.00 aio/0
  157 root      16   0     0    0    0 S  0.0  0.0  77:44.46 kswapd0
Free:

Code: 
             total       used       free     shared    buffers     cached
Mem:           978        808        170          0         41        176
-/+ buffers/cache:        590        388
Swap:         4996        469       4527
Do you guys think the issue is that I need more RAM (due to the swap usage), or is RAM in fact not the problem, but some hardware issue perhaps?

THanks,

 

 

 

 

Top