Confused about outgoing packets being dropped from APF

Over the past few days I've noticed a steadily increasing spike of dropped outbound packets in APF, originating from my server.

Today in my logwatch email it states the following:

"Dropped 594 packets on interface eth0 From xxx.xxx.x.x - 593 packets"

The normal amount of dropped packets has always been around 20 packets or so which has been for over 8 months, but I've been noticing a increase in outbound packets being dropped.

Looking at logwatch, i can see that the outgoing packets that are being dropped are being sent to multiple ip's and multiple ports. It ranges from 1 to 30 packets to each IP.. with each packet being sent to a different port.

This worried me as to me it looked as if my server was possibly trying to do port scans, or may have some type of worm or trojan running. I've been trying for days to try and find the source of this issue. So i ran chrootkit and rookithuner, as well as clamav scans

I've asked both my DC and my server management company if they can help me solve this problem and maybe investigate it further.

My DC said that I shouldn't be worried because that is a small amount of packets being dropped, and it's most likely due to the increasing amount of traffic comming from my server. To investigate the issue further, I would be charged. So i asked if they can atleast give me an idea or something on what might be the cause. Here is what they said "

I would run ifconfig and compare the results ( how many packets dropped) to what apf is saying, then restart apf.

You might also check tcpdump using the command

tcpdump -ettti eth0 -sn 1600 -w outputfile
to output the results of tcp dump to a file.

to read the results from the file

tcpdump -lettti -rn outputfile

look for the word malformed it will show you if your server is generating malformed packets which are failing to send properly.

594 packets is really not all that abnormal considering the millions of packets being sent out, the packets are not actually lost they are merely retransmitted slowing the download a bit but not much especially with so few of the packets being dropped.
Now this was confusing me because both my DC and my management company was saying this is normal, and not to worry about it. To me it doesn't look like normal behavior for the server to just be sending out packets to random IP's and ports(That are closed by the firewall).

Well I had the server management company look into it further, but as I said..they said it was normal too. However they made some adjustments to APF to see if it would've solved the issue, but the problem didn't go away.

Then they tried what my DC said above, and replied back to me with this:
tcpdump -ettti eth0 -sn 1600 -w outputfile

tcpdump -lettti -rn outputfile

Those two commands don't work.

500 packets really isn't anything to worry about.

I'm running tcpdump -ettti eth0 -s 1600 -w outputfile (the output is actually a jumble of stuff and there's no way to read it)

I'm not too experienced with tcpdump, so there may be something I'm missing.

ifconfig shows 0 dropped packets which means it's not a problem with malformed or NIC problems.
I'm running a RHE/Cpanel/Fantastico server.

My questions are basically:

1. Is this abnormal behavior for a webserver, or am i just being paranoid?
2. Is there anyway to find out whats causing the packets to be sent out, and track the origin?
3. What would you do in this situation?

Thank you in advance for all replies.

 

 

 

 

Top