Last Ditch Effort... Potential Ownage?
What starts this voyage is the fact that I have some servers at another DC, that makes available in their control panel stats based on information gathered via SNMP (port 161)... such as running processes, logged in users... etc. I've been noticing that sometimes it works, sometimes it doesn't. It wasn't until today that I had a problem accessing this SNMP information on all three servers at the same time.
Right about this time I got a message from the DC admin saying that my server was sending (or as they like to put it 'spewing') forged/spoofed packets outbound. Luckily their routers filter this, as this is a classic case of DoS attacks (my server(s) being a potential zombie attacker). Similarly a few days ago, the DC mentioned something similar about spewage, and outbound traffic reaching 10mbit.
At any rate, back to SNMP... in doing a netstat I was able to find that snmpd was now listening on port 199 as opposed to 161 (default port)... Which would explain why I had problems pulling up the statistical information via the control panel, but would not explain why it was intermittently pulling it up (sometimes works, sometimes doesn't). ... Which brought my DC admin to think that possibly some kernel-level trojan/process was monitoring legit requests for port 161 and redirecting them to 199 as needed (but not efficiently enough to run undetected by my very brilliant DC admin).
That being said, I recently (after the first spewage incident) found there was an exploit for 'ls' which I immediately updated on all 3 servers. Though I highly doubt this... there may be a chance that the upgrade from RedHat modified the snmp port (though the details of the RPM say those files haven't changed since Feb 11 2003. My MD5 Checksums on the RPM installed packages for snmp verify properly, so I don't think that is the problem.
I may be way off here, but port 199 is for an SNMP multiplexor... possibly used for a DDoS... which would makes sense seeing as 3 of my servers on the same subnet have now had their default SNMP port changed...
The init script for snmp is in tact and verified against a non-affected machine, so that's out of the question.
I'm really at my wits-end with this problem, and dreadfully need to find if in deed the last resort is the only resort (format/restore), or if someone has some glimpse into a potential answer that may save the day.
Heres more information than you ever wanted ::::::::::::::::::::::::::::
Server Specs:
SERVER1
Dell P4 2.6 1GB Ram
RedHat 9
WHM 8.5.4 / cPanel 8.5.4-R7 / WHM X v2.1.1
Kernel: 2.4.20-20.9
OpenSSH_3.5p1, SSH protocols 1.5/2.0, OpenSSL 0x0090701f
Apache Core 1.3.28
Exim 4.24
PHP 4.3.2 4.3.3
FrontPage 5.0.2.2634
mod_ssl 2.8.15
OpenSSL 0.9.7a
PHP Suexec 0.1b
SERVER2
Dell P4 2.6 1GB Ram
RedHat 9
WHM 8.5.4 / cPanel 8.5.4-R7 / WHM X v2.1.1
Kernel: 2.4.20-8
OpenSSH_3.5p1, SSH protocols 1.5/2.0, OpenSSL 0x0090701f
Apache Core 1.3.28
Exim 4.24
PHP 4.3.2 4.3.3
FrontPage 5.0.2.2634
mod_ssl 2.8.15
OpenSSL 0.9.7a
SERVER3
Dell P4 2.6 1GB Ram
RedHat 9
DirectAdmin v1.195
Kernel: 2.4.20-8
OpenSSH_3.5p1, SSH protocols 1.5/2.0, OpenSSL 0x0090701f
Apache 1.3.28
Exim 4.24
MySQL 4.0.14
Named 9.2.1
ProFTPd 1.2.8
sshd Running
vm-Pop3d 1.1.6
OpenSSL 0.9.7a
All with similar netstat output:
# netstat -natp|grep snmp
tcp 0 0 0.0.0.0:199 0.0.0.0:* LISTEN 2189/snmpd
tcp 0 0 127.0.0.1:199 127.0.0.1:32773 ESTABLISHED 2189/snmpd
tcp 0 0 127.0.0.1:32773 127.0.0.1:199 ESTABLISHED 2165/dcsnmp32d
(note the dcsnmp32d is the SNMP app by the DC to collect info for the control panel)
Thank you for your time, I know this was a long post. I appreciate any and all replies to this thread.
Take care,
.:. Mindlash
