getting hammered
For some reason, today, i noticed my site was loading extremely slow
so i logged in the server, did ps aux and saw like a million httpd processes
so i did
lsof -ni | grep "EST" | grep http
and what did i see?
httpd 23047 root 3u IPv4 15446293 TCP 216.18.0.178:http->24.209.75.239:2779 (ESTABLISHED)
httpd 23048 root 3u IPv4 15446394 TCP 216.18.0.178:http->24.209.75.239:3000 (ESTABLISHED)
httpd 23049 root 3u IPv4 15446396 TCP 216.18.0.178:http->24.209.75.239:3001 (ESTABLISHED)
httpd 23050 root 3u IPv4 15446403 TCP 216.18.0.178:http->24.209.75.239:3013 (ESTABLISHED)
httpd 23051 root 3u IPv4 15446404 TCP 216.18.0.178:http->24.209.75.239:3015 (ESTABLISHED)
httpd 23052 root 3u IPv4 15478401 TCP 216.18.0.178:http->24.209.75.239:3016 (ESTABLISHED)
httpd 23053 root 3u IPv4 15478402 TCP 216.18.0.178:http->24.209.75.239:3018 (ESTABLISHED)
httpd 23054 root 3u IPv4 15478404 TCP 216.18.0.178:http->24.209.75.239:3020 (ESTABLISHED)
httpd 23055 root 3u IPv4 15478403 TCP 216.18.0.178:http->24.209.75.239:3019 (ESTABLISHED)
httpd 23056 root 3u IPv4 15478405 TCP 216.18.0.178:http->24.209.75.239:3021 (ESTABLISHED)
httpd 23057 root 3u IPv4 15478406 TCP 216.18.0.178:http->24.209.75.239:3022 (ESTABLISHED)
httpd 23058 root 3u IPv4 15478407 TCP 216.18.0.178:http->24.209.75.239:3023 (ESTABLISHED)
httpd 23059 root 3u IPv4 15478408 TCP 216.18.0.178:http->24.209.75.239:3025 (ESTABLISHED)
httpd 23060 root 3u IPv4 15478409 TCP 216.18.0.178:http->24.209.75.239:3024 (ESTABLISHED)
httpd 23061 root 3u IPv4 15478410 TCP 216.18.0.178:http->24.209.75.239:3026 (ESTABLISHED)
httpd 23062 root 3u IPv4 15478411 TCP 216.18.0.178:http->24.209.75.239:3028 (ESTABLISHED)
httpd 23063 root 3u IPv4 15478412 TCP 216.18.0.178:http->24.209.75.239:3032 (ESTABLISHED)
httpd 23064 root 3u IPv4 15478413 TCP 216.18.0.178:http->24.209.75.239:3033 (ESTABLISHED)
httpd 23065 root 3u IPv4 15478414 TCP 216.18.0.178:http->24.209.75.239:3034 (ESTABLISHED)
httpd 23066 root 3u IPv4 15478415 TCP 216.18.0.178:http->24.209.75.239:3035 (ESTABLISHED)
httpd 23067 root 3u IPv4 15478417 TCP 216.18.0.178:http->24.209.75.239:3039 (ESTABLISHED)
httpd 23068 root 3u IPv4 15478418 TCP 216.18.0.178:http->24.209.75.239:3206 (ESTABLISHED)
httpd 23069 root 3u IPv4 15478419 TCP 216.18.0.178:http->24.209.75.239:3207 (ESTABLISHED)
httpd 23070 root 3u IPv4 15478420 TCP 216.18.0.178:http->24.209.75.239:3208 (ESTABLISHED)
httpd 23071 root 3u IPv4 15478421 TCP 216.18.0.178:http->24.209.75.239:3209 (ESTABLISHED)
httpd 23072 root 3u IPv4 15478422 TCP 216.18.0.178:http->24.209.75.239:3211 (ESTABLISHED)
httpd 23073 root 3u IPv4 15478423 TCP 216.18.0.178:http->24.209.75.239:3210 (ESTABLISHED)
httpd 23074 root 3u IPv4 15478424 TCP 216.18.0.178:http->24.209.75.239:3215 (ESTABLISHED)
httpd 23075 root 3u IPv4 15478425 TCP 216.18.0.178:http->24.209.75.239:3212 (ESTABLISHED)
httpd 23076 root 3u IPv4 15478426 TCP 216.18.0.178:http->24.209.75.239:3213 (ESTABLISHED)
httpd 23077 root 3u IPv4 15478427 TCP 216.18.0.178:http->24.209.75.239:3214 (ESTABLISHED)
httpd 23078 root 3u IPv4 15478428 TCP 216.18.0.178:http->24.209.75.239:3216 (ESTABLISHED)
httpd 23079 root 3u IPv4 15478429 TCP 216.18.0.178:http->24.209.75.239:3217 (ESTABLISHED)
httpd 23080 root 3u IPv4 15478430 TCP 216.18.0.178:http->24.209.75.239:3219 (ESTABLISHED)
httpd 23081 root 3u IPv4 15478431 TCP 216.18.0.178:http->24.209.75.239:3218 (ESTABLISHED)
httpd 23082 root 3u IPv4 15478432 TCP 216.18.0.178:http->24.209.75.239:3220 (ESTABLISHED)
httpd 23083 root 3u IPv4 15478438 TCP 216.18.0.178:http->24.209.75.239:3221 (ESTABLISHED)
see how 24.209.75.239 is making so many connections to 216.18.0.178 (the ip of a site i run/host)
well, about 10000 other ips were also grabbing connections, and holding on to it
funny thing is, none were valid
i checked webserver logs.. all these ips that have estbalished connections to that ip, never ever visited any of the sites
so this brings me to believe that my site is being dos'ed, sort of.
basically tons and tons of "zombies" or just non valid ips are trying to make connection attempts to my webserver, to run up the processes, and make it so my websitse dont load
why this is happening, i don't know.
But does anyone have any solutions?
I mean, raising max clients wont do anything because that will just fit more fake / non valid ips/hosts.
raising it down probably wont make anything better either
any idea on what i should do
i saw a lot of foreign ips, like 211.*.*.* and 202.*.** and when id dns them a lot would be like mail.something.com
so obviously, it seems as though these are "compromised" hosts, or just not valid ones.
So for the most part, ive done a lot of ipchaining, with things like
ipchains -A input -s 211.0.0.0/8 -d 0/0 80 -p tcp -j DENY
yes, i realize what it does. i don't really want any business from 211.*.*.* anyway.
anyhow, the only way i can see to fix this, is to have a program, that runs lsof -ni | grep "EST" | grep http every 30 seconds or so
and if it sees an IP that has several (Talking 15+ ) established connections, then it will ipchain it
knock it down
becuase in raelity. when you visit a site. you only make two - three - four connection attempts / establishments.. then when the page finishes loading... the connection is dropped
So does anyone know what I could do ?
