MSN Bot: Aggresive, CPU hungry... lock it out or not?
It comes around indexing several times a day. Indexes whole sites over and over, no matter whether content on some page has changed or not. Comes in from various IPs (sometimes Germany, sometimes USA, sometimes other), eats a lot of bandwidth due to indexing whole sites over and over (obviously no change in bytes algorythm is implemented).
Eats bandwidth (for some sites it eats almost 600-700 MB per month), but takes pretty much server memory and CPU while it is on. Alexa also eats pretty much data, but you don't feel it on server too much.
Pretty annoying, and obviously very arrogantly and not very well written.
Doesn't really care about 'revisit after' header tags, just keeps pestering several times a day. Some larger site owners started complaining.
Would it be wise to lock it out and how? I have tried locking msnbot.msn.com in hosts.deny, no success. Can't lock per IP, it keeps coming back from different IPs and IP ranges.
Google bot is nice... doesn't interfere much with resources and indexes every day really nicely. MSN bot? Really aggressive and annoying.
Turns out it caches CSS, but not HTML code. So sites that mostly use CSS for layouts and provide their own caching mechanisms (some CMS systems do) suffer much less from it than dynamic sites that don't cache their own content and use HTML formatting.
