I have mixed feelings about this. http://www.cyveillance.com/ periodically scans my site. Nothing major there - it's just another spider. But I really have a problem with what these people do for a living. By scanning my site they are basically saying "we don't trust you, have you stolen anything?" Somewhere on their site I thought they had a nice "add this to your robots.txt file and we won't scan you" but that seem to have been taken off and they now spider with a generic Windows user agent string.
I know, the public parts of my website are just that - public for the entire planet see. But this just bugs me. You're allowed to tell me that I should be ok trusting *something* from near the Washington D.C. area but there is just too damn much DOD stuff there. Placing my tin foil cap on now...
