We have a big issue with people using bots to grab our dynamic information and when one of our customers releases new info the site gets flooded with these scrapers downloading all the new content. We've tried everything from banning IPs but this bites us because they use big IP pools and then we ban real users. We tried setting cookies on the clients computer to make sure they're legit but they started using cookies in their scripts. Now, they are using proxies so the IP is not the same so it makes it very hard to even track if it's the same person but you can tell because the searches are in secuence and very fast. Any ideas on how to get around this?
