You are viewing a single comment's thread.

view the rest of the comments →

MadWorld ago

You could use httrack to mirror a website, preferably on the top of vpn with lower flow-control setting. It should not give you too many "Too Many Requests" stat.

JDN11DBAN ago

You can get around the "Too many requests" by setting your HTTP request user agent to look like the google crawler. - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - Or the mobile version of with this - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html). I have had this work in the past. Good Luck!

MadWorld ago

Thanks for the tip!