How to crawl a quarter billion webpages in 40 hours (2012)

296 points by allenleee

Read the article

Discussion on Hacker News