How to crawl a quarter billion webpages in 40 hours

323 points by cing

Read the article

Discussion on Hacker News