Google’s robots.txt parser is now open source

777 points by dankohn1

Read the article

Discussion on Hacker News