What one may find in robots.txt
393 points by cellover
Read the article
Discussion on Hacker News