What one may find in robots.txt

393 points by cellover

Read the article

Discussion on Hacker News