Robots.txt meant for search engines don’t work well for web archives

295 points by r721

Read the article

Discussion on Hacker News