Robots.txt meant for search engines don’t work well for web archives295 points by r721Read the articleDiscussion on Hacker News