Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

20 points by pera 81 days ago 21 comments
Summary not yet generated for this article.

Original Text

At the beginning of the year I decided to set up a scraping and LLM honeypot on one of my personal websites which included a fake git repo with code containing fake HTTP endpoints. The address to this repo was hidden in a public page inside a comment.

About three weeks ago IP addresses from Amazon Searchbot attempted to make requests to the fake endpoints included inside a shell script.

My robots.txt explicitly includes Amazonbot.

I am honestly surprised that this is coming from Amazon. Is this kind of behavior legal?

Comments (21 of ) ⌛

Loading more comments... ⌛
Scroll down to load more comments
No comments available.
Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open