Sites that block AI training crawlers mostly ignore the answer time bots

We read the robots.txt of the top 10,000 sites. 38% of the GPTBot rules we could date were written in a single quarter of 2023 — right after GPTBot launched and the Times sued. Then every AI company quietly added a second bot, the one that answers questions in real time, and almost nobody wrote a rule for it.
13 points by zeppelin_7 106 days ago sitedex.dev 0 comments
Summary not yet generated for this article.

Comments (0 of ) ⌛

Loading more comments... ⌛
Scroll down to load more comments
No comments available.
Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open