Reddit changes have blocked all search engines except Google amid AI 'misuse' [U]

🌈 Abstract

The article discusses how Reddit has started blocking its search results from appearing in other search engines, apart from Google, with which it has a $60 million deal for using Reddit data to train Google's AI models and prominently display Reddit results in Google Search.

🙋 Q&A

[01] Reddit's Blocking of Search Results

1. What is the reason behind Reddit blocking its search results from appearing in other search engines?

• Reddit has updated its robots.txt file to block all bots from scraping any part of the site, citing the "misuse of public content" by "obviously commercial entities who scrape Reddit" and use the content for various use cases, including AI training.
• While not explicitly stated, it is clear that the focus of this change in policy is on preventing the use of Reddit data for AI training, apart from the deal with Google.

2. Which search engines are affected by this change?

• Bing, DuckDuckGo, Mojeek, and Qwant are all affected, with their results either not showing anything recent or not showing the full site result.
• Kagi, a paid search engine, is still showing data, but only because it buys some of its search index from Google, which continues to have access to Reddit data through its deal with Reddit.

3. What is Reddit's stance on working with other search engines?

• Reddit is "open" to working with others around data crawling and is in discussions with "multiple" search engines, but has not reached deals with all of them due to promises around how Reddit content would be used, including in the training of AI.

[02] Google's Deal with Reddit

1. What is the nature of the deal between Google and Reddit?

• In February 2023, Google announced a new deal with Reddit, which would see Reddit data used for training Google's AI models as well as more prominently showing Reddit results within Google Search.
• The deal was reported to be worth around $60 million.

2. How has this deal impacted the visibility of Reddit results in Google Search?

• Since the deal, Reddit results have been showing up more prominently in Google Search, often outranking websites that Reddit posts are linking to.

</output_format>