Non-Google search engines blocked from showing recent Reddit results

🌈 Abstract

The article discusses recent changes to Reddit's content policy that have resulted in reduced visibility of Reddit content on non-Google search engines. The key points are:

• Reddit has updated its robots.txt file to block crawling of its site without agreeing to Reddit's rules, which include not using Reddit content for AI training without explicit consent.
• This has led to a significant reduction in Reddit results appearing on search engines like Bing, DuckDuckGo, and Mojeek, while Google remains the primary search engine that can show recent Reddit content.
• Reddit's move is seen as tightening Google's stranglehold on the search industry, as Google has a deal to use Reddit data to train its AI models.
• Smaller search engines are concerned about the precedent this sets and the potential impact on their ability to compete.

🙋 Q&A

[01] Recent changes to Reddit's content policy

1. What are the key changes Reddit has made to its content policy?

• Reddit has updated its robots.txt file to block crawling of its site without agreeing to Reddit's rules, which include not using Reddit content for AI training without explicit consent.
• Reddit has approved scrapers from the Internet Archive and some research-focused entities, but has blocked other commercial entities that it believes are misusing its public content.

2. What is the impact of these changes on search engine results?

• Recent Reddit discussions are no longer showing up in non-Google search engine results, such as Bing, DuckDuckGo, and Mojeek.
• Google remains the primary search engine that can show recent Reddit content, as it has a deal to use Reddit data to train its AI models.

3. How have smaller search engines responded to these changes?

• Smaller search engines like Mojeek are concerned about the precedent this sets and the potential impact on their ability to compete, as search engines are the main traffic source for most websites.
• Mojeek's CEO, Colin Hayhurst, reached out to Reddit but did not receive a response, until after the 404 Media report.

[02] Implications for the search industry

1. How has Reddit's move impacted Google's position in the search industry?

• Reddit's changes have inadvertently helped tighten Google's stranglehold on the search industry, as Google is virtually the only search engine that can show recent Reddit results.
• This comes amid recent quality concerns about Google's search results, which have ranked SEO and AI spam farms, ads, and e-commerce links higher than more relevant results.

2. What are the concerns about the precedent set by Reddit's actions?

• There are concerns that the spreading of this behavior, where websites block non-compliant search engines, will further choke off traffic to smaller sites and make it harder for alternative search engines to compete.
• Mojeek's CEO, Colin Hayhurst, believes that if this behavior becomes more widespread, it will impact smaller sites even more than larger ones.

3. What is Reddit's stance on the use of its content by search engines?

• Reddit believes in an open internet, but not the misuse of public content, and has approved scrapers from the Internet Archive and some research-focused entities.
• Reddit has been in talks with multiple search engines, but has been unable to reach agreements with all of them, as some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including for AI training.