Are Chatbots misinforming us about the European Elections? Yes. | Democracy Reporting International

🌈 Abstract

The article discusses the potential impact of AI chatbots on the electoral process, particularly in the context of the upcoming 2024 European Parliament elections. It examines the accuracy and political neutrality of responses from four popular chatbots (ChatGPT 3.5, ChatGPT 4.0, Microsoft's Copilot, and Google's Gemini) to questions related to the electoral process and political issues.

🙋 Q&A

[01] Randomness and Inconsistency of Chatbot Responses

1. What did the article find about the randomness and inconsistency of chatbot responses?

• The quality of responses to questions about the European Parliament elections varied greatly, even within the responses of each chatbot, supporting the idea that the workings of Large Language Models (LLMs) are hard to predict and fine-tune.
• No one chatbot was consistently correct across all questions and languages. Hallucinations and incomplete answers were observed in responses across all four bots.
• Randomness was also an issue within each chatbot, with some chatbots providing different responses to the same question in different languages.

2. What are the implications of this randomness and inconsistency?

• The randomness and inconsistency of chatbot responses raises concerns about their reliability and trustworthiness as a source of information, especially on important topics like electoral processes.
• It suggests that the companies behind these chatbots have not performed rigorous risk assessments and mitigation measures, as required under the EU's Digital Services Act (DSA).

[02] Chatbot Performance on Electoral Process Questions

1. How did the chatbots perform on questions related to the electoral process?

• The chatbots performed poorly on questions about the electoral process, such as voter registration, voting, and election results.
• They regularly "hallucinated" or made up information, with the most glaring examples including wrong election dates.
• When questions could be interpreted in several ways, the chatbots often assumed just one meaning (sometimes a clearly wrong one), or mixed up separate issues in their answers.

2. What are the potential consequences of chatbots providing inaccurate information about the electoral process?

• Inaccurate information about the electoral process can deter voters, cause them to miss deadlines, or make other mistakes, ultimately impacting the right to vote and electoral outcomes.
• This unintentional misinformation goes against the legal obligations under the EU's Digital Services Act (DSA) and the commitments made by companies under the EU's Code of Practice against Disinformation.

[03] Chatbot Performance on Political Questions

1. How did the chatbots perform on political questions?

• The chatbots largely managed to stay non-partisan on political questions, providing a wide variety of responses, such as refusals to respond, generic advice on how to form a political opinion, or overviews of party positions.
• In rare cases, they provided soft recommendations to vote for a particular party group on a specific issue.

2. What are the potential issues with how the chatbots handled political questions?

• While the chatbots generally remained non-partisan, their framings of party positions could be suspect, as they may not accurately reflect the nuances of each party's platform.
• The chatbots should more systematically refer users to voting advice applications when they are looking for specific voting recommendations.

[04] Chatbot Sources and Links

1. What issues did the article find with the sources and links provided by the chatbots?

• The chatbots often provided broken, irrelevant, or incorrect links as sources of information, weakening even strong and informative answers.
• They tended to use information provided by EU sources, whether it was relevant or not, and often failed to prioritize the most authoritative sources, such as national electoral authorities.

2. Why is the quality of sources and links provided by chatbots important?

• Providing reliable and authoritative sources is crucial for ensuring that voters have access to accurate information about the electoral process and can make informed decisions.
• The poor quality of sources and links undermines the trustworthiness of the chatbots' responses and their ability to serve as a reliable source of information for voters.

[05] Recommendations

1. What are the key recommendations made in the article?

• The creator companies should immediately review how their chatbots provide electoral process content, not only for the European Parliament elections, but for any election in Europe and globally.
• Chatbots should be tuned to only provide links to the most authoritative sources of information (the electoral authorities) without generating any information themselves.
• The integration of chatbots into search engines is considered premature and irresponsible, given the propensity of LLMs to "hallucinate" information in areas with limited authoritative sources.

2. Why are these recommendations important?

• These recommendations aim to address the significant issues identified in the article, where the chatbots consistently provided inaccurate, incomplete, or made-up information about electoral processes, which could have serious consequences for voter participation and electoral integrity.
• Ensuring that chatbots only refer users to authoritative sources, rather than generating their own information, is crucial for maintaining the trustworthiness and reliability of these tools, especially in the context of important electoral processes.