OpenAI Takes Action Against Misuse of Its Models in Propaganda

🌈 Abstract

The article discusses how OpenAI discovered that several organizations from Russia, China, Iran, and Israel had used the company's language models to create and/or revise text in attempts to influence international political opinion through disinformation campaigns. The generated content failed to reach a mass audience, and OpenAI banned the accounts involved.

🙋 Q&A

[01] OpenAI's Discovery of Disinformation Campaigns

1. What did OpenAI discover about the use of its models in disinformation campaigns?

• OpenAI discovered that operations based in Russia, China, Iran, and Israel had used the company's models to create and/or revise text in attempts to influence international political opinion.
• The generated media failed to reach a mass audience, and OpenAI banned the accounts involved.

2. How did the different groups use OpenAI's models for disinformation?

• A Russian organization generated large volumes of pro-Russia and anti-Ukraine comments in Russian and English, often with poor grammar or telltale phrases.
• Another Russian group called Doppelganger generated pro-Russia social media comments in English, French, and German, and used OpenAI models to translate articles from Russian into other languages.
• A Chinese operation known as Spamouflage generated Chinese-language social media comments supporting the Chinese government and used OpenAI technology to debug code for a website criticizing opponents of the government.
• An Iranian organization called the International Union of Virtual Media (IUVM) generated English and French articles, headlines, and other text for its website, which is considered a mouthpiece for the Iranian government.
• An Israeli company called STOIC generated articles, social media comments, and fictitious bios for inauthentic social media accounts, including both pro-Israel and anti-Palestine content, as well as comments critical of India's ruling party.

3. What was the impact of these disinformation campaigns?

• The generated content failed to reach a mass audience, and OpenAI was able to detect and shut down the accounts involved.

[02] The Rise of AI-Produced Misinformation

1. What was the trend in AI-produced misinformation on the internet?

• AI-produced misinformation on the internet, mostly in the form of images, videos, and audio clips, rose sharply starting in the first half of 2023.
• By the end of that year, generative AI was responsible for more than 30% of media that was manipulated by computers.

2. Why is the potential proliferation of political disinformation using AI models a concern?

• Many observers are concerned about the potential for AI models that generate realistic text, images, video, and audio to be used for political disinformation, especially with elections in at least 64 countries, including most of the world's most populous nations, scheduled for this year.

3. How does the article view the current impact of AI-generated disinformation?

• While propagandists have taken advantage of OpenAI's models, the article notes that the accounts identified by OpenAI failed to reach significant numbers of viewers or have an impact, and that distribution, not generation, continues to be the limiting factor on disinformation so far.