Disrupting deceptive uses of AI by covert influence operations

🌈 Abstract

The article discusses how OpenAI has disrupted several covert influence operations (IO) that sought to use OpenAI's models for deceptive activities across the internet. It outlines the threat actors involved, the trends observed in their use of AI, and the defensive measures taken by OpenAI.

🙋 Q&A

[01] Disruption of Covert Influence Operations

1. What types of activities did the threat actors use OpenAI's models for?

• Generating short comments and longer articles in various languages
• Creating fake social media accounts with made-up names and bios
• Conducting open-source research
• Debugging simple code
• Translating and proofreading texts

2. What were the specific operations that OpenAI disrupted?

• A Russian operation called "Bad Grammar" that targeted Ukraine, Moldova, the Baltic States, and the US
• A Russian operation called "Doppelganger" that generated content in multiple European languages
• A Chinese network called "Spamouflage" that generated content in several languages
• An Iranian operation called the "International Union of Virtual Media (IUVM)" that generated and translated long-form articles
• An operation by an Israeli company called "STOIC" (nicknamed "Zero Zeno") that generated articles and comments

3. What was the impact of these disruptions?

• The operations did not see a significant increase in audience engagement or reach as a result of using OpenAI's services.
• Using the Brookings' Breakout Scale, none of the operations scored higher than a 2 (activity on multiple platforms, but no breakout into authentic communities).

[02] Attacker Trends

1. How did the threat actors use AI models like OpenAI's?

• To generate text (and occasionally images) in greater volumes and with fewer language errors than would have been possible for human operators alone.
• As one of many types of content, alongside more traditional formats like manually written texts or memes.
• To create the appearance of engagement across social media, such as by generating replies to their own posts.
• To enhance productivity, such as by summarizing social media posts or debugging code.

2. What limitations did the threat actors face in using AI models?

• OpenAI's safety systems imposed friction, with the models refusing to generate certain types of content requested by the actors.
• The threat actors were prone to human errors, such as publishing refusal messages from OpenAI's models on their own platforms.

[03] Defensive Trends

1. How did OpenAI leverage AI to enhance its investigations?

• OpenAI built its own AI-powered tools to make its detection and analysis more effective, allowing investigations to be completed in days rather than weeks or months.
• As OpenAI's models improve, the company will continue to leverage their capabilities to improve its investigations.

2. What other factors contributed to OpenAI's ability to disrupt these operations?

• The distribution of the AI-generated content across multiple platforms, without the threat actors being able to engage a substantial audience.
• Sharing detailed threat indicators with industry peers, which benefited from years of open-source analysis by the wider research community.
• The human limitations and errors of the threat actors, which affected their operations and decision-making.