Vulnerabilities for AI and ML Applications are Skyrocketing

🌈 Abstract

The article discusses the increasing number of vulnerabilities in AI and machine learning (ML) applications, highlighting the growing threat of AI-related zero-day vulnerabilities. It examines the surge in vulnerabilities uncovered in popular open-source software (OSS) projects used in AI/ML development, with a particular focus on remote code execution (RCE) vulnerabilities that can allow attackers to gain full control of compromised systems.

🙋 Q&A

[01] Vulnerabilities for AI and ML Applications are Skyrocketing

1. What are the key findings from the Protect AI's huntr community report?

• The number of AI-related zero-day vulnerabilities has tripled since November 2023.
• In April 2024 alone, 48 vulnerabilities have been uncovered in widely used OSS projects like MLFlow, Ray, and Triton Inference Server, representing a 220% increase from the 15 vulnerabilities reported in November.
• A prevalent threat is Remote Code Execution (RCE), which allows attackers to run commands or programs on a victim's computer or server without physical access, potentially leading to unauthorized access, data breaches, system compromise, and total system takeover.

2. What do the statistics from Protect AI's report suggest? The statistics underscore the accelerating scale and velocity of the AI/ML zero-day problem, suggesting a growing need for enhanced security measures in AI development environments.

[02] Old Vulnerabilities, New Practices

1. What are the notable vulnerabilities highlighted in the report? The report highlights the PyTorch Serve RCE and the BentoML RCE vulnerabilities, which allow attackers to gain RCE on the servers running these popular AI/ML inference server projects.

2. What is the report's biggest surprise? The report's biggest surprise is the quantity of basic web application vulnerabilities discovered in these AI/ML projects, which are rarely seen in modern web applications due to the prevalence of secure coding practices and web frameworks with built-in security guardrails.

3. What does the resurgence of these vulnerabilities indicate? The resurgence of these vulnerabilities indicates that security is an afterthought in AI/ML-related tooling, going against the lessons learned over the past decade about the importance of security in software development.

[03] LLM Tooling a Security Weakness

1. What are the concerns regarding the rapid adoption of LLM-based security projects?

• Organizations may feel compelled to adopt LLM-based security projects due to competitive pressures or the desire to stay ahead in the threat landscape, but this rapid adoption raises concerns about security maturity.
• In their haste to deploy LLM tools, organizations may overlook crucial aspects of security, such as comprehensive risk assessments, robust testing protocols, and adherence to industry best practices.
• This could result in the deployment of solutions that are not adequately fortified against emerging threats or lack the safeguards to protect sensitive data and assets.

2. What must organizations prioritize alongside innovation? Organizations must prioritize security maturity alongside innovation when adopting AI and LLM-based tools.

[04] Adopting Least Privilege, Zero Trust

1. What security practices does the article recommend for organizations to protect themselves against the expanding AI/ML threat landscape?

• Adopting the concept of least privilege
• Adopting security models including Zero Trust
• Training developers and AI engineers in secure coding practices and basic security principles
• Conducting internal security audits of new AI/ML tools and libraries before deployment

2. What is the author's prediction regarding the future of AI/ML security? The author cautions that companies will be breached more often as a consequence of rushing to adopt AI/ML tooling, as it is extremely hard to make accurate predictions about the rate of acceleration in this space over the next 12-24 months.

[05] AI as Weakness and Advantage

1. What are the dual implications of GenAI adoption discussed in the article?

• GenAI adoption by malicious actors is bringing new security risks to organizations.
• However, the implementation of AI-based cyber tools could also help organizations that are struggling to meet growing threats.