Ensuring the safety and security of AI models is paramount as their adoption accelerates. At Virtue AI, our advanced redteaming platform VirtueRed employs over 100 specialized algorithms to rigorously test AI models across multiple safety and security domains. This analysis compares Meta’s recently released Llama 4 Scout and Maverick models against OpenAI’s GPT-4.5.

Overview

• Llama 4 Scout presents substantial risks in regulatory compliance, bias mitigation, privacy protection, and multi-modal security.
• Llama 4 Maverick shows some safety improvements compared to Scout, yet its overall security and safety performance remains inadequate when benchmarked against industry-leading models like GPT-4.5

VirtueRed: Automated Red-Teaming for AI Models & Applications

VirtueRed systematically performs penetration testing and evaluates AI safety & security of AI models and applications with a series of advanced and adaptive red teaming algorithms, assessing:

• Practical use-case driven risks (e.g., hallucination, privacy, over-cautiousness, bias)
• Regulatory compliance risks (e.g., EU AI Act, AI company policies)
• Adaptive multi-modal vulnerabilities (e.g., security risks given adaptive multimodal jailbreaks)
• CodeGen-related risks (e.g., malicious and risky code generation)

Through diverse advanced red teaming algorithms, VirtueRed identifies systemic weaknesses and real-world safety challenges for foundation models and applications, enabling organizations to strengthen safe and secure AI products deployment, focus on product development, and minimize the time to market.

Key Findings from VirtueRed’s Red-Teaming Evaluation

Regulatory Compliance & AI Policy Risks

• ⚠️ Llama 4 Maverick maintains moderate compliance, excelling in minimizing weapon-related and misrepresentation risks.
• ❌ Llama 4 Scout has significant compliance risks, including automated decision-making and perpetuation of harmful beliefs, posing high regulatory challenges under frameworks like the EU AI Act.
• ✅ GPT-4.5 offers stronger regulatory compliance compared to both Llama 4 models

Privacy & Security Vulnerabilities

• ⚠️ Llama 4 Scout and Maverick both show moderate privacy risks, vulnerable to sensitive data extraction and PII leakage, though Maverick is more resilient comparatively.
• ✅ GPT-4.5 demonstrates notably lower risks of privacy violations, providing robust defenses against PII leakage and extraction attacks.

Code Generation Attacks

• ❌ Llama 4 Scout and Maverick both exhibit high risks in code generation, prone to generating exploitable code and inadequately responding to adversarial prompts.
• ✅ GPT-4.5 exhibits low risk for malware generation and cybersecurity threats

Multi-Modal Safety Risks

• ⚠️ Llama 4 Scout and Maverick display moderate susceptibility to visual jailbreak attacks
• ✅ GPT-4.5 provides superior resilience against multi-modal adversarial attacks, including harmful image jailbreaks

Bias & Fairness

• ⚠️ Llama 4 Scout and Maverick both carry moderate bias concerns, especially regarding stereotypes related to terrorism, crime, and other protected characteristics
• ✅ GPT-4.5 maintains low bias across sensitive categories, significantly outperforming Llama models

Hallucination

• ✅ GPT-4.5 leads with substantial hallucination mitigation, consistently providing accurate, reliable outputs.
• ✅ Llama 4 Maverick is strong as well, although not as substantial as GPT-4.5
• ❌ Llama 4 Scout still exhibits moderate hallucination issues

Over-Cautiousness

• ✅ Llama 4 Scout offers the best user experience, balancing cautiousness with practical responsiveness.
• ✅ Llama 4 Maverick also performs well regarding cautiousness, although less effectively than Scout.
• ❌ GPT-4.5 is overly cautious, frequently rejecting benign queries, limiting usability.

Conclusion

Overall, the safety and security performance of Llama 4 Scout and Maverick models is concerning, especially in direct comparison to GPT-4.5. 

While Llama 4 Maverick demonstrates moderate strengths in user experience and some reduction in hallucinations, both Maverick and Scout face significant shortcomings across regulatory compliance, privacy, cybersecurity, and multi-modal vulnerabilities. 

The pronounced weaknesses in code generation security and bias further emphasize the substantial risks associated with deploying these models without rigorous protective measures. GPT-4.5 clearly sets a higher safety benchmark, excelling in critical safety domains despite usability challenges related to cautiousness. 

Organizations should exercise caution and implement comprehensive guardrails when considering deployment of the Llama 4 models. 

Discover how Virtue AI’s guardrails and redteaming services can strengthen your AI strategy by scheduling a demo here.