Nicholas Carlini on Breaking AI Defenses
Nicholas Carlini of Google DeepMind joins The Cognitive Revolution to discuss why AI systems remain persistently vulnerable to adversarial attacks despite years of defense research. He frames most attacks as gradient-based optimization problems, emphasizing that the simplest loss functions usually suffice and that defenders face a fundamental asymmetry—attackers go second, only need to succeed once, and benefit from high-dimensional geometry where nearly everything is close to some decision boundary. Carlini explains how he pattern-matches new defenses against a mental rolodex of past failures, why "gradient masking" defenses (including recent unfinetunable/tamper-resistant models) collapse under techniques like random restarts and learning rate tweaks, and why even the best image adversarial training only achieves ~70% accuracy under attack—far below the reliability humans tolerate in other security domains. The conversation extends to memorization and privacy (models can regurgitate training data seen only hundreds of times), the gap between human and model robustness (possibly due to contextual reasoning rather than raw perception), and why social-engineering jailbreaks remain empirically stronger than gradient-based attacks—a fact Carlini finds philosophically puzzling. Drawing a cryptography analogy, he suggests we may need to accept empirically-validated robustness through iterative red-teaming rather than principled proofs, and favors defense-in-depth approaches (layers of detectors, system-level constraints) over hoping for any single robust solution. The episode also explores implications for open-source AI. Carlini expresses ambivalence: open models have been critical for safety research and he fears concentration of power, yet his own work strongly suggests we cannot reliably prevent capable open models from being fine-tuned to remove safety guardrails. He sees data unlearning and targeted fact-editing as currently fragile, and argues that achieving robust safety may require either breakthroughs in model robustness or accepting that deployment safety must come from system-level design—external constraints, human review, and sandboxed environments—rather than from the models themselves.
Original sources
Podcast The Cognitive Revolution
Episode:The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind ↗
YouTube The Cognitive Revolution
Video:The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind ↗
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.