Adversarial Machine Learning and the Evolving Landscape of Computer Security

The conversation with Dawn Song, a computer science professor at UC Berkeley, delves into the pervasive and evolving nature of security vulnerabilities, arguing that completely bug-free code is an elusive ideal due to the broad and ever-changing definition of "attacks." While formal verification techniques offer provable guarantees for specific security properties in systems like microkernels and compilers, they cannot cover all attack vectors. A central argument is that the attack surface is shifting, moving "up the stack" from traditional system-level exploits (like memory safety or buffer overflows) towards human vulnerabilities, making social engineering, phishing, and fake news increasingly critical threats.

Song highlights the distinction between static and dynamic program verification, emphasizing that most robust verification is done statically, without running the code, to provide broader guarantees. A significant nuance is the "weakest link" concept, identifying humans as the most challenging component to secure, as they cannot be "patched" like software or hardware. The discussion then pivots to adversarial machine learning, distinguishing between attacks at the inference stage (subtle, often imperceptible perturbations to inputs causing misclassification) and the training stage (data poisoning, including stealthy "backdoor attacks" where a model appears normal but behaves maliciously under specific, attacker-controlled triggers).

To counter human vulnerabilities, Song proposes using AI, specifically NLP and chatbot techniques, as a defense mechanism. These "security bots" could observe conversations, detect patterns indicative of social engineering (e.g., phishing attempts), and engage in "challenge and response" protocols to verify identities or intentions. While still developing, this approach offers a vision for AI assisting humans in navigating complex digital interactions. For machine learning systems, the research demonstrates the critical need for robustness against both digital and physical adversarial examples, as shown by the creation of physically altered stop signs that fool autonomous driving systems.

The broader implications of this research are profound, touching upon the future of cyber warfare, where human manipulation and AI system compromise could be primary battlegrounds. The ability to create physical adversarial objects, like modified traffic signs, underscores the real-world safety risks for critical AI applications such as autonomous vehicles. The conversation also implicitly raises questions about trust in AI, the trade-offs between security and privacy (e.g., a personal security bot's knowledge), and the ongoing challenge of defining and achieving "security" in an increasingly complex and interconnected world where the line between digital and physical threats blurs.

systems will always have security vulnerabilities

it's very difficult to write completely bug-free code and code that has no one in policy

security is job security

the attacks are actually moving more anymore from the systems south yeah towards to humans

in security we say the weakest link actually have the system's oftentimes is actually humans themselves

we don't have a way to say to a software upgrade out to a hardware for humans

the goal is to fold the machining system me into making the wrong decision

you add really small perturbations changes to the image it can be so subtle that to human eyes it's hard to it's even imperceptible

by manipulating the by giving particular type of poisons training data to the to the Machine immune system actually not only that's in this case we can have your impersonates as tranfer whatever it's nice to be the president yeah actually we can make it in such a way that's for example if you wear a certain type of glasses then we can make it in in such a way that anyone not just you anyone that wears that couple classes will be will be recognized as trump

it's possible to have a physical address zero attacks in the physical world that are robust to this kind of viewing distance do angle and so on