OneFlip: A threat to AI that could make vehicles crash and facial recognition fail
SECURITY WEEK
A research team from George Mason University, led by associate professor Qiang Zeng, presented a paper (PDF) at this year’s August USENIX Security Symposium describing a process that can flip a single bit to alter a targeted weight. The effect could change a benign and beneficial outcome to a potentially dangerous and disastrous outcome.
Example effects could alter an AV’s interpretation of its environment (for example, recognizing a stop sign as a minimum speed sign), or a facial recognition system (for example, interpreting anyone wearing a specified type of glasses as the company CEO). And let’s not even imagine the harm that could be done through altering the outcome of a medical imaging system.
All this is possible. It is difficult, but achievable. Flipping a specific bit would be relatively easy with Rowhammer. (By selecting which rows to hammer, an attacker can flip specific bits in memory). Finding a suitable bit to flip among the multiple billions in use is complex, but can be done offline if the attacker has white-box access to the model.
The researchers have largely automated the process of locating suitable single bits that could be flipped to dramatically change individual weight value. Since this is just one weight among hundreds of millions it will not affect the performance of the model. The AI compromise will have built-in stealth, and the cause of any resultant ‘accident’ would probably never be discovered.
Read more | SECURITY WEEK