Browse / Unsafe Model Loading and Supply Chain / AML.CS0009
Tay Poisoning
- Also involves
- Data and Model Poisoning, Model Evasion
- Target
- Microsoft's Tay AI Chatbot (Incident)
How it worked
- AI Model Access · AI-Enabled Product or Service
Adversaries were able to interact with Tay via Twitter messages.
- Initial Access · Supply Chain Compromise: Data Repository
Tay bot used the interactions with its Twitter users as training data to improve its conversations.
Adversaries were able to coordinate with the intent of defacing Tay bot by exploiting this feedback loop. - Persistence · Training Data Poisoning
By repeatedly interacting with Tay using racist and offensive language, they were able to skew Tay's dataset towards that language as well. This was done by adversaries using the "repeat after me" function, a command that forced Tay to repeat anything said to it.
- Impact · Erode AI Model Integrity
As a result of this coordinated attack, Tay's conversation algorithms began to learn to generate reprehensible material. Tay's internalization of this detestable language caused it to be unpromptedly repeated during interactions with innocent users.
Summary
Microsoft created Tay, a Twitter chatbot designed to engage and entertain users.
While previous chatbots used pre-programmed scripts
to respond to prompts, Tay's machine learning capabilities allowed it to be
directly influenced by its conversations.
A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay,
which eventually led to Tay generating similarly inflammatory content towards other users.
Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology
with lessons learned from the bot's failure.
References
- https://incidentdatabase.ai/cite/6
- https://avidml.org/database/avid-2022-v013/
- https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
- https://spectrum.ieee.org/tech-talk/artificial-intelligence/machine-learning/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation
Source: MITRE ATLAS case studies, used under Apache 2.0.