INCIDENT MAP

Browse / Unsafe Model Loading and Supply Chain / AML.CS0009

Tay Poisoning

AML.CS0009 · 2016-03-23 case study

Class
Unsafe Model Loading and Supply Chain
Technique
Supply Chain Compromise: Data Repository
Also involves
Data and Model Poisoning, Model Evasion
Target
Microsoft's Tay AI Chatbot (Incident)

How it worked

  1. AI Model Access · AI-Enabled Product or Service

    Adversaries were able to interact with Tay via Twitter messages.

  2. Initial Access · Supply Chain Compromise: Data Repository

    Tay bot used the interactions with its Twitter users as training data to improve its conversations.
    Adversaries were able to coordinate with the intent of defacing Tay bot by exploiting this feedback loop.

  3. Persistence · Training Data Poisoning

    By repeatedly interacting with Tay using racist and offensive language, they were able to skew Tay's dataset towards that language as well. This was done by adversaries using the "repeat after me" function, a command that forced Tay to repeat anything said to it.

  4. Impact · Erode AI Model Integrity

    As a result of this coordinated attack, Tay's conversation algorithms began to learn to generate reprehensible material. Tay's internalization of this detestable language caused it to be unpromptedly repeated during interactions with innocent users.

Summary

Microsoft created Tay, a Twitter chatbot designed to engage and entertain users.
While previous chatbots used pre-programmed scripts
to respond to prompts, Tay's machine learning capabilities allowed it to be
directly influenced by its conversations.

A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay,
which eventually led to Tay generating similarly inflammatory content towards other users.

Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology
with lessons learned from the bot's failure.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.