INCIDENT MAP

Browse / Model Evasion / AML.CS0005

Attack on Machine Translation Services

AML.CS0005 · 2020-04-30 case study

Class
Model Evasion
Technique
Craft Adversarial Data: Black-Box Transfer
Also involves
Model Theft and Extraction, Misinformation and External Harm
Target
Google Translate, Bing Translator, Systran Translate (Exercise)

How it worked

  1. Reconnaissance · Search Open Technical Databases

    The researchers used published research papers to identify the datasets and model architectures used by the target translation services.

  2. Resource Development · Acquire Public AI Artifacts: Datasets

    The researchers gathered similar datasets that the target translation services used.

  3. Resource Development · Acquire Public AI Artifacts: Models

    The researchers gathered similar model architectures that the target translation services used.

  4. Collection · Outputs from AI Models

    They abused a public facing application to query the model and produced machine translated sentence pairs as training data.

  5. AI Attack Adaptation · Replicate AI Model Behavior: Replicate Behavior from Model Outputs

    Using these translated sentence pairs, the researchers trained a model that replicates the behavior of the target model.

  6. Impact · External Harms: AI Intellectual Property Theft

    By replicating the model with high fidelity, the researchers demonstrated that an adversary could steal a model and violate the victim's intellectual property rights.

  7. AI Attack Adaptation · Craft Adversarial Data: Black-Box Transfer

    The replicated models were used to generate adversarial examples that successfully transferred to the black-box translation services.

  8. Impact · Evade AI Model

    The adversarial examples were used to evade the machine translation services by a variety of means. This included targeted word flips, vulgar outputs, and dropped sentences.

  9. Impact · Erode AI Model Integrity

    Adversarial attacks can cause errors that cause reputational damage to the company of the translation service and decrease user trust in AI-powered services.

Summary

Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs.
A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality.
Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services.
These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.