INCIDENT MAP

Browse / Model Theft and Extraction / AML.CS0007

GPT-2 Model Replication

AML.CS0007 · 2019-08-22 case study

Class
Model Theft and Extraction
Technique
Replicate AI Model Behavior: Replicate Behavior from Gathered AI Artifacts
Target
OpenAI GPT-2 (Exercise)

How it worked

  1. Reconnaissance · Search Open Technical Databases

    Using the public documentation about GPT-2, the researchers gathered information about the dataset, model architecture, and training hyper-parameters.

  2. Resource Development · Acquire Public AI Artifacts: Models

    The researchers obtained a reference implementation of a similar publicly available model called Grover.

  3. Resource Development · Acquire Public AI Artifacts: Datasets

    The researchers were able to manually recreate the dataset used in the original GPT-2 paper using the gathered documentation.

  4. Resource Development · Acquire Infrastructure: AI Development Workspaces

    The researchers were able to use TensorFlow Research Cloud via their academic credentials.

  5. AI Attack Adaptation · Replicate AI Model Behavior: Replicate Behavior from Gathered AI Artifacts

    The researchers modified Grover's objective function to reflect GPT-2's objective function and then trained on the dataset they curated using used Grover's initial hyperparameters. The resulting model functionally replicates GPT-2, obtaining similar performance on most datasets.
    A bad actor who followed the same procedure as the researchers could then use the replicated GPT-2 model for malicious purposes.

Summary

OpenAI built GPT-2, a language model capable of generating high quality text samples. Over concerns that GPT-2 could be used for malicious purposes such as impersonating others, or generating misleading news articles, fake social media content, or spam, OpenAI adopted a tiered release schedule. They initially released a smaller, less powerful version of GPT-2 along with a technical description of the approach, but held back the full trained model.

Before the full model was released by OpenAI, researchers at Brown University successfully replicated the model using information released by OpenAI and open source ML artifacts. This demonstrates that a bad actor with sufficient technical skill and compute resources could have replicated GPT-2 and used it for harmful goals before the AI Security community is prepared.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.