INCIDENT MAP

Browse / Model Evasion / AML.CS0008

ProofPoint Evasion

AML.CS0008 · 2019-09-09 case study

Class
Model Evasion
Technique
Craft Adversarial Data: Black-Box Transfer
Also involves
Model Theft and Extraction
Target
ProofPoint Email Protection System (Exercise)

How it worked

  1. Discovery · Discover AI Model Outputs

    The researchers discovered that ProofPoint's Email Protection left model output scores in email headers.

  2. Collection · Outputs from AI Models

    The researchers sent many emails through the system to collect model outputs from the headers.

  3. AI Attack Adaptation · Replicate AI Model Behavior: Replicate Behavior from Model Outputs

    The researchers used the emails and collected scores as a dataset, which they used to train a functional copy of the ProofPoint model.

    Basic correlation was used to decide which score variable speaks generally about the security of an email. The "mlxlogscore" was selected in this case due to its relationship with spam, phish, and core mlx and was used as the label. Each "mlxlogscore" was generally between 1 and 999 (higher score = safer sample). Training was performed using an Artificial Neural Network (ANN) and Bag of Words tokenizing.

  4. AI Attack Adaptation · Craft Adversarial Data: Black-Box Transfer

    Next, the ML researchers algorithmically found samples from this "offline" proxy model that helped give desired insight into its behavior and influential variables.

    Examples of good scoring samples include "calculation", "asset", and "tyson".
    Examples of bad scoring samples include "software", "99", and "unsub".

  5. Impact · Evade AI Model

    Finally, these insights from the "offline" proxy model allowed the researchers to create malicious emails that received preferable scores from the real ProofPoint email protection system, hence bypassing it.

Summary

Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.