INCIDENT MAP

Browse / Model Theft and Extraction / AML.CS0058

Google Photos AI Model Extraction

AML.CS0058 · 2025-03-01 case study

Class
Model Theft and Extraction
Technique
AI Artifact Collection
Also involves
Model Evasion, Misinformation and External Harm
Target
Google Photos Android App (Exercise)

How it worked

  1. Discovery · Discover AI Artifacts

    Skyld researchers analyzed the Google Photos Android Package (APK) and identified TensorFlow Lite as the machine learning framework used by the app. They searched the application package and native libraries for TFLite artifacts using the TFL3 file identifier.

  2. Collection · AI Artifact Collection

    The researchers collected TensorFlow Lite model artifacts from multiple locations in the APK, including unencrypted assets, files embedded in the native library, and application-specific folders.

  3. Exfiltration · Exfiltration via Cyber Means

    The researchers used static analysis and Frida instrumentation to recover model files. For encrypted models, they intercepted decrypted TFLite files during runtime as Google Photos loaded them for execution.

  4. AI Model Access · Full AI Model Access

    Exfiltrating model files from the APK gave the researchers full access to the Google Photos AI models, including those used for tasks such as face detection, object detection, segmentation, depth estimation, image quality assessment, and blur detection.

  5. AI Attack Adaptation · Craft Adversarial Data: White-Box Optimization

    The recovered TensorFlow Lite models could enable white-box adversarial example generation.

  6. Impact · Evade AI Model

    Adversarial data could be used to evade or otherwise degrade the Google Photos models including face and object detection.

  7. Impact · External Harms: AI Intellectual Property Theft

    The recovered models represented proprietary Google Photos AI assets. An adversary or competitor could use the extracted models to study, reuse, or replicate Google Photos capabilities, reducing the cost of independently developing similar features.

Summary

Skyld researchers analyzed the Google Photos Android application and recovered TensorFlow Lite models used by AI-powered photo editing and image analysis features. The researchers found models stored unencrypted in the application's assets, embedded in the native library, and encrypted on disk. They used static reverse engineering to locate TFLite artifacts and dynamic instrumentation with Frida to capture encrypted models after runtime decryption. The recovered models provided white-box access to proprietary Google Photos model artifacts.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.