Skip to content

Latest commit

 

History

History
80 lines (67 loc) · 6.4 KB

threat-modeling.md

File metadata and controls

80 lines (67 loc) · 6.4 KB

Threat Model Analysis for microsoft/cntk

Threat: Model Poisoning (Training Data Poisoning)

  • Description: An attacker injects malicious data into the training dataset used to create the CNTK model. This could involve adding mislabeled examples, modifying existing examples, or introducing entirely new, malicious examples. The attacker's goal is to subtly alter the model's behavior during training.
    • Impact: The trained model will have degraded performance or exhibit biased behavior, potentially favoring the attacker's goals. The model may appear to function normally on clean data but fail on specific inputs or exhibit unexpected behavior.
    • CNTK Component Affected: The training process itself, specifically the cntk.Trainer and the data readers (cntk.io) used to feed data to the trainer. The resulting cntk.ops.functions.Function (the trained model) is the compromised artifact.
    • Risk Severity: High.
    • Mitigation Strategies:
      • Data Provenance: Maintain a clear and auditable record of the origin and history of all training data.
      • Data Sanitization and Validation: Rigorously validate and sanitize all training data before use. Look for anomalies, outliers, and inconsistencies.
      • Trusted Data Sources: Use only trusted and verified sources for training data.
      • Anomaly Detection: Employ anomaly detection techniques on the training data to identify potentially malicious samples.
      • Differential Privacy (During Training): Add noise during training to limit the influence of any single data point.

Threat: Model File Tampering

  • Description: An attacker gains unauthorized access to the stored CNTK model file (e.g., .model, .dnn) and modifies its contents. This could involve changing weights, biases, network architecture, or inserting malicious code (if the file format allows it).
    • Impact: The modified model will behave unpredictably, potentially producing incorrect results, exhibiting biased behavior, or even executing malicious code (if the attacker can inject code).
    • CNTK Component Affected: The cntk.ops.functions.Function.load() function (used to load the model) and the model file itself.
    • Risk Severity: Critical.
    • Mitigation Strategies:
      • File Integrity Monitoring: Use cryptographic hashes (e.g., SHA-256).
      • Digital Signatures: Digitally sign the model file.
      • Access Control: Implement strict access control restrictions.
      • Secure Storage: Store model files in a secure location.
      • Regular Audits: Periodically audit model files.

Threat: CNTK Library Tampering (Supply Chain Attack)

  • Description: An attacker compromises the CNTK library itself (e.g., DLLs, .so files, or Python files) before or during installation. This is a supply chain attack.
    • Impact: The compromised CNTK library could introduce arbitrary vulnerabilities, allowing the attacker to control model execution, steal data, or execute arbitrary code.
    • CNTK Component Affected: Potentially any part of the CNTK library, including cntk.ops, cntk.io, cntk.Trainer.
    • Risk Severity: Critical.
    • Mitigation Strategies:
      • Verified Package Managers: Use a trusted package manager with signature verification.
      • Hash Verification: Manually verify the hash of the downloaded CNTK library files.
      • Containerization: Use a containerized environment with a known-good CNTK image.
      • Migrate to PyTorch: This is the most effective long-term mitigation.

Threat: Resource Exhaustion (CNTK-Specific DoS)

  • Description: An attacker sends computationally expensive inputs to the CNTK model, causing excessive CPU, GPU, or memory consumption.
    • Impact: Denial of service.
    • CNTK Component Affected: The cntk.ops.functions.Function object, specifically the forward pass computation (model(input)).
    • Risk Severity: High.
    • Mitigation Strategies:
      • Input Size Limits: Enforce strict limits on input size and complexity.
      • Resource Quotas: Set resource quotas (CPU, GPU, memory).
      • Timeouts: Implement timeouts for model inference.
      • Load Balancing: Distribute requests across multiple instances.
      • Asynchronous Processing: Use asynchronous processing.

Threat: Code Execution via Malicious Model File

  • Description: An attacker crafts a malicious CNTK model file that exploits a vulnerability in CNTK's model loading to execute arbitrary code.
    • Impact: Complete system compromise.
    • CNTK Component Affected: The cntk.ops.functions.Function.load() function, and potentially other parts of the CNTK library involved in parsing the model file.
    • Risk Severity: Critical.
    • Mitigation Strategies:
      • Input Validation (Model File): Validate the model file's structure and contents before loading.
      • Sandboxing: Load and execute the model in a sandboxed environment.
      • Least Privilege: Run the CNTK process with minimum necessary privileges.
      • Secure Deserialization: Use a secure deserialization library or technique.
      • Migrate to PyTorch: For better security and active vulnerability patching.

Threat: Adversarial Input (Model Evasion)

  • Description: Attacker crafts malicious input to fool the CNTK model, causing misclassification.
    • Impact: Incorrect model predictions, leading to incorrect application behavior, potentially with serious consequences.
    • CNTK Component Affected: cntk.ops.functions.Function object, specifically the forward pass.
    • Risk Severity: High.
    • Mitigation Strategies:
      • Adversarial Training: Train on adversarial examples.
      • Input Sanitization (Model-Specific): Model-specific input validation.
      • Defensive Distillation: Train a "distilled" model.
      • Ensemble Methods: Use multiple models.