Skip to content

Latest commit

 

History

History
92 lines (81 loc) · 8.04 KB

threat-modeling.md

File metadata and controls

92 lines (81 loc) · 8.04 KB

Threat Model Analysis for ultralytics/yolov5

Threat: Malicious Model Substitution

  • Description: An attacker replaces the legitimate weights file (e.g., yolov5s.pt) and potentially the corresponding configuration file (*.yaml) with a crafted malicious version. The attacker could gain access through compromised infrastructure, supply chain attacks (e.g., compromised download source), or physical access to the device.
  • Impact:
    • Complete control over model predictions.
    • Denial of service (incorrect or no detections).
    • Execution of arbitrary code (if the malicious model exploits vulnerabilities in the loading process).
    • Information leakage (the malicious model could exfiltrate data).
  • YOLOv5 Component Affected:
    • models/ directory (specifically the weight files .pt and configuration files .yaml).
    • torch.load() function (used to load the model).
    • Potentially any part of the code that uses the loaded model.
  • Risk Severity: Critical
  • Mitigation Strategies:
    • Code Signing and Verification: Digitally sign the .pt and .yaml files. Modify the detect.py or equivalent loading script to verify the signature before calling torch.load().
    • Secure Model Storage: Store models in a secure, access-controlled location (e.g., encrypted volume, cloud storage with strict IAM policies).
    • Hash Verification: Before loading, calculate the SHA-256 hash of the .pt and .yaml files and compare it to a known-good hash stored securely (e.g., in a separate configuration file or database). Integrate this check into the loading process.
    • Immutable Infrastructure: Deploy the model as part of an immutable container image (e.g., Docker). Any changes would require rebuilding the image.

Threat: Unauthorized Model Modification (Subtle Weight Changes)

  • Description: An attacker gains write access to the model file (.pt) but doesn't replace it entirely. They make small, targeted changes to the weights, potentially introducing subtle biases or vulnerabilities. This is harder to detect than complete replacement.
  • Impact:
    • Degraded model accuracy.
    • Introduction of specific blind spots or misclassifications.
    • Increased susceptibility to adversarial examples.
  • YOLOv5 Component Affected:
    • models/ directory (specifically the weight file .pt).
    • The entire model architecture (as any weight could be modified).
  • Risk Severity: High
  • Mitigation Strategies:
    • Strict File System Permissions: Ensure the model file is read-only for the application user. Only a dedicated, highly privileged process should have write access (e.g., during model updates).
    • File Integrity Monitoring (FIM): Use a FIM tool (e.g., AIDE, Tripwire, OS-specific tools) to monitor the .pt file for any changes. Alert on unauthorized modifications.
    • Regular Model Retraining/Re-downloading: Periodically replace the deployed model with a freshly trained or downloaded version from a trusted source. This reduces the window of opportunity for an attacker.

Threat: Adversarial Example Attack (Evasion)

  • Description: An attacker crafts an image or video frame with subtle, imperceptible perturbations designed to cause the model to misclassify objects or fail to detect them. This is done at inference time. The attacker does not need access to the model files.
  • Impact:
    • Incorrect object detection results.
    • Bypassing security systems (e.g., intrusion detection).
    • Denial of service (if the application relies on accurate detections).
  • YOLOv5 Component Affected:
    • detect.py (or equivalent inference script).
    • The entire model architecture (as the attack targets the model's decision-making process).
    • utils/general.py (functions related to non-maximum suppression and confidence thresholding could be manipulated by adversarial examples).
  • Risk Severity: High
  • Mitigation Strategies:
    • Adversarial Training: Include adversarial examples in the training dataset. This requires modifying the training scripts (train.py) and potentially using specialized libraries for generating adversarial examples.
    • Input Preprocessing: Apply transformations like random resizing, cropping, blurring, or adding small amounts of noise to the input image before passing it to the model. This can disrupt the carefully crafted perturbations. Implement this in detect.py or a dedicated preprocessing module.
    • Ensemble Methods: Use multiple YOLOv5 models (potentially with different architectures or training data) and combine their predictions (e.g., majority voting). This makes the system more robust.
    • Defensive Distillation: (More complex) Train a "distilled" model that is less sensitive to small input changes.

Threat: Data Poisoning (Training Time Attack)

  • Description: An attacker injects malicious data into the training dataset used to train the YOLOv5 model. This happens before deployment. The attacker might subtly alter images or labels to introduce biases or weaknesses.
  • Impact:
    • Reduced model accuracy.
    • Introduction of specific vulnerabilities or blind spots.
    • Biased or unfair predictions.
  • YOLOv5 Component Affected:
    • train.py (and associated data loading functions).
    • The training dataset itself (images and labels).
    • The resulting trained model (.pt file).
  • Risk Severity: High
  • Mitigation Strategies:
    • Data Source Verification: Use only trusted datasets or carefully vet the source of your training data.
    • Data Sanitization: Implement rigorous data cleaning and filtering procedures to remove outliers, inconsistencies, and potentially malicious data points. This might involve manual inspection or automated anomaly detection techniques.
    • Data Augmentation (Careful Use): While data augmentation is generally beneficial, be cautious about using techniques that could inadvertently amplify the effects of poisoned data.
    • Regularization: Use appropriate regularization techniques during training (e.g., weight decay, dropout) to prevent the model from overfitting to the poisoned data.

Threat: Inference Data Exfiltration

  • Description: An attacker gains unauthorized access to the system (e.g., through a separate vulnerability or insider threat) and steals the images or videos being processed by YOLOv5. This is particularly relevant if the application handles sensitive data.
    • Impact:
      • Loss of confidentiality of sensitive image data.
      • Privacy violations.
      • Compliance violations (e.g., GDPR, HIPAA).
    • YOLOv5 Component Affected:
      • detect.py (or equivalent inference script) - where the image data is loaded and processed.
      • Any temporary storage used for images.
      • Potentially the network interface if images are transmitted unencrypted.
    • Risk Severity: High (if sensitive data is involved)
    • Mitigation Strategies:
      • Encryption at Rest: Encrypt the storage where images are stored, both before and after processing.
      • Encryption in Transit: Use HTTPS or other secure protocols to transmit images to and from the YOLOv5 processing component.
      • Access Control: Implement strict access control lists (ACLs) and least privilege principles to limit access to the system and the data.
      • Auditing: Enable detailed audit logs to track all access to the system and the image data. Monitor these logs for suspicious activity.
      • Memory Management: Ensure that image data is securely erased from memory after processing. Avoid unnecessary caching or persistence of image data.