Skip to content

Latest commit

 

History

History
144 lines (107 loc) · 9.15 KB

mitigations.md

File metadata and controls

144 lines (107 loc) · 9.15 KB

Mitigation Strategies Analysis for ray-project/ray

Mitigation Strategy: Enable Ray Authentication

Description:

  1. Choose Authentication Method: Ray supports password-based authentication.
  2. Configure Ray: When starting the Ray cluster (using ray start), set the RAY_ADDRESS and --redis-password flags. Example: RAY_ADDRESS='auto' ray start --head --node-ip-address="<head_node_ip>" --port=6379 --dashboard-host=0.0.0.0 --redis-password="<your_strong_password>"
  3. Client-Side Configuration: When connecting to the cluster from client code (using ray.init()), provide the authentication credentials (the password).
  4. Regular Password Rotation: Establish a policy and process for regularly changing the Ray cluster password. Automate this if possible, and ensure the updated password is used in all client connection configurations.

  • Threats Mitigated:

    • Unauthorized Cluster Access (Severity: Critical): Prevents attackers from connecting to the Ray cluster and executing arbitrary code, accessing data, or disrupting operations.
    • Unauthorized Dashboard Access (Severity: High): Prevents attackers from gaining insights into the cluster's state.
    • Data Exfiltration (Severity: High): Reduces the risk of attackers stealing data by preventing unauthorized access.
    • Denial of Service (DoS) (Severity: High): Makes it harder for attackers to launch DoS attacks.

  • Impact:

    • Unauthorized Cluster Access: Risk reduced from Critical to Low.
    • Unauthorized Dashboard Access: Risk reduced from High to Low.
    • Data Exfiltration: Risk significantly reduced.
    • Denial of Service (DoS): Risk reduced.

  • Currently Implemented:

    • (Example: "Authentication is enabled for all client connections and the dashboard using a shared password, configured in the start_ray_cluster.sh script.")

  • Missing Implementation:

    • (Example: "We need to implement a script to automate password rotation and update the start_ray_cluster.sh script and client connection code accordingly.")

Mitigation Strategy: Enable Ray TLS Encryption

Description:

  1. Generate Certificates: Generate TLS certificates (and private keys) for the Ray head node and worker nodes.
  2. Configure Ray Head Node: When starting the Ray head node, specify the paths to the certificate and private key files using the command-line arguments: --node-cert-path, --node-private-key-path.
  3. Configure Ray Worker Nodes: Configure worker nodes to use TLS certificates when connecting to the head node, using similar configuration options.
  4. Configure Ray Client: Configure the Ray client (ray.init()) to use TLS when connecting to the cluster. This may involve specifying the CA certificate.
  5. Certificate Rotation: Implement a process for regularly rotating the TLS certificates.

  • Threats Mitigated:

    • Man-in-the-Middle (MITM) Attacks (Severity: High): Prevents attackers from intercepting and modifying communication.
    • Eavesdropping (Severity: High): Protects data in transit.
    • Data Tampering (Severity: High): Ensures data integrity.

  • Impact:

    • MITM Attacks: Risk reduced from High to Very Low.
    • Eavesdropping: Risk reduced from High to Very Low.
    • Data Tampering: Risk reduced from High to Very Low.

  • Currently Implemented:

    • (Example: "TLS encryption is enabled using self-signed certificates. Configuration is done in start_ray_cluster.sh.")

  • Missing Implementation:

    • (Example: "We need to switch to certificates from a trusted CA and implement automated certificate rotation.")

Mitigation Strategy: Configure Ray Resource Limits

Description:

  1. CPU Limits: Specify CPU limits for Ray tasks and actors using the @ray.remote(num_cpus=...) decorator.
  2. Memory Limits: Specify memory limits using @ray.remote(memory=...).
  3. GPU Limits: Specify GPU limits using @ray.remote(num_gpus=...).
  4. Custom Resources: Define and use custom resources using @ray.remote(resources={"custom_resource": ...}) if needed.
  5. Object Store Memory: Limit the Ray object store memory using the --object-store-memory flag when starting Ray.

  • Threats Mitigated:

    • Denial of Service (DoS) (Severity: High): Prevents resource exhaustion.
    • Resource Exhaustion (Severity: High): Protects the cluster.
    • Performance Degradation (Severity: Medium): Ensures responsiveness.

  • Impact:

    • DoS: Risk significantly reduced.
    • Resource Exhaustion: Risk significantly reduced.
    • Performance Degradation: Risk significantly reduced.

  • Currently Implemented:

    • (Example: "CPU and memory limits are set for all tasks using the @ray.remote decorator.")

  • Missing Implementation:

    • (Example: "We need to define and apply GPU limits for tasks that utilize GPUs.")

Mitigation Strategy: Use Ray Task Prioritization

Description:

  1. Identify Critical Tasks: Determine which tasks are most critical to the application's functionality.
  2. Assign Priorities: Use the @ray.remote(priority=...) decorator to assign higher priorities to critical tasks. Ray uses numerical priorities (higher numbers indicate higher priority).

  • Threats Mitigated:

    • Denial of Service (DoS) (Severity: High): Helps ensure critical tasks are executed even under heavy load.
    • Performance Degradation (Severity: Medium): Improves responsiveness for important tasks.

  • Impact:

    • DoS: Risk reduced (helps mitigate, but not a complete solution).
    • Performance Degradation: Risk reduced for prioritized tasks.

  • Currently Implemented:

    • (Example: "Task prioritization is not currently implemented.")

  • Missing Implementation:

    • (Example: "We need to identify critical tasks and add the @ray.remote(priority=...) decorator to their definitions.")

Mitigation Strategy: Secure Ray Serialization (Custom Serializers)

Description:

  1. Avoid Pickle with Untrusted Data: Do not use pickle to deserialize data from untrusted sources.
  2. Prefer Safer Formats: Use JSON, Protocol Buffers, or Apache Arrow for external data.
  3. Custom Serializers (If Pickle is Necessary): If you must use pickle for internal data transfer, implement custom serializers and deserializers using Ray's custom serialization API. This involves:
    • Defining classes that inherit from ray.serialization.SerializationContext.
    • Implementing the serialize and deserialize methods with strict validation to prevent code execution. This is a complex and error-prone process.

  • Threats Mitigated:

    • Remote Code Execution (RCE) via Pickle (Severity: Critical): Prevents code execution through malicious pickle payloads.
    • Data Injection (Severity: High): Reduces the risk of data injection.

  • Impact:

    • RCE via Pickle: Risk reduced from Critical to Very Low (with properly implemented custom serializers).
    • Data Injection: Risk reduced.

  • Currently Implemented:

    • (Example: "We are using JSON for all external data. We are using default pickle for internal data.")

  • Missing Implementation:

    • (Example: "We need to implement custom serializers for all internal data transfer to eliminate the use of default pickle.")

Mitigation Strategy: Configure Ray Logging

Description:

  1. Set Logging Level: When starting Ray, use the --logging-level flag to control the verbosity of logs (e.g., ray start --head --logging-level=info). Choose from debug, info, warning, error, or critical.
  2. Log File Rotation: Configure log file rotation to prevent log files from growing indefinitely. This is typically handled by the underlying logging system, but Ray's configuration might influence it.
  3. Structured Logging (Advanced): Consider using structured logging (e.g., logging in JSON format) to make it easier to parse and analyze logs. This may require custom configuration within your Ray tasks.

  • Threats Mitigated:

    • Undetected Attacks (Severity: High): Provides visibility into cluster activity.
    • Delayed Incident Response (Severity: High): Enables faster response.
    • Lack of Forensic Evidence (Severity: High): Provides logs for analysis.

  • Impact:

    • Undetected Attacks: Risk significantly reduced.
    • Delayed Incident Response: Risk significantly reduced.
    • Lack of Forensic Evidence: Risk significantly reduced.

  • Currently Implemented:

    • (Example: "Ray logging is enabled with the default info level.")

  • Missing Implementation:

    • (Example: "We need to configure log file rotation and consider switching to structured logging for easier analysis.")