Skip to content

Latest commit

 

History

History
278 lines (223 loc) · 25.1 KB

mitigations.md

File metadata and controls

278 lines (223 loc) · 25.1 KB

Mitigation Strategies Analysis for apache/hadoop

Mitigation Strategy: Kerberos Authentication

  • Description:

    • Step 1: Install and configure a Kerberos Key Distribution Center (KDC). This server will manage authentication tickets.
    • Step 2: Integrate Hadoop services (NameNode, DataNode, ResourceManager, NodeManager, etc.) with Kerberos. This involves modifying Hadoop configuration files (e.g., core-site.xml, hdfs-site.xml, yarn-site.xml) to enable Kerberos security.
    • Step 3: Create Kerberos principals for each Hadoop service and user that needs to access the Hadoop cluster. Use kadmin.local or similar Kerberos administration tools for principal creation.
    • Step 4: Generate keytab files for each service principal. Keytabs are securely stored files containing the service's Kerberos credentials.
    • Step 5: Distribute keytab files securely to the servers running the respective Hadoop services and configure Hadoop services to use these keytabs for authentication.
    • Step 6: Configure Hadoop clients (e.g., command-line tools, applications) to use Kerberos for authentication. This often involves setting environment variables or using kinit to obtain Kerberos tickets.
    • Step 7: Test Kerberos authentication thoroughly to ensure all services and clients can authenticate correctly.

  • List of Threats Mitigated:

    • Unauthorized Access (High Severity): Prevents unauthorized users and services from accessing Hadoop resources and data. Without Kerberos, default simple authentication is easily bypassed.
    • Spoofing (High Severity): Reduces the risk of attackers impersonating legitimate users or services to gain unauthorized access.
    • Man-in-the-Middle Attacks (Medium Severity): Mitigates the risk of attackers intercepting and stealing authentication credentials transmitted over the network.
    • Replay Attacks (Medium Severity): Reduces the risk of attackers capturing and replaying authentication credentials to gain unauthorized access.

  • Impact:

    • Unauthorized Access: High reduction in risk. Kerberos significantly strengthens authentication, making unauthorized access much more difficult.
    • Spoofing: High reduction in risk. Kerberos provides strong identity verification, making spoofing highly challenging.
    • Man-in-the-Middle Attacks: Medium reduction in risk. Kerberos uses encryption and secure ticket exchange, reducing the effectiveness of man-in-the-middle attacks on authentication.
    • Replay Attacks: Medium reduction in risk. Kerberos tickets have a limited lifespan and are designed to prevent replay attacks.

  • Currently Implemented:

    • Currently implemented for HDFS NameNode and DataNodes in the staging environment. Configuration files are located in /etc/hadoop/conf/hdfs-site.xml and /etc/hadoop/conf/core-site.xml on the respective servers.

  • Missing Implementation:

    • Not yet implemented for YARN ResourceManager and NodeManagers in any environment.
    • Kerberos integration with Hive and HBase is planned but not yet started.
    • Production environment implementation is not yet started and is a high priority.

Mitigation Strategy: Hadoop Access Control Lists (ACLs)

  • Description:

    • Step 1: Enable ACLs in HDFS. This is typically done by setting dfs.namenode.acls.enabled to true in hdfs-site.xml and restarting the NameNode.
    • Step 2: Define a clear access control policy based on user roles and data sensitivity. Determine which users or groups should have read, write, or execute permissions on specific HDFS directories and files.
    • Step 3: Use Hadoop command-line tools (e.g., hdfs dfs -setfacl) or programmatic APIs to set ACLs on HDFS directories and files. ACLs can be set for users, groups, and masks.
    • Step 4: Regularly review and update ACLs as user roles and data access requirements change. Implement a process for managing and auditing ACL changes.
    • Step 5: For YARN, configure YARN ACLs in yarn-site.xml to control access to applications, queues, and administrative functions.

  • List of Threats Mitigated:

    • Unauthorized Data Access (High Severity): Prevents users from accessing data they are not authorized to view, modify, or delete.
    • Data Breaches (High Severity): Reduces the risk of data breaches by limiting access to sensitive data to only authorized personnel.
    • Privilege Escalation (Medium Severity): Limits the impact of compromised accounts by restricting their access to only necessary resources.
    • Insider Threats (Medium Severity): Mitigates risks from malicious insiders by enforcing the principle of least privilege.

  • Impact:

    • Unauthorized Data Access: High reduction in risk. ACLs provide granular control over data access, significantly reducing unauthorized access.
    • Data Breaches: High reduction in risk. By limiting access, ACLs minimize the potential scope of a data breach.
    • Privilege Escalation: Medium reduction in risk. ACLs help contain the damage from compromised accounts by limiting their privileges.
    • Insider Threats: Medium reduction in risk. ACLs enforce least privilege, making it harder for insiders to misuse their access.

  • Currently Implemented:

    • Basic HDFS ACLs are enabled in the development environment. Some initial ACLs are set for project directories under /user/project_data/.

  • Missing Implementation:

    • Comprehensive ACL policy definition and implementation are missing across all environments.
    • YARN ACLs are not configured.
    • No process for regular ACL review and updates is in place.
    • Production environment ACL configuration is not started.

Mitigation Strategy: Apache Ranger for Centralized Authorization

  • Description:

    • Step 1: Install and configure Apache Ranger. This involves deploying Ranger Admin and Ranger Agents for the Hadoop services you want to secure (HDFS, Hive, HBase, YARN, etc.).
    • Step 2: Integrate Ranger with your existing identity management system (e.g., LDAP, Active Directory) for user and group synchronization.
    • Step 3: Define authorization policies in Ranger Admin through its web UI or API. Policies specify who (users, groups) can perform what actions (read, write, execute, etc.) on which resources (HDFS paths, Hive databases/tables, HBase tables/column families, YARN queues, etc.).
    • Step 4: Deploy Ranger Agents to Hadoop nodes. Agents intercept access requests to Hadoop services and enforce the policies defined in Ranger Admin.
    • Step 5: Configure Ranger audit logging to track access attempts and policy enforcement decisions. Integrate Ranger audit logs with a SIEM system for monitoring and analysis.
    • Step 6: Regularly review and refine Ranger policies to adapt to changing security requirements and access patterns.

  • List of Threats Mitigated:

    • Inconsistent Authorization Policies (Medium Severity): Ranger ensures consistent policy enforcement across different Hadoop components, preventing inconsistencies and gaps in security.
    • Complex ACL Management (Medium Severity): Ranger simplifies policy management compared to managing individual ACLs for each service.
    • Lack of Centralized Audit (Medium Severity): Ranger provides centralized auditing of access attempts and policy enforcement, improving security monitoring and compliance.
    • Policy Management Overhead (Medium Severity): Without a centralized system, managing authorization policies across a large Hadoop cluster can become complex and error-prone.

  • Impact:

    • Inconsistent Authorization Policies: Medium reduction in risk. Ranger enforces policies consistently, reducing the risk of misconfigurations and security gaps.
    • Complex ACL Management: Medium reduction in risk. Ranger simplifies policy management, making it easier to maintain and audit authorization rules.
    • Lack of Centralized Audit: Medium reduction in risk. Centralized audit logs improve security visibility and incident response capabilities.
    • Policy Management Overhead: Medium reduction in risk. Ranger reduces the administrative burden of managing authorization policies.

  • Currently Implemented:

    • Apache Ranger is not currently implemented in any environment.

  • Missing Implementation:

    • Ranger deployment and configuration are completely missing.
    • Integration with identity management system is not planned yet.
    • Policy definition and agent deployment for Hadoop services are not started.
    • Audit logging and SIEM integration are not considered.

Mitigation Strategy: HDFS Encryption at Rest

  • Description:

    • Step 1: Choose an encryption method for HDFS at rest. Options include Encryption Zones (using Hadoop KMS) or Transparent Encryption. Encryption Zones are generally recommended for granular control.
    • Step 2: Set up and configure Hadoop Key Management Server (KMS). KMS is responsible for managing encryption keys.
    • Step 3: Create encryption keys in KMS for HDFS encryption.
    • Step 4: Create Encryption Zones in HDFS for directories containing sensitive data. Specify the encryption key to be used for each zone.
    • Step 5: Data written to Encryption Zones will be automatically encrypted. Data read from Encryption Zones will be automatically decrypted for authorized users.
    • Step 6: Implement key rotation policies for encryption keys to enhance security.
    • Step 7: Securely manage access to KMS and encryption keys. Implement strong authentication and authorization for KMS administrators.

  • List of Threats Mitigated:

    • Data Theft from Stolen Storage Media (High Severity): Protects data if physical storage media (disks, tapes) are stolen or improperly disposed of.
    • Unauthorized Physical Access to Data (High Severity): Prevents unauthorized access to data by individuals with physical access to the storage infrastructure.
    • Insider Threats with Physical Access (Medium Severity): Mitigates risks from malicious insiders who might gain physical access to storage media.
    • Data Breaches due to Storage Misconfiguration (Medium Severity): Reduces the risk of data exposure due to misconfigured storage systems.

  • Impact:

    • Data Theft from Stolen Storage Media: High reduction in risk. Encryption renders data unreadable without the encryption keys, making stolen media useless to attackers.
    • Unauthorized Physical Access to Data: High reduction in risk. Encryption protects data even if physical access is gained to the storage.
    • Insider Threats with Physical Access: Medium reduction in risk. Encryption adds a layer of protection against insiders with physical access but not key access.
    • Data Breaches due to Storage Misconfiguration: Medium reduction in risk. Encryption can protect data even if storage access controls are misconfigured.

  • Currently Implemented:

    • HDFS Encryption at Rest is not currently implemented in any environment.

  • Missing Implementation:

    • KMS setup and configuration are missing.
    • Encryption key creation and management are not implemented.
    • Encryption Zones are not defined in HDFS.
    • Key rotation policies are not planned.

Mitigation Strategy: Data Encryption in Transit (TLS/SSL)

  • Description:

    • Step 1: Obtain TLS/SSL certificates for Hadoop services. Use certificates signed by a trusted Certificate Authority (CA) or generate self-signed certificates for testing environments.
    • Step 2: Configure Hadoop web UIs (NameNode UI, ResourceManager UI, etc.) to use HTTPS. This involves configuring web server settings within Hadoop service configurations to enable TLS/SSL and specify the certificate and key files.
    • Step 3: Enable RPC encryption for inter-node communication within the Hadoop cluster. This is typically configured in Hadoop configuration files (e.g., core-site.xml, hdfs-site.xml, yarn-site.xml) by setting properties to enable RPC encryption and specify the encryption protocol (e.g., using Kerberos or SASL).
    • Step 4: Ensure that clients connecting to Hadoop services are configured to use encrypted connections (e.g., using HTTPS for web UIs, secure RPC protocols for programmatic access).
    • Step 5: Regularly update TLS/SSL certificates to maintain security and prevent certificate expiration.

  • List of Threats Mitigated:

    • Eavesdropping (High Severity): Prevents attackers from intercepting and reading sensitive data transmitted over the network.
    • Man-in-the-Middle Attacks (Medium Severity): Reduces the risk of attackers intercepting and manipulating data in transit.
    • Data Tampering in Transit (Medium Severity): Encryption can provide integrity checks, reducing the risk of data modification during transmission.
    • Credential Theft in Transit (Medium Severity): Encrypting communication channels protects authentication credentials from being intercepted.

  • Impact:

    • Eavesdropping: High reduction in risk. Encryption makes data unreadable to eavesdroppers.
    • Man-in-the-Middle Attacks: Medium reduction in risk. TLS/SSL provides authentication and encryption, making man-in-the-middle attacks more difficult.
    • Data Tampering in Transit: Medium reduction in risk. Encryption can provide integrity checks to detect tampering.
    • Credential Theft in Transit: Medium reduction in risk. Encrypted channels protect credentials during transmission.

  • Currently Implemented:

    • HTTPS is enabled for NameNode and DataNode web UIs in the development environment using self-signed certificates.

  • Missing Implementation:

    • HTTPS is not enabled for ResourceManager and other Hadoop service UIs.
    • RPC encryption is not enabled for inter-node communication in any environment.
    • Production environment certificate management and deployment are not planned.
    • Proper certificate management and rotation processes are missing.

Mitigation Strategy: YARN Resource Quotas and Limits

  • Description:

    • Step 1: Define resource quotas and limits for YARN queues based on organizational units, projects, or user groups. Determine the maximum resources (CPU, memory, containers) that each queue can consume.
    • Step 2: Configure YARN Capacity Scheduler or Fair Scheduler to enforce resource quotas and limits. Set queue capacities, maximum capacities, and user/group limits in capacity-scheduler.xml or fair-scheduler.xml.
    • Step 3: Implement monitoring of YARN queue resource usage to track consumption against quotas and limits. Use YARN ResourceManager UI or monitoring tools to visualize resource usage.
    • Step 4: Set up alerts to notify administrators when queues approach or exceed their resource quotas or limits.
    • Step 5: Regularly review and adjust resource quotas and limits based on changing workload patterns and resource requirements.

  • List of Threats Mitigated:

    • Resource Exhaustion (High Severity): Prevents a single application or user from monopolizing cluster resources and starving other applications.
    • Denial of Service (DoS) (Medium Severity): Mitigates DoS attacks caused by malicious or poorly written applications consuming excessive resources.
    • Runaway Applications (Medium Severity): Limits the impact of runaway applications that might consume excessive resources due to bugs or misconfigurations.
    • Performance Degradation (Medium Severity): Ensures fair resource sharing and prevents performance degradation for other applications due to resource contention.

  • Impact:

    • Resource Exhaustion: High reduction in risk. Resource quotas and limits effectively prevent resource exhaustion by individual applications or users.
    • Denial of Service (DoS): Medium reduction in risk. Resource limits help mitigate resource-based DoS attacks.
    • Runaway Applications: Medium reduction in risk. Resource limits contain the impact of runaway applications.
    • Performance Degradation: Medium reduction in risk. Fair resource sharing improves overall cluster performance and prevents starvation.

  • Currently Implemented:

    • Basic YARN Capacity Scheduler is configured in the development environment with default queue settings.

  • Missing Implementation:

    • No specific resource quotas or limits are defined for queues or users.
    • Resource usage monitoring and alerting are not implemented.
    • Regular review and adjustment of resource quotas are not in place.
    • Production environment YARN resource management is not configured.

Mitigation Strategy: Hadoop Auditing

  • Description:

    • Step 1: Enable auditing for Hadoop services (HDFS, YARN, Hive, HBase, etc.). This is typically configured in service-specific configuration files (e.g., hdfs-site.xml, yarn-site.xml, hive-site.xml) by enabling audit logging and specifying audit log destinations (e.g., files, syslog, HDFS).
    • Step 2: Configure audit logging levels to capture relevant security events, such as authentication attempts, authorization decisions, data access operations, and administrative actions.
    • Step 3: Implement log rotation and retention policies for audit logs to manage log storage and ensure logs are retained for compliance and security investigations.
    • Step 4: Integrate Hadoop audit logs with a centralized logging and SIEM system for security monitoring, analysis, and alerting.
    • Step 5: Regularly review audit logs to detect suspicious activities, security breaches, and policy violations. Set up alerts for critical security events.

  • List of Threats Mitigated:

    • Undetected Security Breaches (High Severity): Auditing provides visibility into security events, enabling detection of breaches that might otherwise go unnoticed.
    • Lack of Accountability (Medium Severity): Audit logs provide a record of user actions, improving accountability and enabling investigation of security incidents.
    • Compliance Violations (Medium Severity): Auditing helps meet compliance requirements by providing auditable logs of security-relevant events.
    • Insider Threats (Medium Severity): Audit logs can help detect and investigate malicious activities by insiders.

  • Impact:

    • Undetected Security Breaches: High reduction in risk. Auditing significantly improves the ability to detect security breaches.
    • Lack of Accountability: Medium reduction in risk. Audit logs enhance accountability and facilitate incident investigation.
    • Compliance Violations: Medium reduction in risk. Auditing helps meet compliance requirements.
    • Insider Threats: Medium reduction in risk. Audit logs can aid in detecting and investigating insider threats.

  • Currently Implemented:

    • Basic audit logging is enabled for HDFS NameNode in the development environment, writing logs to local files.

  • Missing Implementation:

    • Auditing is not enabled for other Hadoop services (YARN, Hive, HBase).
    • Centralized logging and SIEM integration are missing.
    • Log rotation and retention policies are not defined.
    • Security monitoring and alerting based on audit logs are not implemented.
    • Production environment audit logging is not configured.

Mitigation Strategy: Hadoop Security Hardening Guides

  • Description:

    • Step 1: Obtain security hardening guides and best practices documentation for your specific Hadoop distribution and version. Consult documentation from your vendor (e.g., Cloudera, Hortonworks/Cloudera Data Platform, MapR) and the Apache Hadoop project.
    • Step 2: Review the hardening guides and identify applicable security configuration recommendations for your Hadoop environment.
    • Step 3: Implement the recommended security configurations across all Hadoop services and components. This may involve modifying Hadoop configuration files, operating system settings, and network configurations.
    • Step 4: Document all implemented hardening configurations and maintain a record of deviations from default settings.
    • Step 5: Regularly review and update hardening configurations to align with evolving security best practices and new Hadoop versions.

  • List of Threats Mitigated:

    • Misconfigurations (Medium Severity): Hardening guides help prevent common security misconfigurations that can leave Hadoop systems vulnerable.
    • Default Settings Vulnerabilities (Medium Severity): Hadoop's default settings may not be secure enough for production environments. Hardening guides address these default setting vulnerabilities.
    • Weak Security Posture (Medium Severity): Following hardening guides improves the overall security posture of the Hadoop cluster.
    • Compliance Issues (Medium Severity): Hardening guides often align with industry security best practices and compliance requirements.

  • Impact:

    • Misconfigurations: Medium reduction in risk. Hardening guides help avoid common misconfigurations.
    • Default Settings Vulnerabilities: Medium reduction in risk. Hardening addresses vulnerabilities inherent in default settings.
    • Weak Security Posture: Medium reduction in risk. Hardening improves overall security posture.
    • Compliance Issues: Medium reduction in risk. Hardening can help meet compliance requirements.

  • Currently Implemented:

    • Security hardening guides have not been formally reviewed or implemented. Some basic security configurations might be in place, but not based on a systematic hardening approach.

  • Missing Implementation:

    • No formal review of Hadoop security hardening guides has been conducted.
    • No systematic implementation of hardening recommendations is in place.
    • Documentation of hardening configurations is missing.
    • Regular review and update of hardening configurations are not planned.

Mitigation Strategy: Minimize Attack Surface

  • Description:

    • Step 1: Identify all Hadoop services and components running in your environment.
    • Step 2: Disable any Hadoop services or components that are not strictly necessary for your application or workload. For example, if you are not using HBase, disable HBase services.
    • Step 3: Restrict network access to Hadoop services to only authorized clients and networks. Use firewalls and network security groups to control inbound and outbound traffic to Hadoop ports.
    • Step 4: Limit user accounts and privileges to the minimum necessary for each user's role. Remove or disable unnecessary user accounts.
    • Step 5: Regularly review and audit running services, network access rules, and user accounts to ensure the attack surface remains minimized.

  • List of Threats Mitigated:

    • Unnecessary Service Exploitation (Medium Severity): Disabling unused services reduces the number of potential attack vectors.
    • Network-Based Attacks (Medium Severity): Restricting network access limits the ability of attackers to reach Hadoop services from unauthorized networks.
    • Lateral Movement (Medium Severity): Minimizing user privileges and accounts limits the potential for lateral movement within the Hadoop cluster if an account is compromised.
    • Accidental Exposure (Low Severity): Reducing the attack surface minimizes the risk of accidental exposure of sensitive services or data.

  • Impact:

    • Unnecessary Service Exploitation: Medium reduction in risk. Disabling services eliminates potential vulnerabilities in those services.
    • Network-Based Attacks: Medium reduction in risk. Network restrictions limit attack opportunities from external networks.
    • Lateral Movement: Medium reduction in risk. Least privilege and account minimization limit lateral movement.
    • Accidental Exposure: Low reduction in risk. Reduced attack surface minimizes accidental exposure.

  • Currently Implemented:

    • Basic firewall rules are in place to restrict access to Hadoop ports from outside the internal network in the development environment.

  • Missing Implementation:

    • No systematic review of running Hadoop services and components has been conducted to disable unnecessary services.
    • Network access restrictions are not finely tuned and might be overly permissive.
    • User account and privilege minimization has not been systematically implemented.
    • Regular attack surface reviews and audits are not in place.
    • Production environment attack surface minimization is not planned.