Skip to content

Latest commit

 

History

History
120 lines (105 loc) · 12.8 KB

mitigations.md

File metadata and controls

120 lines (105 loc) · 12.8 KB

Mitigation Strategies Analysis for app-vnext/polly

Mitigation Strategy: Implement Rate Limiting for Retry Policies

  • Mitigation Strategy: Rate Limiting in Polly Retry Policies
  • Description:
    1. Define Retry Policy with Limits: When creating Polly retry policies using RetryPolicyBuilder or similar, explicitly set limits to the number of retries and the delay between retries.
    2. Set RetryCount: Use the RetryCount(int retryCount) method to define a maximum number of retry attempts. For example, policyBuilder.RetryCount(3).
    3. Implement Exponential Backoff with WaitAndRetry and maxDelay: Utilize WaitAndRetry or WaitAndRetryAsync and configure the sleepDurationProvider to use exponential backoff. Crucially, set a maxDelay to prevent unbounded delays. Example: policyBuilder.WaitAndRetryAsync(retryCount, attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)), maxDelay: TimeSpan.FromMinutes(1)).
    4. Integrate Circuit Breaker: Combine retry policies with a Polly circuit breaker policy. The circuit breaker will halt retries when the circuit is open, effectively limiting retries during outages.
  • Threats Mitigated:
    • Denial of Service (DoS) against Downstream Services (High Severity): Uncontrolled Polly retries can overload failing services.
    • Resource Exhaustion in Own Application (Medium Severity): Excessive Polly retries can consume application resources.
  • Impact:
    • DoS against Downstream Services: High reduction in risk by preventing retry storms initiated by Polly.
    • Resource Exhaustion in Own Application: Medium reduction in risk by limiting Polly's retry activity.
  • Currently Implemented: Implemented in the OrderService and PaymentService API calls using RetryPolicyBuilder with RetryCount set to 3 and exponential backoff with a max delay of 30 seconds. Configuration is in Startup.cs of each service.
  • Missing Implementation: Not yet implemented for background job processing in the BackgroundWorkerService. Polly retries in background jobs are currently unbounded.

Mitigation Strategy: Carefully Configure Circuit Breaker Thresholds and Durations

  • Mitigation Strategy: Polly Circuit Breaker Configuration Tuning
  • Description:
    1. Use CircuitBreakerPolicyBuilder: Implement circuit breaker policies using Polly's CircuitBreakerPolicyBuilder or AdvancedCircuitBreakerPolicyBuilder.
    2. Tune FailureThreshold and MinimumThroughput: Adjust the FailureThreshold (e.g., percentage of failures) and MinimumThroughput (minimum calls before considering failures) in the CircuitBreakerPolicyBuilder.
      • Example: policyBuilder.CircuitBreakerAsync(exceptionsAllowedBeforeBreaking: 5, durationOfBreak: TimeSpan.FromSeconds(30)). Or using percentage: policyBuilder.AdvancedCircuitBreakerAsync(failureThreshold: 0.5, samplingDuration: TimeSpan.FromSeconds(10), minimumThroughput: 10, durationOfBreak: TimeSpan.FromMinutes(1)).
    3. Tune BreakDuration: Set an appropriate BreakDuration (time the circuit remains open) in the CircuitBreakerPolicyBuilder.
    4. Monitor Polly Circuit Breaker State: Utilize Polly's OnCircuitBreakerOpen, OnCircuitBreakerClose, and OnHalfOpen delegates to log and monitor circuit breaker state changes.
  • Threats Mitigated:
    • Cascading Failures (High Severity): Poorly configured Polly circuit breakers might fail to prevent cascading failures.
    • Reduced Availability (Medium Severity): Overly sensitive Polly circuit breakers can lead to premature circuit breaks.
  • Impact:
    • Cascading Failures: High reduction in risk by effectively using Polly to isolate failures.
    • Reduced Availability: Medium reduction in risk through balanced Polly circuit breaker configuration.
  • Currently Implemented: Circuit breakers are implemented for all external API calls in ApiService classes using CircuitBreakerPolicyBuilder. Default thresholds are set to 20% failure rate and 10 seconds break duration. Configuration is in base ApiService class.
  • Missing Implementation: Thresholds and break durations in Polly circuit breakers are currently default values and haven't been specifically tuned for each downstream service.

Mitigation Strategy: Sanitize Polly Logging and Telemetry

  • Mitigation Strategy: Secure Polly Logging Configuration
  • Description:
    1. Review Polly Logging Delegates: Examine any logging delegates configured within Polly policies (e.g., OnRetry, OnBreak, OnHalfOpen).
    2. Sanitize Data in Logging Delegates: Within these delegates, ensure that sensitive data is not directly logged.
      • Filter Sensitive Parameters: Avoid logging entire request or response objects. Log only necessary information.
      • Mask Sensitive Data: If logging parameters, mask or redact any sensitive information before logging within the delegate.
      • Example: Instead of logging request.Body, log a summary or only non-sensitive headers.
    3. Use Structured Logging: If possible, use structured logging within Polly logging delegates to make logs easier to analyze and sanitize programmatically.
  • Threats Mitigated:
    • Information Disclosure (Medium Severity): Unsanitized Polly logs can expose sensitive data.
  • Impact:
    • Information Disclosure: Medium to High reduction in risk by preventing sensitive data from being logged by Polly.
  • Currently Implemented: Basic logging is enabled for Polly policies using Logger delegates in policy builders. Logs are written to application logs using Serilog.
  • Missing Implementation: Log sanitization within Polly logging delegates is not implemented. Polly logs might currently contain request details that could be considered sensitive.

Mitigation Strategy: Implement Exponential Backoff and Jitter in Polly Retry Policies

  • Mitigation Strategy: Exponential Backoff with Jitter in Polly Retries
  • Description:
    1. Use WaitAndRetry with sleepDurationProvider: Implement retry policies using Polly's WaitAndRetry or WaitAndRetryAsync.
    2. Exponential Backoff Calculation: Configure the sleepDurationProvider to calculate delays exponentially. Example: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)).
    3. Introduce Jitter with Randomness: Add jitter to the backoff delay by incorporating a random element within the sleepDurationProvider. Example: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)) + TimeSpan.FromMilliseconds(new Random().Next(0, 1000)).
  • Threats Mitigated:
    • Retry Storms (High Severity): Fixed backoff in Polly retries can lead to retry storms.
    • Increased Downstream Service Load (Medium Severity): Synchronized Polly retries can overload downstream services.
  • Impact:
    • Retry Storms: High reduction in risk by using Polly to prevent synchronized retries.
    • Increased Downstream Service Load: Medium reduction in risk by smoothing out Polly retry traffic.
  • Currently Implemented: Exponential backoff is implemented in OrderService and PaymentService Polly retry policies.
  • Missing Implementation: Jitter is not currently implemented in Polly retry policies.

Mitigation Strategy: Align Polly Policies with Security Context

  • Mitigation Strategy: Security-Aware Polly Policy Design
  • Description:
    1. Contextual Policy Application: When applying Polly policies using PolicyWrap or individual policy application, consider the security context of the operation.
    2. Re-authentication/Re-authorization Logic in Polly Delegates: Within Polly's ExecuteAndCaptureAsync or similar methods, and potentially within OnRetry delegates, incorporate logic to re-validate authentication tokens or re-perform authorization checks before retrying security-sensitive operations.
    3. Conditional Policy Application: Use conditional policy application based on the type of operation or resource being accessed. Apply more restrictive or security-focused Polly policies to sensitive operations.
  • Threats Mitigated:
    • Bypassing Security Controls (Medium Severity): Polly retries might inadvertently bypass security checks.
    • Unauthorized Access (Medium Severity): Uncontrolled Polly retries without re-authorization could lead to unauthorized access.
  • Impact:
    • Bypassing Security Controls: Medium reduction in risk by ensuring Polly respects security boundaries.
    • Unauthorized Access: Medium reduction in risk by incorporating re-authentication/re-authorization into Polly usage.
  • Currently Implemented: Basic authentication and authorization are in place for API endpoints. Polly policies are applied to API calls but without explicit consideration for re-authentication/re-authorization within Polly itself.
  • Missing Implementation: Explicit re-authentication/re-authorization logic within Polly retry policies or execution context is missing.

Mitigation Strategy: Regularly Review and Audit Polly Configurations

  • Mitigation Strategy: Polly Configuration Audits and Reviews
  • Description:
    1. Schedule Polly Policy Reviews: Establish a schedule for periodic reviews of all Polly policy configurations in the application code.
    2. Document Polly Policies: Maintain documentation of each Polly policy, its purpose, and configuration parameters.
    3. Version Control Polly Configurations: Manage Polly policy definitions as code under version control to track changes and enable audits.
    4. Include in Code Reviews: Ensure Polly policy configurations are reviewed as part of standard code review processes.
  • Threats Mitigated:
    • Misconfigurations (Medium Severity): Polly misconfigurations can weaken resilience or introduce vulnerabilities.
    • Outdated Policies (Low Severity): Polly policies might become ineffective over time if not reviewed.
  • Impact:
    • Misconfigurations: Medium reduction in risk by proactively identifying and correcting Polly misconfigurations.
    • Outdated Policies: Low reduction in risk by ensuring Polly policies remain relevant.
  • Currently Implemented: Polly configurations are defined in code within service projects and are subject to basic code reviews.
  • Missing Implementation: No formal scheduled reviews or audits specifically focused on Polly configurations are in place. Dedicated documentation of Polly policies is limited.

Mitigation Strategy: Consider Bulkhead Isolation with Polly

  • Mitigation Strategy: Polly Bulkhead Isolation for Critical Operations
  • Description:
    1. Identify Critical Operations for Bulkheads: Determine operations that would benefit from bulkhead isolation using Polly.
    2. Implement BulkheadPolicy: Apply Polly's BulkheadPolicy or BulkheadPolicyAsync to critical operations using PolicyWrap or individual policy application.
    3. Configure MaxParallelization and MaxQueuingActions: Set appropriate values for MaxParallelization (maximum concurrent executions) and optionally MaxQueuingActions (maximum queued requests) in the BulkheadPolicyBuilder. Example: policyBuilder.BulkheadAsync(maxParallelization: 5, maxQueuingActions: 10).
    4. Monitor Polly Bulkhead Metrics: Utilize Polly's bulkhead policy events or integrate with monitoring systems to track bulkhead usage and performance.
  • Threats Mitigated:
    • Resource Exhaustion (Medium Severity): Uncontrolled concurrency can lead to resource exhaustion, which Polly bulkheads can mitigate.
    • Impact of Failures on Unrelated Operations (Medium Severity): Polly bulkheads can isolate failures and prevent them from spreading.
  • Impact:
    • Resource Exhaustion: Medium reduction in risk by using Polly to limit concurrency for critical operations.
    • Impact of Failures on Unrelated Operations: Medium reduction in risk by using Polly to isolate critical operations.
  • Currently Implemented: Bulkhead isolation using Polly is not currently implemented in the project.
  • Missing Implementation: Polly bulkhead policies should be considered for critical operations like payment processing and order placement.