Crons: Detect ingestion outages during clock ticks #79328
Comments
Part of GH-79328 --------- Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com>
Part of GH-79328 --------- Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com>
This will be used to inform the clock tick tasks that the tick detected an abnormal amount of check-in volume for the previous minute. Part of getsentry/sentry#79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
This will be used to inform the clock tick tasks that the tick detected an abnormal amount of check-in volume for the previous minute. Part of getsentry/sentry#79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
This will be used to inform the clock tick tasks that the tick detected an abnormal amount of check-in volume for the previous minute. Part of getsentry/sentry#79328
When a click tick is marked as having an abnormal volume we may have lost check-is that should have been processed during this minute. In this scenario we do not want to notify on misses, and instead should create them as unknown misses. Part of getsentry/sentry#79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
This adds a function `_evaluate_tick_decision` which looks back at the last MONITOR_VOLUME_RETENTION days worth of history and compares the minute we just ticked past to that data. We record 3 metrics from this comparison - `z_value`: This is measured as a ratio of standard deviations from the mean value - `pct_deviation`: This is the percentage we've deviated from the mean - `count`: This is the number of historic data points we're considering The z_value and pct_deviation will be most helpful in making our decision as to whether we've entered an "incident" state or not. Part of #79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
This will be used to pass around the result of anomaly detection during clock ticks. Part of #79328
When a click tick is marked as having an abnormal volume we may have lost check-is that should have been processed during this minute. In this scenario we do not want to notify on misses, and instead should create them as unknown misses. Part of getsentry/sentry#79328
This will be used to pass around the result of anomaly detection during clock ticks. Part of #79328
This task will be triggered when we detect an anomaly in check-in volume during a clock tick. When this happens we are unable to know that all check-ins before that tick have been received and will need to mark all in-progress monitors as resulting in a 'unknown' state. Part of getsentry/sentry#79328
Part of getsentry/sentry#79328 This creates the new topic necessary for delaying issue occurrences so that we can delay creation of notifications in the case where we detect an anomalous system incident.
Part of getsentry/sentry#79328 This creates the new topic necessary for delaying issue occurrences so that we can delay creation of notifications in the case where we detect an anomalous system incident.
Part of getsentry/sentry#79328 This creates the new topic necessary for delaying issue occurrences so that we can delay creation of notifications in the case where we detect an anomalous system incident.
Part of getsentry/sentry#79328 This creates the new topic necessary for delaying issue occurrences so that we can delay creation of notifications in the case where we detect an anomalous system incident.
Part of getsentry/sentry#79328 This creates the new topic necessary for delaying issue occurrences so that we can delay creation of notifications in the case where we detect an anomalous system incident.
|
Hey @swanson thank you for the feedback! That's our goal, to complete eliminate these misleading alerts and regain your trust in our product. I wanted to follow up on this bullet point:
Today you are able to configure the alerting thresholds on a per-monitor basis. Can you clarify if these consecutive failures/successes would be enough for your use case, or if you're referring to something else? |
I was imagining something like a toggle field per-monitor for "Don't alert on unknown status" that would allow us to let some monitors opt-in to behavior changes related to service degradation. The current consecutive failures helps, but is not sufficient and is not that legible (we would want it to alert immediately, just that service degradation is so often the cause that we have it as a workaround) |
Since I'll be adding a `clock_tick` argument to `mark_failed` it was
going to start becoming confusing what the `ts` argument in
`mark_failed` means.
This updates mark_{ok,failed} to use more appropriate names for what the
timestamp represents
Refactoring as part of GH-79328
|
Hey thanks for the comments on this @swanson!
This is an approach we considered, but this also has it's drawbacks since it means that system needs to be highly fault resistant itself, otherwise we risk a false positive of that trace not coming through causing true misses / time-out's not to be sent out. The new approach we have that delays sending notifications when the system detects an anomaly in check-in volume should work in a similar way, but will have the ability to self correct based on
This is super helpful feedback since I was trying to weight the added complexity of trying to mark check-ins as unknown once we "think" we're in an incident, and then if we aren't go back and correctly mark them as missed / timeout status. I think we probably won't do that, but we will mark incorrectly generated misses / timeouts as unknown once we know we are in an incident. Thanks again for the feedback! |
By default we won't alert if there is a system incident. We considered an option that still sends notifications in the case that we think we're producing false positive misses (for example, if you have a job that is so critical that you want to know if we think it could be down). But I think the need for this is going to be pretty rare, so we're probably not going to build this (at least not right away). |
This function returns a DecisionResult which encapsulates the TickAnomalyDecision and AnomalyTransition values for a particular clock tick. In the future this logic will be run at each clock tick and the result will later be used to decide if we can process issue occurrences in the incident_occurrences consumer for a specific clock tick. Part of GH-79328
There is a failure scenario that can be very disruptive for our customers.
If we have an outage in our ingestion of cron check-ins, specifically where we are dropping check-ins, then we may incorrectly mark customers cron monitors as having missed check-ins. This only happens if we drop check-ins, if we are delayed with check-ins, the clock ticks which drive missed and time-out detections will slow down to match the consumption of check-ins in our topic.
This is highly problematic as it means customers are unable to trust that cron alerts are accurate. This is, however, a difficult problem, since if check-ins never make it into the check-ins topic, how can we differentiate between a customers job failing and not sending a check-in, and us failing to ingest their check-in?
In most of our ingestion failure scenarios we have had a significant drop in check ins. That typically looks something like this:
Improved behavior
If we were able to detect this extreme drop in volume, we could create clock ticks that are marked as being
unknownticks. meaning we are moving the clock forward, but we have a high certainty that we may have lost many check-ins. When this happens instead of creatingmissedandtimed-outcheck-ins that trigger alerts, we can create missed check-ins that have a "unknown" status and mark in-progress check-ins that are past their latest check-in time as "unknown", again not alerting customers. Once we are certain that we have recovered from our incident, the clock will resume producing ticks that are regular, and not marked as unknown.Detecting ingestion volume drops
The tricky part here is deciding if we are in an incident state. Ideally we are not relying on an external service telling us that we may be in an incident, since that service itself may be part of the incident (eg, if we had relay report that it was having problems, there's no guarantee that when it's having problems it will just fail to report to us).
My proposed detection solution is rather simple. As we consume check-ins, we keep a bucket for each minute worth of check-ins, that bucket is a counter of how many check-ins were consumed for that minute. We will keep these buckets for 7 days worth of data, that's 10080 buckets.
Each time the clock ticks across a minute, we will look at the last 7 days of that particular minute we ticked over, take some type of average of those 7 counts, and compare that with the count of the minute we just ticked past. If we find that the count here is some percentage different from the previous 7 days of that minute, we will produce our clock tick with a "unknown" marker, meaning we are unsure if we collected enough data for this minute and are likely in an incident. In which case we will create misses and time-outs as "unknown".
Ignoring previous incidents
When a minute was detected as having a abnormally low volume we should reset it's count to some sentinel value like
-1so that when we pick up this minute the next 7 days, we know to ignore the data, since it will not be an accurate representation of typical volume.Not enough data
Warning
What should we do if we don't have enough data to determine if the past minute is within the expected volume?
Implementation
We should start by implementing this as a simple metric that we track, so we can understand what our typical difference looks like each day. It's possible some days may have many more check-ins, such as Monday's ad midnight. So maybe we will need a different way to evaluate anomalies.
Warning
The implementation described above has changed. See the comment later in this issue for a description of the new approach
Implementation
PRs needed for both approaches
mark_unknownclock task sentry-kafka-schemas#340mark_unknownclock tick task #79735mark_unknown#79785Outdated PRs for previous approach
volume_anomaly_resultto mark_missing task sentry-kafka-schemas#341volume_anomaly_result#79729PRs needed for new approach
make_clock_tick_decision#80640The text was updated successfully, but these errors were encountered: