Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alertmanager: Expermental pre-notification webhooks. #10718

Merged
merged 3 commits into from
Feb 26, 2025

Conversation

stevesg
Copy link
Contributor

@stevesg stevesg commented Feb 21, 2025

Adds an experimental feature where administrators can configure hooks to run before notifications are sent. This can be used for logging or auditing purposes, or for adding additional context to alert payloads. Everything here is subject to change.

Introduces a command line flag to enable and disable the code path entirely (-alertmanager.notify-hooks-enabled), and three -per-tenant overrides to configure the hook:

  • alertmanager_notify_hook_url: The URL of the service to call out to.
  • alertmanager_notify_hook_receivers: If blank, all notifications are passed to the hook, otherwise, only notifications are passed for receivers where the name is specified in this list.
  • alertmanager_notify_hook_timeout: Maximum time to wait for the hook to complete before performing the notification regardless. Note that failed hook invocations should never prevent notifications.

If we find this feature valuable and pursue it further, considerations will be taken allow more configuration of the HTTP client (e.g. TLS, basic auth). For now, this is omitted.

@stevesg stevesg force-pushed the stevesg/alertmanager-pre-notification-hooks branch from ea788a9 to bbea9f7 Compare February 21, 2025 21:26
Copy link
Contributor

github-actions bot commented Feb 21, 2025

💻 Deploy preview deleted.

@stevesg stevesg force-pushed the stevesg/alertmanager-pre-notification-hooks branch from bbea9f7 to 2ab834b Compare February 21, 2025 21:37
@stevesg stevesg force-pushed the stevesg/alertmanager-pre-notification-hooks branch from 2ab834b to 6633b39 Compare February 21, 2025 21:50
@stevesg stevesg marked this pull request as ready for review February 21, 2025 22:01
@stevesg stevesg requested review from a team and tacole02 as code owners February 21, 2025 22:01
Copy link
Contributor

@alexander-akhmetov alexander-akhmetov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

A question: do we want some default value for notify hook timeout instead of 0?

@stevesg stevesg merged commit dc137af into main Feb 26, 2025
30 checks passed
@stevesg stevesg deleted the stevesg/alertmanager-pre-notification-hooks branch February 26, 2025 21:30
stevesg added a commit that referenced this pull request Feb 27, 2025
* Alertmanager: Expermental pre-notification webhooks.

* Update timeout default and description

* Apply rate limits before hooks

(cherry picked from commit dc137af)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants