Prometheus Alertmanager


Reading time: less than 1 minute

Run Prometheus / PromQL queries, send emails based on query results (alert conditions). Also works with VictoriaMetrics if you use vmalert.

Ignore TLS errors

A TLS error from Alertmanager might look like this.

machine-001 alertmanager[5857]: ts=2023-02-16T08:13:15.195Z caller=notify.go:732 level=warn component=dispatcher receiver=mail-receiver integration=email[0] msg="Notify attempt failed, will retry later" attempts=1 err="send STARTTLS command: x509: cannot validate certificate for 10.10.10.35 because it doesn't contain any IP SANs"

The Alertmanager configuration file is documented here. It has a tls_config section.

In order to ignore TLS errors with Alertmanager, you need to add a tls_config section. Here’s what it might end up looking after you add the relevant config keys. Specifically, we’re interested in the insecure_skip_verify option here.

receivers:
  - name: 'mail-receiver'
    email_configs:
    - to: alerts@example.com
      from: alertsender@gmail.com
      send_resolved: true
      smarthost: 10.10.10.35:587
      tls_config:
        insecure_skip_verify: true

After adding the tls_config section, Alertmanager should be able to send email without complaining about TLS certificates.

Citation

If you find this work useful, please cite it as:
@article{yaltirakli,
  title   = "Prometheus Alertmanager",
  author  = "Yaltirakli, Gokberk",
  journal = "gkbrk.com",
  year    = "2024",
  url     = "https://www.gkbrk.com/prometheus-alertmanager"
}
Not using BibTeX? Click here for more citation styles.
IEEE Citation
Gokberk Yaltirakli, "Prometheus Alertmanager", November, 2024. [Online]. Available: https://www.gkbrk.com/prometheus-alertmanager. [Accessed Nov. 12, 2024].
APA Style
Yaltirakli, G. (2024, November 12). Prometheus Alertmanager. https://www.gkbrk.com/prometheus-alertmanager
Bluebook Style
Gokberk Yaltirakli, Prometheus Alertmanager, GKBRK.COM (Nov. 12, 2024), https://www.gkbrk.com/prometheus-alertmanager

Comments

© 2024 Gokberk Yaltirakli