Monitoring infra should alert when an expected collector can't be scraped #251

New issue

Open

opened 2025-08-05 14:59:32 +00:00 by delroth · 1 comment

delroth commented

2025-08-05 14:59:32 +00:00

Owner

When I deployed the smartctl exporter on bm-12 it at first couldn't start due to disk brokenness, then was broken due to a bug in the collector. In neither case did this show up anywhere in the monitoring, and the absence of metrics could be hiding other conditions. We should figure out how to bubble up the fact that the local agent can't scrape a target and alert on this.

raito commented

2025-08-05 15:02:30 +00:00

Owner

              {
                alert = "FailedScrape";
                labels.severity = "warning";
                annotations.summary = "Scrape failed";
                annotations.description = "The job was not successfully scraped";
                for = "2m";
                expr = ''
                  up == 0
                '';
              }

could do the job

``` { alert = "FailedScrape"; labels.severity = "warning"; annotations.summary = "Scrape failed"; annotations.description = "The job was not successfully scraped"; for = "2m"; expr = '' up == 0 ''; } ``` could do the job