Hi everyone,
I have been using prometheus blackbox exporter operator for some months. I have thousands of websites I need to monitor. I created a ServiceMonitor for each website. Initially, there were around 2000 websites. Then slowly the number increased to somewhere between 4000-5000. Now, with each new website, a new service monitor was created.
I ran into issues with the blackbox exporter with such a scale. There were periods when no data was recorded for websites. Is blackbox exporter built for this kind of scale?
I want to know has anyone used blackbox exporter with this kind of scale? If yes, what recommendations do you have for me regarding the desing/architectural aspects?
I was using grafana agent to collect the metrics from the exporter and then forwarding it to the prometheus server.
Thank you.