Hi,
I have a custom node exporter from which prometheus scrapes every 30 seconds.
Issue is that sometimes -
a) if the data volume is large ,node exporter takes longer , On top of that ,
b) i need to clear the metrics to remove the old stale values before i set the metrics to the respective values again.
The scenario a) causes mixup of data from 2 iterations and in scenario b) scrape cycle may get 0 values .
Expected behaviour → Iteration 1 starts → Node exporter collects raw data → clear metrics → process data and set metrics -->Iteration 1 complete → Prometheus Scrapes Data.
Problem is : a) Iteration 1 starts → Node exporter collects raw data → clear metrics → ( Prometheus Scrapes Data ) → process data and set metric #1 → . process and set metric #N -->Iteration 1 complete
b) Iteration1 starts → Node exporter collects raw data → clear metrics → process data and set metric#1 → ( Prometheus Scrapes Data ) -->… process and set metric#n–> Iteration 1 complete
So, Is there a way to enforce some syncronization between node exporter and prometheus scrape cycle ? example-
Scenario 1) The node exporter prepared data in 20 seconds and Prometheus should start scraping from ( say) 30th second of every minute i.e.
02:34:30
Scenario 2) To expose all the metrics in one shot , example -
Iteration2 starts → Node exporter collects raw data → clear metrics → process data and set metric#1 → -->… process and set metric#n–> Iteration 2 complete → expose metric.
something like - metric1.labels(STATE=x).set (y)
metric1.expose()
I am okay to scrape old data , but dont want to -
mix Iteration 1 and Iteration 2’s data or ,
scrape the data when node exporter has cleared metric.
Scenario 3: make scrape wait till data collection iteration is complete.
Please do let me know if more information is required from my end here.
Regards
Puneet