I am a student trying to figure out the scrape performance of Prometheus.
Here is what I did.There are 100 VM with Node-exporter and cAdvisor in my server. Scrape_interval and scrape_timeout are setting as 1 second. I run the script “curl localhost:9090/api/v1/query?query=scrape_duration_seconds > vm100_duration.txt”. And the scrape durations about one target, such as “xxx.xxx.xxx.xx:8080”(cadvisor) or “xxx.xxx.xxx.xx:9100”(node-exporter), are from 0.025002347 to 0.671868273.
There are some questions about Promehteus in my mind. How to get the total duration to pull the 200 targets? What’s the meaning of scrape_duration, just from sending pull-order to receieving value? Should I just sum(scrape_duration_seconds) ?And how to set scrape_interval and scrape_timeout by the target num and scrape_duration_seconds?
Thanks a lot!