We are working on one case where we want to collect data for CPU/Mem utilization during POD is starting, so that we can find the peak utilisation during service startup.
Is that possible using Prometheus? I am quite new to Prometheus, so if it is possible, kindly let me know how it can be achieved.
Thanks in advanced!
There is nothing particulary special about the startup of a pod. Metrics around the pod cpu/memory usage come from Kubernetes/cAdvisor, so you just need to look there. One thing to be aware of is the scrape frequency - if it is say only every 30 seconds then you might not see a pod until 30s after it has started and you might only get a single data point before it is fully running.
The good thing about counters (which I think the CPU metric usually is) is that it shows the usage up until that point, so it is easier to get an idea of usage even if you don’t scrape when a peak happens - although you can only get an average over a time period, rather than a peak value. For memory that is generally a gauge, so you only get to see how big it gets at the times the scrape happens, with no visibility around what was happening between scrapes.
Thanks @stuart for reply, one more query does Prometheus able to collect Database transaction related metrics too?
Prometheus can collect database related metrics, depending on the database. There are various exporters available for the major options, such as PostgreSQL & MySQL. Some databases even have native Prometheus metrics being exposed.