I’m fairly new to prometheus, I have a basic understanding on how to collect metrics and display visualizations in Grafana. I have the current use case and was wondering if there was some guidance on how this would be handled.
Problem
Similar to http_request_duration, we have events that we’d like to track the time duration of. We’ve created one histogram for this time duration.
As part of processing each event, this could cause our application to evaluate n documents stored in a database. For this, I was thinking that I’d add a 2nd histogram with the buckets corresponding to the document count that was evaluated for the event. I was planning on an exponential bucket size doubling between 100-12000 (around 8 buckets total).
What I’d like to see is a visualization that shows event duration time and visually correlates it with the number of documents that were evaluated for each event. Maybe something like this?
Question
Given the problem statement above, is two histograms the best way to model this? Or could the same be done with a gauge & histogram? Or am I looking at this the wrong way and there’s a better way to visualize things?