Topic: Handling Missing HAProxy Logs in Prometheus Across Multiple Clusters
Context:
I’m currently working on monitoring HAProxy logs using Loki and Prometheus. I’ve created a recording rule in Loki to count the logs:
Loki Query:
sum by (hostname, k8s_cluster_name) (count_over_time({hostname=~".*haproxy.*"} |= "" | json [5m]))
This query aggregates the log counts over 5-minute intervals, grouped by hostname and k8s_cluster_name.
Prometheus Query:
avg_over_time(haproxylogs{}[15m])
This PromQL query calculates the average number of HAProxy logs over a 15-minute window.
Problem:
I want to handle scenarios where logs are not received for a particular cluster, and these missing logs should be represented as zero values in Prometheus. I found the absent function in Prometheus, which converts “no data” to zero values. However, the absent function only works for single clusters, not across multiple clusters.
Question:
How can I create a Prometheus query that handles missing HAProxy logs and shows zero values across multiple clusters?
I would appreciate any insights or suggestions on this topic. Thank you!