Hey I’m trying to explore methods to prevent data loss when the endpoint is down so i’m using remote_write_queue with prometheus remote write exporter in opentelemetry in gateway mode on a Virtual Machine. My endpoint is a Victoria Metrics database on a LKE cluster.
Config used:
.
.
.
processors:
batch:
send_batch_size: 10
timeout: 5s
send_batch_max_size: 15
memory_limiter:
check_interval: 1s
limit_percentage: 90
spike_limit_percentage: 16
exporters:
logging:
verbosity: normal
prometheusremotewrite/1:
endpoint: "http://victoriametrics.com/insert/0/prometheus/"
remote_write_queue:
enabled: true
num_consumers: 2
queue_size: 20
service:
pipelines:
metrics/1:
receivers: [otlp, prometheus,hostmetrics]
processors: [batch,memory_limiter,metricstransform]
exporters: [logging,prometheusremotewrite/1]
These are the set of processors and exporters that i am using. When the Victoria Metrics endpoint is down, the queue size should increase right? However, I haven’t seen any increase in the queue size.
Please let me know under what cases or config will I be able to see the remote write queue size increase.
Thank you