Why is prometheus constantly writing 100 MB/s to disk?

General Help/Support
1

Hi, I have a prometheus server running in a docker container that is constantly writing 100 MB/s to my hdd. Is this normal? I’m concerned that this constant writing to my hdd will shorten it’s life span / health. Should I be worried?

image
image678×572 50.1 KB

Thanks

2

That’s very difficult to answer as you’ve not given much information. How many targets are you scraping? How many metrics? What recording rules do you have? Do you have any remote read/write?

3

Here’s what my /etc/prometheus/prometheus.yml script looks like:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label job=<job_name> to any timeseries scraped from this config.
  - job_name: 'prometheus'

    static_configs:
      - targets: ['0.0.0.0:9101']
      - targets: ['10.0.0.154:9100']
      - targets: ['10.0.0.154:12798']
        labels:
          alias: 'block-producer-node'
          type:  'cardano-node'
      - targets: ['0.0.0.0:12799']
        labels:
          alias: 'relaynode1'
          type:  'cardano-node'

As far as I know, this is the only configurations I’ve done since installing prometheus. This docker container targets another docker container that has prometheus-node-exporter running.

Here’s my original post I made just recently that might provide a bit more context

4

SOLVED by uninstalling prometheus sudo apt-get purge prometheus and then re-installing.