Limit the recording rule memory usage

In my prometheus server, I have 2000-4000 recording rules calculated per 5 minutes.
When it is calculated, I can see a spike of Memory usage as below.

But I don’t need those recording rules to be calculated so fast, I want it to use less memory and take longer time to finish the calculation. Is there any parameter can control the memory usage of recording rules calculation? Or any metric can show the memory usage of recording rule calculation in prometheus? Can do this?
Thanks very much.

BR/Lei Yang

One more question, if I have 4k of recording rules, which way will use less memory?

  1. put all of them in one recording rule group.
  2. put them in 100 recording rule groups.

BR/Lei Yang

Each rule within a rule group is run sequentially. This means that it will run all the rules in a short period of time, which won’t get cleaned up for some number of seconds due to Go garbage collection parameters.

There is no way to control how much memory is used, as the memory is a function of the data being computed for each rule. The memory use (RSS) is managed by the Go garbage collection.

Splitting up rules into multiple groups will smooth out memory allocations since each rule group is an independent cycle. This, in theory, should reduce the spikiness of the memory use.

It’s also worth making sure your version of Prometheus is up to date with the latest releases. We have made many memory efficiency improvements in the last year.