We are evaluating the use of Prometheus server and alert manager to push out alerts to either slack or teams. There is a scenario though we need to address. Following is a sample scenario that explains what exactly we are trying to achieve.
In terms of Azure, there are storage accounts in a subscription. There is a limit on the maximum number of storage accounts per subscription. Let us say there are 240 out of 250 maximum allowed storage accounts. The alerts are generated and delivered just as expected. However over a period of time if the amount reaches 250 and another subscription is created to handle more storage accounts then how does one stop delivering the alerts for the previous subscription which is full? The concept is similar to horizontal scaling. If we scale horizontally in terms of the counts, then how do we stop / inhibit alerts for previously full resource containers.
Look forward to discussing this further.