Hello. Trying to use the sns_config configuration in order to send alerts to a AWS SNS topic. In doing so would like to replace the alertmanager-sns-forwarder, which we are currently using to send alerts to AWS SNS. Running into several issues around permissions trying to use sns_config. I am using prometheus-operator v0.54.2 (which supports sns_config) and alertmanager v0.24.0 (which should be the latest). My alertmanager.yml file has the following for the receiver for sns_config:
sns_configs:
- api_url: https://sns.us-east-1.amazonaws.com
topic_arn: arn:aws:sns:us-east-2:0987654321:alertTopic
sigv4:
region: us-east-1
role_arn: arn:aws:iam::0987654321:role/alertmanager_role
attributes:
- key: severity
value: SEV2
Upon deploying this, I get an error in the alertmanager logs that says that the role used by the node where alertmanager pod is running does not have permission to assume the alertmanager_role. We previously used KIAM, but replaced KIAM with IRSA (Iam Roles for Service Accounts). I attempted to modify alertmanager to use IRSA by creating a role, adding the role annotation to the service account, referencing the service account in the yaml, and adding the service account to the role trust policy. Upon deploying this code I get the following error:
InvalidAction: Could not find operation AssumeRoleWithWebIdentity for version 2011-06-15 status code: 400
For grins I tried manually adding SNS permissions to the node role, but now I get an error:
Invalid parameter: Subject"
I thought Subject was only for sending to email endpoints. I tried adding subject: ‘Test’ to the config, but not getting any alerts published to the topic. So question(s) is/are, does alertmanager not work with IRSA? Why do I need to pass a Subject? Any help resolving this would be appreciated.