metricRelabeling in Prometheus duplicates metrics

General Help/Support
1

Hi all,

I’ve asked this in other forums before but have not been able to solve this problem so apologies for the repost. I’m attempting to do a basic metric relabeling where I replace the __name__ by removing a prefix (specifically workload: ). This is taken directly from the upstream istio documentation. However, I am able to create a new metric without the workload prefix, but am totally unable to either replace nor drop the existing workload metric in order to save space. This is further confirmed when looking at the TSDB status endpoint of my main prometheus.

This is my ServiceMonitor definition:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: istio-federation
namespace: prometheus-system
labels:
app.kubernetes.io/name: istio-prometheus
spec:
namespaceSelector:
matchNames:
- istio-system
selector:
matchLabels:
release: prometheus-istio
endpoints:

  • interval: 30s
    scrapeTimeout: 30s
    params:
    ‘match’:
    • ‘{name=~“workload:(.*)”}’
      path: /federate
      targetPort: 9090
      honorLabels: true
      metricRelabelings:
    • sourceLabels: [“name”]
      regex: ‘workload:(.*)’
      targetLabel: “name”
      action: replace

Any help would be greatly appreciated!