---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: istio-ingressgateway-remote
namespace: istio-system
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: istio-ingressgateway-remote
endpoints:
- port: metrics
path: /stats/prometheus
metricRelabelings:
- action: drop
regex: '^unknown(.*)'
sourceLabels: [destination_service]
What I’m try to do is dropping those metrics from istio which have destination_service=“unkonwn”, but after I apply those configs and make sure they already available via prometheus console, check below prometheus configs snippets
What did you expect to see?
What I expect is after I restart istio pod, and simulate some 404 requests ,there should be no related metrics (eg istio_requests_total)scraped by promtheus .
What did you see instead? Under which circumstances?
I can still see metrics with label destination_service=“unknown” scraped
Prometheus config file
- job_name: serviceMonitor/istio-system/istio-ingressgateway-remote/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /stats/prometheus
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [job]
separator: ;
regex: (.*)
target_label: __tmp_prometheus_job_name
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
separator: ;
regex: (istio-ingressgateway-remote);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
metric_relabel_configs:
- source_labels: [destination_service]
separator: ;
regex: ^unknown(.*)
replacement: $1
action: drop
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
namespaces:
names:
- istio-system