How to get istio metrics

I am running istio pods via kubernetes is the “istio-system” namespace.
I’m trying to scrape istiod so I can get some metrics such as “istio_requests_total”

My prometheus pod is running on the monitoring namespace. I’m configuring the jobs to run like this:

       - job_name: 'istiod'

          scrape_interval: 5s

          kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              names:
              - istio-system

          relabel_configs:
          - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
            action: keep
            regex: istiod;http-monitoring

I’m not sure what I am missing

Istio uses a non-standard metric path. Istio sidecar injection automatically adds the following annotations to your pods, which should be all you need to start scraping istio’s metrics (assuming prometheus was installed in k8s using the community helm chart).

  "prometheus.io/path": "/stats/prometheus",
  "prometheus.io/port": "15020",
  "prometheus.io/scrape": "true",