Why Prometheus stack and adapter not fetching the custom metrics in GKE

Prometheus server Scaling / Clustering / Long-Term Storage
1

Hi Community,

As same as below document, we have implemented the custom metrics pod autoscaling in our project.

We have installed the Prometheus stack and adapter in our GKE cluster using helm charts.

Our application exposed the below metrics and we planned to implement the custom metrics HPA using this metrics in GKE cluster.

Metric name: http_server_requests_seconds_count

  • We have updated the below values in Prometheus stack values.yaml file and installed Prometheus stack using this values.yaml file.
additionalPrometheusRulesMap:
rule-name:
  groups:
   - name: sample-recordingrules
    rules:
     - record: sample_record_1m
      expr: (sum by (kubernetes_pod_name) (rate(http_server_requests_seconds_count{job="kubernetes-pods", app="sample-app", kubernetes_pod_name=~".*"}[1m])))
  • We have updated the below values in Prometheus adapter values.yaml file and installed Prometheus adapter using this values.yaml file.
rules:
default: true
custom:
- seriesQuery: 'sample_record_1m'
  seriesFilters: []
  resources:
   overrides:
    kubernetes_pod_name:
     resource: pod
    kubernetes_namespace:
     resource: namespace
  name:
   matches: "sample_record_1m"
   as: "sample_custom"
  metricsQuery: (predict_linear(sample_record_1m[1m],1*60))
  • We have updated the below values in annotation field in our application deployment yaml.
  prometheus.io/path: /sampleservices/actuator/prometheus
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"
  • We have updated the custom rules updated in Prometheus adapter in HPA yaml files in below.
metrics:
  - type: Pods
    pods:
      metric:
        name: sample_custom
      target:
        type: AverageValue
        averageValue: 17000m

Note:

  • We have implemented the same procedure in AWS EKS cluster, its working fine and based on custom metrics values the application scales up & down without any issues.
  • But, when we replicated the same procedure in GCP GKE cluster and its not working properly.
  • If we execute the below command to check the custom Prometheus list. But it’s not displayed any custom metrics list.
    kubectl get --raw “/apis/custom.metrics.k8s.io/v1beta1/” | jq . | grep custom

Request:

  • We are looking for any possible help based on the details shared on why Prometheus stack and adapter not fetching the metrics from application pods to Prometheus in GKE cluster?
  • Do we need to follow up any other extra steps if we implement on GKE cluster to fetch the custom metrics?