I have a microk8s cluster running where the Observability add-on is enabled. So now i have a Prometheus instance. In the same cluster i have a Kafka consumer pod. I wrote a simple python script that runs in the Kafka pod reads a single value and then appends it to this endpoint “start_http_server(9001)”. The endpoint is only accessible within the cluster.
Now my question is how can I add additional Scrape Configuration to Prometheus so it could detect this port and read the data. I tried using this yaml file:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: my-pod-monitor
spec:
selector:
matchLabels:
name: consumer-pod
namespaceSelector:
matchNames:
- default
podMetricsEndpoints:
- interval: 5s
- port: ‘9001’
path: /metrics
but nothing workes, I am sceptical about the line "-port: ‘9001’ " but i don’t know how to fix the problem.
Also the Prometheus Operator has this log:
level=info ts=2023-05-15T16:05:26.620121568Z caller=operator.go:1162 component=prometheusoperator key=observability/kube-prom-stack-kube-prome-prometheus msg=“sync prometheus”
level=info ts=2023-05-15T16:05:26.625812751Z caller=operator.go:1330 component=prometheusoperator key=default/prometheus msg=“update prometheus status”
level=info ts=2023-05-15T16:05:26.626069483Z caller=operator.go:1357 component=prometheusoperator key=default/prometheus statefulset=default/prometheus-prometheus shard=0 msg=“not found”
level=info ts=2023-05-15T16:05:26.789965412Z caller=operator.go:1330 component=prometheusoperator key=observability/kube-prom-stack-kube-prome-prometheus msg=“update prometheus status”
level=info ts=2023-05-15T16:05:32.285240087Z caller=operator.go:1162 component=prometheusoperator key=default/prometheus msg=“sync prometheus”
level=error ts=2023-05-15T16:05:32.289598584Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="sync "default/prometheus" failed: creating config failed: loading additional scrape configs from Secret failed: secret additional-scrape-configs could not be found