I have a weird issue with Prometheus using the community helm chart.
I’m deploying it fine, it runs, I can scrape metrics endpoints for some other services over HTTPS, have gotten services working with HTTPS/TLS, however I keep getting my logs spammed with a TLS handshake error every 5ish seconds after Prometheus boots up. Each request also appears to be from a different IP address. Some examples of the error are:
time=2025-01-14T02:57:05.864Z level=ERROR source=server.go:3487 msg="http: TLS handshake error from 127.0.0.1:54256: client sent an HTTP request to an HTTPS server" component=web
time=2025-01-14T02:57:10.868Z level=ERROR source=server.go:3487 msg="http: TLS handshake error from 127.0.0.1:60034: client sent an HTTP request to an HTTPS server" component=web
time=2025-01-14T02:57:15.865Z level=ERROR source=server.go:3487 msg="http: TLS handshake error from 127.0.0.1:60038: client sent an HTTP request to an HTTPS server" component=web
time=2025-01-14T02:57:20.865Z level=ERROR source=server.go:3487 msg="http: TLS handshake error from 127.0.0.1:35652: client sent an HTTP request to an HTTPS server" component=web
time=2025-01-14T03:06:05.876Z level=ERROR source=server.go:3487 msg="http: TLS handshake error from 127.0.0.1:54826: EOF" component=web
Something is apparently making non-stop HTTP requests to my prometheus server, but I can’t figure out what. I’ve tried narrowing down the culprits by:
- Removing all health probes.
- Removing all scraping targets.
- Removed Prometheus as a data source in Grafana.
I’m not sure what else could be making this request at this point. I thought it was for sure the probes, but I’ve set them to null within the templates and I’m still getting the errors. I’m at the end of my rope! Any ideas?