Scraping actuator on springboot up returns - remote error: tls: handshake failure

What did you do?
I have added scrape config for servers. FW are opened.

What did you expect to see?
Targets scraped without - remote error: tls: handshake failure.

What did you see instead? Under which circumstances?
image

Environment
Tested on prometheus 2.26.0,2.28.0,2.28.1 - kube-prometheus-stack helm chart

  • Prometheus configuration file:
       - job_name: "<NAME>"
          honor_timestamps: true
          scrape_interval: 30s
          scrape_timeout: 10s
          metrics_path: /actuator/prometheus
          scheme: https
          tls_config:
            cert_file: "/etc/prometheus/secrets/prometheus-client-cert/tls.crt"
            key_file: "/etc/prometheus/secrets/prometheus-client-cert/tls.key"
            ca_file: "/etc/prometheus/secrets/<CA>"
            insecure_skip_verify: false
          follow_redirects: true
          metrics_path: "/actuator/prometheus"
          scheme: https
          static_configs:
            - targets:
              - <server1>:6472
              - <server2>:6472
              - <server3>:6472      

Debug log shows only same problem as Prometheus UI.
I did try to use wget for scrape endpoint (not working), also curl from kubernetes node (works).
Also curl from my pc with same certificates works.

What other debug should be done ? I know curl is security issue in images, but there should be some option to test this connection more.

Also TLS ciphers offered by server matches possible TLS basic ciphers from go.

Testing ECDHE-RSA-AES256-GCM-SHA384…YES
Testing ECDHE-RSA-AES256-SHA384…YES
Testing ECDHE-RSA-AES256-SHA…YES
Testing DHE-RSA-AES256-GCM-SHA384…YES
Testing DHE-RSA-AES256-SHA256…YES
Testing DHE-RSA-AES256-SHA…YES
Testing ECDHE-RSA-AES128-GCM-SHA256…YES
Testing ECDHE-RSA-AES128-SHA256…YES
Testing ECDHE-RSA-AES128-SHA…YES
Testing DHE-RSA-AES128-GCM-SHA256…YES
Testing DHE-RSA-AES128-SHA256…YES
Testing DHE-RSA-AES128-SHA…YES

https://golang.org/pkg/crypto/tls/#pkg-constants

After doing :

nmap --script ssl-enum-ciphers -p 6472 <server1>

It does shows us that

Key exchange (dh 1024) of lower strength than certificate key

Which causes the issue .