Scraping actuator on springboot up returns - remote error: tls: handshake failure

Prometheus server
1

What did you do?
I have added scrape config for servers. FW are opened.

What did you expect to see?
Targets scraped without - remote error: tls: handshake failure.

What did you see instead? Under which circumstances?
image

Environment
Tested on prometheus 2.26.0,2.28.0,2.28.1 - kube-prometheus-stack helm chart

  • Prometheus configuration file:
       - job_name: "<NAME>"
          honor_timestamps: true
          scrape_interval: 30s
          scrape_timeout: 10s
          metrics_path: /actuator/prometheus
          scheme: https
          tls_config:
            cert_file: "/etc/prometheus/secrets/prometheus-client-cert/tls.crt"
            key_file: "/etc/prometheus/secrets/prometheus-client-cert/tls.key"
            ca_file: "/etc/prometheus/secrets/<CA>"
            insecure_skip_verify: false
          follow_redirects: true
          metrics_path: "/actuator/prometheus"
          scheme: https
          static_configs:
            - targets:
              - <server1>:6472
              - <server2>:6472
              - <server3>:6472

Debug log shows only same problem as Prometheus UI.
I did try to use wget for scrape endpoint (not working), also curl from kubernetes node (works).
Also curl from my pc with same certificates works.

What other debug should be done ? I know curl is security issue in images, but there should be some option to test this connection more.

Also TLS ciphers offered by server matches possible TLS basic ciphers from go.

Testing ECDHE-RSA-AES256-GCM-SHA384…YES
Testing ECDHE-RSA-AES256-SHA384…YES
Testing ECDHE-RSA-AES256-SHA…YES
Testing DHE-RSA-AES256-GCM-SHA384…YES
Testing DHE-RSA-AES256-SHA256…YES
Testing DHE-RSA-AES256-SHA…YES
Testing ECDHE-RSA-AES128-GCM-SHA256…YES
Testing ECDHE-RSA-AES128-SHA256…YES
Testing ECDHE-RSA-AES128-SHA…YES
Testing DHE-RSA-AES128-GCM-SHA256…YES
Testing DHE-RSA-AES128-SHA256…YES
Testing DHE-RSA-AES128-SHA…YES

https://golang.org/pkg/crypto/tls/#pkg-constants

2

After doing :

nmap --script ssl-enum-ciphers -p 6472 <server1>

It does shows us that

Key exchange (dh 1024) of lower strength than certificate key

Which causes the issue .

3

I’m stuck with the same problem.
How did you solve it?
Prometheus error says “handshake failure”
I can’t see any error with nmap
I’ve tried with insecure_skip_verify: true but still failing

4

| ssl-dh-params:
| VULNERABLE:
| Diffie-Hellman Key Exchange Insufficient Group Strength
| State: VULNERABLE
| Transport Layer Security (TLS) services that use Diffie-Hellman groups
| of insufficient strength, especially those using one of a few commonly
| shared groups, may be susceptible to passive eavesdropping attacks.
| Check results:
| WEAK DH GROUP 1
| Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
| Modulus Type: Safe prime
| Modulus Source: RFC2409/Oakley Group 2
| Modulus Length: 1024
| Generator Length: 8
| Public Key Length: 1024
| References:
|_ https://weakdh.org

5

Solved
It’s a OpenJDK 11 bug
https://bugs.openjdk.java.net/browse/JDK-8211806

Disabling tls1.3 (or forcing tls1.2) in Spring boot then it works