Hi,
I’m after some assistance on what I’m hoping is most likely a simple fix to an issue I have with Blackbox http requests. I have installed Blackbox Exporter and Prometheus for the first time and have setup a few targets in prometheus.yml. One being https ://www.youtube.com which I get a success from as well as https ://prometheus.io.
However I am trying to probe the local host and another server on the same subnet using http but both return errors.
The errors I am seeing for both are
LocalHost Log
Logs for the probe:
ts=2021-04-21T16:21:40.017443757Z caller=main.go:304 module=http_2xx target=http:// localhost level=info msg=“Beginning probe” probe=http timeout_seconds=5
ts=2021-04-21T16:21:40.017610381Z caller=http.go:342 module=http_2xx target=http:/ /localhost level=info msg=“Resolving target address” ip_protocol=ip4
ts=2021-04-21T16:21:40.017660672Z caller=http.go:342 module=http_2xx target=http:/ /localhost level=info msg=“Resolved target address” ip=127.0.0.1
ts=2021-04-21T16:21:40.017720794Z caller=client.go:252 module=http_2xx target=http:/ /localhost level=info msg=“Making HTTP request” url=http://127.0.0.1 host=localhost
ts=2021-04-21T16:21:40.01832259Z caller=main.go:119 module=http_2xx target=http:// localhost level=info msg=“Received HTTP response” status_code=401
ts=2021-04-21T16:21:40.01835153Z caller=main.go:119 module=http_2xx target=http:// localhost level=info msg=“Invalid HTTP response status code, wanted 2xx” status_code=401
ts=2021-04-21T16:21:40.018431942Z caller=main.go:119 module=http_2xx target=http:/ /localhost level=info msg=“Response timings for roundtrip” roundtrip=0 start=2021-04-21T16:21:40.017794856Z dnsDone=2021-04-21T16:21:40.017794856Z connectDone=2021-04-21T16:21:40.018020239Z gotConn=2021-04-21T16:21:40.018048648Z responseStart=2021-04-21T16:21:40.018290128Z end=2021-04-21T16:21:40.018423161Z
ts=2021-04-21T16:21:40.018461986Z caller=main.go:304 module=http_2xx target=http:// localhost level=error msg=“Probe failed” duration_seconds=0.000951985
and the remote server Log (redacted public IP)
Logs for the probe:
ts=2021-04-21T19:49:50.803022341Z caller=main.go:304 module=http_2xx target=http:// xx.xx.xx.xx:8080 level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2021-04-21T19:49:50.803159954Z caller=http.go:342 module=http_2xx target=http://x x.xx.xx.xx:8080 level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-04-21T19:49:50.803182958Z caller=http.go:342 module=http_2xx target=http://xx .xx.xx.xx:8080 level=info msg="Resolved target address" ip=xx.xx.xx.xx
ts=2021-04-21T19:49:50.803250429Z caller=client.go:252 module=http_2xx target=http://xx.xx .xx.xx:8080 level=info msg="Making HTTP request" url=http:// xx.xx.xx.xx:8080 host=xx.xx.xx.xx:8080
ts=2021-04-21T19:49:50.803711407Z caller=main.go:119 module=http_2xx target=http://xx.xx.xx.xx:8080 level=error msg="Error for HTTP request" err="Get \"http://xx.xx.xx.xx:8080\": dial tcp xx.xx.xx.xx:8080: connect: connection refused"
ts=2021-04-21T19:49:50.803744373Z caller=main.go:119 module=http_2xx target=http://xx.xx.xx.xx:8080 level=info msg="Response timings for roundtrip" roundtrip=0 start=2021-04-21T19:49:50.803334267Z dnsDone=2021-04-21T19:49:50.803334267Z connectDone=2021-04-21T19:49:50.803687729Z gotConn=0001-01-01T00:00:00Z responseStart=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2021-04-21T19:49:50.803797415Z caller=main.go:304 module=http_2xx target=http://xx.xx.xx.xx:8080 level=error msg="Probe failed" duration_seconds=0.000732487
prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090', 'localhost:9115']
- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['xx.xx.xx.xx:9100']
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- http:// localhost
- http:// xx.xx.xx.xx:8080
- https://www. youtube .com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115
and blackbox.yml
modules:
http_2xx:
prober: http
timeout: 5s
http:
method: GET
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
# valid_http_versions:
# - HTTP/1.1
# - HTTP/2
# valid_status_codes: []
preferred_ip_protocol: "ip4" #defaults to "ipv6"
ip_protocol_fallback: false # no fallback to "ipv6"
http_post_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
pop3s_banner:
prober: tcp
tcp:
query_response:
- expect: "^+OK"
tls: true
tls_config:
insecure_skip_verify: false
ssh_banner:
prober: tcp
tcp:
query_response:
- expect: "^SSH-2.0-"
irc_banner:
prober: tcp
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp:
prober: icmp
both local host and the remote server are on AWS and have the local firewalls turned off. I believe I have enabled the correct ports in the respective AWS security Groups
Any pointers would be greatly appreciated
thanks
p.s. added some spaces in the url’s to allow posting of links