Scraping Error after retrieving metrics via browser / curl

Exporters and Metrics
1

Hello together!
a couple of days i am trying to figure out a problem dealing with .NET Metrics scraped by Prometheus and delivered to Grafana with a project which should be eneabled to send OpenTelemetry Data.

Moving from my project to a general ASP.NET Core Otel-Example (opentelemetry-dotnet/examples/AspNetCore at main · open-telemetry/opentelemetry-dotnet · GitHub) which is configured the same as my project
i tried the mentioned example, and configured prometheus as follows:

- job_name: 'OpenTelemetry_OriginalTestProjekt'
    scrape_interval: 5s
    scheme: https
    static_configs:
      - targets: ['localhost:5001']

Releasing some GET-Requests via Swagger, metrics are created, scraped by prometheus and shown in Grafana.

BUT:

Issueing a request to see the scraped data that also prometheus sees results in an error message:

ts=2024-04-24T09:49:10.481Z caller=scrape.go:1345 level=debug component="scrape manager" scrape_pool=OpenTelemetry_OriginalTestProjekt target=https://localhost:5001/metrics msg="Append failed" err="expected label value, got \"#\" (\"INVALID\") while parsing: \"kestrel_active_connections{otel_scope_name=\\\"Microsoft.AspNetCore.Server.Kestrel\\\",network_transport=\\\"tcp\\\",network_type=#\""

Afterwards the errormessage appears again and again and no more metrics are scraped anymore. As a consequence no new data is povided to grafana.

After reopening the Example prometheus reconnects and scrapes again data.

What is the problem here or what is the reason prometheus is delivering this error and stops scraping?

2

The error is caused by the fact that the format is not what prometheus is expecting that each label withing the curly braces is in the form label_name="<text>", but for network type that is not the case. It is seeing a hash rather than double quotes: network_type=#.

What is causing that to be the case, I cannot tell you though. Do you have the whole line being parsed?

3

Hi, thanks for replying!

The line which should be parsed by prometheus is the following:

# TYPE kestrel_active_connections gauge
# HELP kestrel_active_connections Number of connections that are currently active on the server.
kestrel_active_connections{otel_scope_name="Microsoft.AspNetCore.Server.Kestrel",network_transport="tcp",network_type="ipv6",server_address="::1",server_port="5174"} 4 1714031208626

According to the message there is something wrong parsing the ipv6 part. But why?

4

Hi @securom87

Did you manged to resolved this issue? As i am experiencing the exact same error as you.