Add Stacktrace to Metrics, to find related source code lines

Exporters and Metrics
1

I have an complex application written in Go.

Unfortunately it does too many requests to a third party API.

We hit rate-limiting.

Now I want to know: Which source code lines produce the api calls.

We can provide a custom http client with a custom transport to count the
amount of api calls we do.

But the number of api-calls is not enough.

I would like to know which source code lines make the api call.

I could use debug.Stack to create a Stacktrace.

But how can I attach that string to the prometheus metrics?

If prometheus client is not the right tool for this job, which alternative solution do you think is better?

2

Are you wanting just aggregate counts of different calls, or the ability to interrogate a specific call?

Prometheus can help with the former, but logs & traces are the tools for specific details.

When you increment your counter (of number of API calls made) you can attach labels. Examples would be the name of the API call being made and the response code (which can then identify calls which have been rate limited). It is also common to include a label to identify what initiated the call (such as a subsystem name) - you’d then need to figure out how to obtain this. The simplest option would be to edit your code to pass a string to the code making the API call, but I’d imagine you could extract the caller details programmatically via stack traces, etc.

3

Thank you for your reply.

I cloud create the current stacktrace, create a hash sum and add this to the metric value.

Then I need some key value store to retrieve the stacktrace from the hash.

The cardinality of the stacktraces should be low.

Thank you for your answer!