I am very new to the Prometheus and this is my first posting in this forum.
However, I am skilled C++ and Java developer, specializing on the back-end, with over a quarter of century experience. Unfortunately, I haven’t done any GO development, but willing to learn it if needed.
We have a extensive set of APIs running on a Tomcat under RHEL8.
We do need to collect those API metrics, i.e. collect timeseries of API request/response with URL matching pattern that will contain HTTP method (GET, POST, PUT, …), HTTP response code (20x, 500, 40x, etc.) and duration of the call in milliseconds. This is the least what we need for the beginning. Later we might need to add headers and request parameters.
I have browsed Prometheus documentation but wasn’t able to find any exporters that would do something like what we need.
Therefore my questions are:
Does such exporter exist? If so, can you, please, provide links to it?
If it does not exist, how to create it, preferably in Java? (I haven’t done any C++ coding for 10+years). I was thinking about parsing CATALINA log files, but may be you can provide better idea. Unfortunately, in the docs I didn’t see any topics about custom exporters.
How to handle horizontal scaling, when we’ll have several boxes running same APIs?
Any helpful tips and pointers are very appreciated.
Ideally you’d directly instrument your application. You can then add exactly the metrics you are wanting. In this case you’d generally have a total count of requests & a total duration of those requests (possibly split into multiple request duration buckets via a histogram). You can then calculate how average/quartile request durations change over time. With direct instrumentation you can also add exemplars to give you real examples of slow requests, etc.
Alternatively you could use the JMX exporter which just allows you to expose existing metrics that might already be being created by the libraries you are using. If the only thing you have are log files take a look at something like mtail, which reads log files and rung “mini programs” to create metrics (such as counting log lines or totalling fields within a line).
One point regarding your expectations - you need to keep a careful eye on cardinality. HTTP method & response code are generally fine, but you need to be increasingly careful when you start looking to include things which are coming from the request, such as headers or the URL. It is generally best either not to use them at all, or do use a processed version - for example the user agent is bad, but something which has processed it to give you the product name is ok; raw URL is bad, but the name of the handler invoked for a particular website section is ok.
Thank you for your comment, however, it did not clarify my concern.
As I said, I am very new to Prometheus and haven’t had time to study documentation carefully.
I am working on several projects simultaneously (writing and supporting several APIs) and this new project was given to me just two weeks ago. Can you please:
Confirm that no such exporter exist (HTTP);
Clarify, what instrumenting application means. Provide some links, please. Does it mean adding annotation to the code and adding spring listeners? It is not desirable, as code already been through security assessment and any changes will require to repeat it.
JMX exporter. I didn’t see it. Can you please provide a link.
I didn’t understand why URL and headers are bad. We do need to group requests per URL patterns and shall be able to query per particular header name-values. (I.e. we do have user ID number passed in a header).
Instrumenting means changing your code to add calls to create metrics - so creating counters, gauges & histograms and then incrementing/updating them, as well as adding the central code to allow a HTTP endpoint for those metrics to be scraped.
It is important to control cardinality of your labels. Things like URLs & hearders that come directly from a user (without any processing to restrict to specific values) can explode as there is no control over their values. Someone could do a million requests to your application with slightly different URLs. That would result in a million time series being created, using up loads of resources both within your application & Prometheus itself, possibly causing both to fail (depending on how much CPU & memory has been allocated).
Exporters are created to take an existing metrics mechanism (e.g. a custom API) and converting it into Prometheus format. So there isn’t such a thing as a “HTTP Exporter”. There are things like the JSON Exporter which can convert JSON into Prometheus metrics, as well as tools like mtail which can process log files into metrics as well, but they are highly dependant on the application producing things in a suitable format already, as well as lots of configuration/coding within the tool.