For applications that do not have a web layer and where there is resistance to adding a web layer (in our case, Kafka workers), we are contemplating deploying the push gateway as a sidecar in the worker pods.
The push gateway would act as the web layer that the application team does not want in their application, but still be tied to the pods lifecycle. The application would then push metrics to the sidecar over localhost via a background thread and prometheus would scrape the sidecar push gateway as if it was the application itself.
We POC’d this idea and proved it functionally works as a concept, but since I cannot find any examples of this being done we wanted to engage the community. Is there anything about the push gateway that makes this a bad idea? Are there scale/monitoring/ops gotchas I should consider? If it is a bad idea, other than forcing these teams to implement a web layer, are there better options? Thank you for your time, expertise and advise.