Assume a dataflow as in the picture. Data is being transported from a location to the cloud via a data gateway and bridged MQTT brokers to do Stuff™. The firewalls allows you to initialize communication from a lower network layer to a higher one, but not vice-versa. You can also not skip network layers.
The data gateway, OS, and MQTT brokers have /metrics endpoints that have metrics in the Prometheus format. I would like to push these metrics to the cloud so that I can monitor the data flow and servers from there.
I read about federation, and the push gateway. Using one of these would at least solve pushing metrics through one network layer. However the documentation recommends against using push gateway for services that are always running. It would also mean implementing the push model in all services, and opening more firewall ports.
I could create a set of services for replicating metrics in the network layer through the existing MQTT dataflow. But that would be a custom solution.
Is there a standard way of setting up Prometheus to push metrics through several network layers? Or existing tooling?