We are using Prometheus to monitor Airflow DAGs.
Below are metrics provided by Airflow for monitoring,
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html
We tried using every possible metrics but we are unable to get below points:
- The only last run status of DAGs, if a DAG runs multiple times in a day
- The timestamp at which it has been successful or failed
- And no. of times it failed or no. of re-runs after failed