I’m trying to scrape Api Gateway with a specific tag from AWS using cloudwatch exporter on EKS v0.15.5.
My guess is that the GetMetricData API call returns no data points, but I have no idea how the config should look like in order to fetch the data.
I will appreciate any advice on how to proceed, as we have many api gateways in this account and scraping all of them without tags is not an option.
I have several other resources which already works with this strategy, for example, scraping kinesis data works:
region: us-east-2
use_get_metric_data: true
range_seconds: 60
metrics:
- aws_namespace: AWS/Kinesis
aws_metric_name: ReadProvisionedThroughputExceeded
aws_dimensions: [StreamName]
aws_tag_select:
resource_type_selection: "kinesis:stream"
resource_id_dimension: StreamName
tag_selections:
cloud-monitor: ["True"]
aws_statistics: [Average]
Using the following configuration, I’m unable to get the metrics
region: us-east-2
use_get_metric_data: true
metrics:
- aws_namespace: AWS/ApiGateway
aws_metric_name: 4XXError
aws_dimensions: [ApiName]
aws_tag_select:
resource_type_selection: "apigateway:restapi"
resource_id_dimension: ApiName
tag_selections:
cloud-monitor: ["True"]
aws_statistics: [Average]
However, using the configuration without the specified tags works:
region: us-east-2
use_get_metric_data: true
metrics:
- aws_namespace: AWS/ApiGateway
aws_metric_name: 4XXError
aws_dimensions: [ApiName]
aws_statistics: [Average]
I am aware that there is a configuration with arn_resource_id_regexp, but this also, does not work:
- aws_namespace: AWS/ApiGateway
aws_metric_name: Count
aws_dimensions: [ApiId]
aws_tag_select:
resource_type_selection: "apigateway:"
resource_id_dimension: ApiId
arn_resource_id_regexp: "apis/(.+)$"
tag_selections:
cloud-monitor: ["True"]
aws_statistics: [Average]
I know that the apigateways are found because when querying the exporter directly i can see the resource info.
# HELP tagging_api_requests_total API requests made to the Resource Groups Tagging API
# TYPE tagging_api_requests_total counter
tagging_api_requests_total{action="getResources",resource_type="apigateway:",} 3.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 16.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 6.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 16.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 16.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 6.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 2.0
jvm_threads_state{state="TIMED_WAITING",} 8.0
jvm_threads_state{state="UNKNOWN",} 0.0
# HELP cloudwatch_exporter_build_info Non-zero if build info scrape failed.
# TYPE cloudwatch_exporter_build_info gauge
cloudwatch_exporter_build_info{build_version="0.15.4",release_date="2023-06-02",} 1.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3.24
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.717501755797E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 15.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 3.596689408E9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.29499136E8
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Eclipse Adoptium",version="17.0.7+7",} 1.0
# HELP aws_resource_info AWS information available for resource
# TYPE aws_resource_info gauge
aws_resource_info{job="aws_apigateway",instance="",arn="arn:aws:apigateway:us-east-1::/restapis/<id>",api_name="restapis/<id>",tag_cloud_monitor="True",} 1.0
# HELP cloudwatch_exporter_scrape_duration_seconds Time this CloudWatch scrape took, in seconds.
# TYPE cloudwatch_exporter_scrape_duration_seconds gauge
cloudwatch_exporter_scrape_duration_seconds 0.194506808
# HELP cloudwatch_exporter_scrape_error Non-zero if this scrape failed.
# TYPE cloudwatch_exporter_scrape_error gauge
cloudwatch_exporter_scrape_error 0.0
# HELP cloudwatch_metrics_requested_total Metrics requested by either GetMetricStatistics or GetMetricData
# TYPE cloudwatch_metrics_requested_total counter
cloudwatch_metrics_requested_total{metric_name="4XXError",namespace="AWS/ApiGateway",} 4.0
# HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="Eden Space",} 6.1311888E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 6477696.0
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1363584.0
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 3345472.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 2.6307712E7
jvm_memory_pool_allocated_bytes_total{pool="Tenured Gen",} 1.1981128E7
jvm_memory_pool_allocated_bytes_total{pool="Survivor Space",} 2071096.0
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1298432.0
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="Copy",} 13.0
jvm_gc_collection_seconds_sum{gc="Copy",} 0.108
jvm_gc_collection_seconds_count{gc="MarkSweepCompact",} 1.0
jvm_gc_collection_seconds_sum{gc="MarkSweepCompact",} 0.015
# HELP cloudwatch_requests_total API requests made to CloudWatch
# TYPE cloudwatch_requests_total counter
cloudwatch_requests_total{action="listMetrics",namespace="AWS/ApiGateway",} 4.0
# HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
# TYPE jvm_memory_objects_pending_finalization gauge
jvm_memory_objects_pending_finalization 0.0
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 1.5628568E7
jvm_memory_bytes_used{area="nonheap",} 3.9081664E7
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 3.047424E7
jvm_memory_bytes_committed{area="nonheap",} 4.2074112E7
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 4.84573184E8
jvm_memory_bytes_max{area="nonheap",} -1.0
# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_init gauge
jvm_memory_bytes_init{area="heap",} 3.145728E7
jvm_memory_bytes_init{area="nonheap",} 7667712.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1298432.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 2.6360408E7
jvm_memory_pool_bytes_used{pool="Tenured Gen",} 1.1981128E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 6657792.0
jvm_memory_pool_bytes_used{pool="Eden Space",} 3299224.0
jvm_memory_pool_bytes_used{pool="Survivor Space",} 348216.0
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 3346024.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 1419008.0
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_committed{pool="Metaspace",} 2.6738688E7
jvm_memory_pool_bytes_committed{pool="Tenured Gen",} 2.097152E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 6684672.0
jvm_memory_pool_bytes_committed{pool="Eden Space",} 8454144.0
jvm_memory_pool_bytes_committed{pool="Survivor Space",} 1048576.0
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 3538944.0
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="Tenured Gen",} 3.34168064E8
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_pool_bytes_max{pool="Eden Space",} 1.33758976E8
jvm_memory_pool_bytes_max{pool="Survivor Space",} 1.6646144E7
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_init gauge
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="Tenured Gen",} 2.097152E7
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Eden Space",} 8388608.0
jvm_memory_pool_bytes_init{pool="Survivor Space",} 1048576.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
# HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_used_bytes gauge
jvm_memory_pool_collection_used_bytes{pool="Tenured Gen",} 1.0768776E7
jvm_memory_pool_collection_used_bytes{pool="Eden Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="Survivor Space",} 348216.0
# HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_committed_bytes gauge
jvm_memory_pool_collection_committed_bytes{pool="Tenured Gen",} 2.097152E7
jvm_memory_pool_collection_committed_bytes{pool="Eden Space",} 8454144.0
jvm_memory_pool_collection_committed_bytes{pool="Survivor Space",} 1048576.0
# HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_max_bytes gauge
jvm_memory_pool_collection_max_bytes{pool="Tenured Gen",} 3.34168064E8
jvm_memory_pool_collection_max_bytes{pool="Eden Space",} 1.33758976E8
jvm_memory_pool_collection_max_bytes{pool="Survivor Space",} 1.6646144E7
# HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_init_bytes gauge
jvm_memory_pool_collection_init_bytes{pool="Tenured Gen",} 2.097152E7
jvm_memory_pool_collection_init_bytes{pool="Eden Space",} 8388608.0
jvm_memory_pool_collection_init_bytes{pool="Survivor Space",} 1048576.0
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 84064.0
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
jvm_buffer_pool_capacity_bytes{pool="direct",} 84064.0
jvm_buffer_pool_capacity_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
jvm_buffer_pool_used_buffers{pool="direct",} 9.0
jvm_buffer_pool_used_buffers{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 6315.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 6315.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP cloudwatch_metrics_requested_created Metrics requested by either GetMetricStatistics or GetMetricData
# TYPE cloudwatch_metrics_requested_created gauge
cloudwatch_metrics_requested_created{metric_name="4XXError",namespace="AWS/ApiGateway",} 1.717501777207E9
# HELP cloudwatch_requests_created API requests made to CloudWatch
# TYPE cloudwatch_requests_created gauge
cloudwatch_requests_created{action="listMetrics",namespace="AWS/ApiGateway",} 1.717501777204E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="Eden Space",} 1.7175017572E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.717501757199E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.7175017572E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.7175017572E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.7175017572E9
jvm_memory_pool_allocated_bytes_created{pool="Tenured Gen",} 1.7175017572E9
jvm_memory_pool_allocated_bytes_created{pool="Survivor Space",} 1.7175017572E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.7175017572E9
# HELP tagging_api_requests_created API requests made to the Resource Groups Tagging API
# TYPE tagging_api_requests_created gauge
tagging_api_requests_created{action="getResources",resource_type="apigateway:",} 1.717501776065E9
Also, an error shows in the exporter logs:
Jun 04, 2024 12:05:23 PM org.eclipse.jetty.server.AsyncContentProducer isError
FINE: isError = false AsyncContentProducer@2a3c7163[r=ErrorContent [org.eclipse.jetty.util.StaticException: Unconsumed content],t=ErrorContent [org.eclipse.jetty.util.StaticException: Unconsumed content],i=null,error=false,c=HttpChannelOverHttp@4fae80e1{s=HttpChannelState@1f30d2ad{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0},r=1,c=true/true,a=IDLE,uri=http://<endpoint>/metrics,age=185}]
Jun 04, 2024 12:05:23 PM org.eclipse.jetty.server.HttpInput isError
FINE: isError=false HttpInput@1288994443 cs=HttpChannelState@1f30d2ad{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0} cp=org.eclipse.jetty.server.BlockingContentProducer@42c0e479 eof=true
Jun 04, 2024 12:05:23 PM org.eclipse.jetty.server.HttpChannelState recycle
FINE: recycle HttpChannelState@1f30d2ad{s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0}
Jun 04, 2024 12:05:23 PM org.eclipse.jetty.server.HttpInput recycle
FINE: recycle HttpInput@1288994443 cs=HttpChannelState@1f30d2ad{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} cp=org.eclipse.jetty.server.BlockingContentProducer@42c0e479 eof=true
Jun 04, 2024 12:05:23 PM org.eclipse.jetty.server.BlockingContentProducer recycle
FINE: recycling org.eclipse.jetty.server.BlockingContentProducer@42c0e479
Jun 04, 2024 12:05:23 PM org.eclipse.jetty.server.AsyncContentProducer recycle
FINE: recycling AsyncContentProducer@2a3c7163[r=ErrorContent [org.eclipse.jetty.util.StaticException: Unconsumed content],t=ErrorContent [org.eclipse.jetty.util.StaticException: Unconsumed content],i=null,error=false,c=HttpChannelOverHttp@4fae80e1{s=HttpChannelState@1f30d2ad{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=http://<endpoint>/metrics,age=185}]