> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Monitoring & Alert Via Prometheus
> AutoMQ offers cloud-native scalability and cost efficiency with Kafka compatibility. Monitor Kafka metrics effortlessly using Prometheus and Grafana, integrating custom alerts seamlessly.
Metrics are crucial analytical data for system observability. AutoMQ supports exposing various native Apache Kafka® Metrics data through Prometheus. This article outlines the details of the Metrics exposed by AutoMQ.
## Metrics collection methods
AutoMQ supports Prometheus-compatible monitoring through the following methods:
1. **Remote Write:** AutoMQ pushes metrics directly to a user-provided Prometheus Remote Write endpoint. Configure the endpoint and authentication when you create or update the instance.
2. **Exporter:** AutoMQ exposes a Prometheus-compatible scrape endpoint that your monitoring system pulls.
Exporter service discovery depends on the deployment platform:
* **Kubernetes deployments:** Discover the scrape endpoint by using standard Prometheus annotations on Kubernetes resources.
* **Virtual machine deployments:** Discover the hosts by using the host tags provided by the cloud provider.
Both methods provide metrics collection and delivery. Configure dashboards and alerts in your Prometheus-compatible monitoring system. AutoMQ also provides [Grafana dashboard templates](https://download.automq.com/automq-dashboard/dashboard-20260203.zip) and [Prometheus alert rule templates](#kafka-cluster-alert-templates).
## Prometheus Metrics Definitions
For metric names, labels, and definitions, see [Prometheus Metrics in the AutoMQ open-source documentation](/automq/observability/prometheus-metrics).
## Grafana Dashboard Example
As mentioned above, AutoMQ Cloud does not offer managed Grafana dashboard services. Users can refer to the Grafana templates provided by AutoMQ to quickly configure dashboards. The related dashboard templates can be downloaded from [this link](https://download.automq.com/automq-dashboard/dashboard-20260203.zip).
The pre-configured Grafana dashboard templates offer metric monitoring across different dimensions:
* **Cluster Overview:** Provides cluster-level monitoring, including node count, data size, cluster traffic, etc. It also includes metric overviews for Topic, Group, and Broker dimensions and offers drill-down capabilities for detailed monitoring.
* **Broker Metrics:** Offers broker-level monitoring, including connection count, partition count, node traffic, node requests, and more.
* **Topic Metrics:** Provides topic-level metric monitoring, including message throughput, total data volume, partition count, and consumption latency.
* **Group Metrics:** Delivers group-level metric monitoring, including consumption rate and consumption latency.
## Business Monitoring and Alerts
After sending AutoMQ metrics to Prometheus, configure alerts for cluster capacity, topic writes, and consumer time lag according to your workload.
### Kafka Cluster Alert Templates
These Prometheus alert templates cover cluster capacity, topic writes, and consumer freshness. Select the rules your applications need, then adjust the filters, thresholds, and duration to match the workload.
Prometheus evaluates the rules; Alertmanager routes notifications. Confirm that your deployment exports the required metrics. Metrics and labels may vary by deployment and version. The examples assume `job` identifies a cluster. If `job` identifies a scrape target, use the label that identifies your cluster.
| Alert | Example condition | Suitable workload |
| - | - | - |
| Broker CPU reaches 80% of the AutoMQ safety watermark | A broker's `automq_cpu_ratio` exceeds `0.8`. | You want notice before a broker reaches AutoMQ's CPU safety watermark. |
| Broker network usage above 80% | A broker's `automq_network_ratio * 0.8` exceeds `0.8`. | You want to track underlying network utilization. |
| Topic total write rate drop | The topic total over the current 5 minutes falls below 50% of the 5-minute window from 10 minutes earlier, provided that baseline rate was at least 5 MiB/s. | A topic receives steady writes. |
| Consumer time lag above 2 minutes | The highest estimated lag among a group's topic partitions exceeds 120,000 ms. | Consumers normally follow the latest records. |
| Consumer time lag up 10 minutes from yesterday | The group's highest estimated lag exceeds its value at the same time yesterday by 600,000 ms. | A group accumulates predictable lag during daily peaks. |
#### Prometheus rule examples
By default, the templates match all nonempty topics and consumer groups. Adjust the `topic` and `group` filters and comparison thresholds to match the resources and workload you want to monitor.
```yaml theme={null}
groups:
- name: automq_customer_alerts
rules:
- alert: AutoMQBrokerCPUSafetyWatermarkHigh
expr: max by (job) (automq_cpu_ratio) > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Cluster {{ $labels.job }} reached 80% of AutoMQ's CPU safety watermark on its busiest broker"
- alert: AutoMQBrokerNetworkUsageHigh
expr: max by (job) (automq_network_ratio * 0.8) > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Cluster {{ $labels.job }} network usage exceeded 80% on its busiest broker"
- alert: AutoMQTopicWriteRateDrop
expr: |
(
sum by (job, topic) (max by (job, topic, partition) (
rate(kafka_network_io_bytes_total{direction="in", topic=~".+"}[5m])
))
< 0.5 *
sum by (job, topic) (max by (job, topic, partition) (
rate(kafka_network_io_bytes_total{direction="in", topic=~".+"}[5m] offset 10m)
))
)
and on (job, topic)
sum by (job, topic) (max by (job, topic, partition) (
rate(kafka_network_io_bytes_total{direction="in", topic=~".+"}[5m] offset 10m)
)) >= 5 * 1024 * 1024
for: 5m
labels:
severity: warning
annotations:
summary: "Topic {{ $labels.topic }} total write rate dropped in cluster {{ $labels.job }}"
- alert: AutoMQConsumerTimeLagHigh
expr: |
max by (job, group, topic) (
consumer_estimated_time_lag_milliseconds{group=~".+", topic=~".+"} >= 0
) > 120000
for: 5m
labels:
severity: warning
annotations:
summary: "Consumer group {{ $labels.group }} in cluster {{ $labels.job }} is more than 2 minutes behind on topic {{ $labels.topic }}"
- alert: AutoMQConsumerTimeLagAboveYesterday
expr: |
(
max by (job, group, topic) (
consumer_estimated_time_lag_milliseconds{group=~".+", topic=~".+"} >= 0
)
-
max by (job, group, topic) (
consumer_estimated_time_lag_milliseconds{group=~".+", topic=~".+"} offset 1d >= 0
)
) > 600000
for: 5m
labels:
severity: warning
annotations:
summary: "Consumer group {{ $labels.group }} in cluster {{ $labels.job }} exceeds yesterday's lag by 10 minutes on topic {{ $labels.topic }}"
```
Check the expressions against your metrics before routing notifications. An absent metric or an unavailable yesterday value does not satisfy these comparisons; configure a separate missing-metrics alert if you need to monitor data collection.
### Managed Connector Alert Templates
AutoMQ also provides a series of alert templates for Managed Connector scenarios to monitor Connector task status, resource usage, and data processing anomalies.
The list of alert rule templates is as follows:
| **Alert Rule Item** | **Function and Scenario** |
| - | - |
| TasksFailed | - Alert Rule: Connector Task Failure Alert
- Function and Scenario: Monitors if the number of failed Connector tasks is greater than 0, used to detect task anomalies promptly.
|
| CPUUsage | - Alert Rule: High CPU Usage Alert
- Function and Scenario: Monitors if the CPU usage of a Connector instance exceeds 80%, used to monitor resource pressure.
|
| TotalRecordFailures | - Alert Rule: Record Processing Failure Alert
- Function and Scenario: Monitors if the total number of failed records processed by Connector tasks is greater than 0, used to detect data processing anomalies.
|
| FailedAuthenticationConnections | - Alert Rule: Authentication Failure Connection Alert
- Function and Scenario: Monitors the number of failed authentication connections between Connector and Kafka cluster, used to detect connection configuration issues.
|
A complete alert template YAML file can be found below for copying and importing.
```yaml theme={null}
groups:
- name: connect_alerts
rules:
- alert: TasksFailed
expr: sum(kafka_connect_worker_metrics_connector_failed_task_count{connect_instance_id=~"", connector=~""}) > 0
for: 1m
labels:
severity: critical
annotations:
summary: "Connector tasks failed for {{ $labels.connector }} in connect instance {{ $labels.connect_instance_id }}"
description: "The number of failed tasks for connector {{ $labels.connector }} is greater than 0."
- alert: CPUUsage
expr: rate(jvm_cpu_time_seconds_total{connect_instance_id=~""}[5m]) > 0.8
for: 1m
labels:
severity: warning
annotations:
summary: "High CPU usage {{ printf \"%0.2f\" $value }} for connect instance {{ $labels.connect_instance_id }}"
description: "The CPU usage of connect instance {{ $labels.connect_instance_id }} is exceeding 80%."
- alert: TotalRecordFailures
expr: kafka_task_error_metrics_total_record_failures_total{connect_instance_id=~"", connector=~""} > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Record failures detected for connector {{ $labels.connector }} task {{ $labels.task }}"
description: "The total record failures for connector {{ $labels.connector }} task {{ $labels.task }} is greater than 0."
- alert: FailedAuthenticationConnections
expr: kafka_connect_metrics_failed_authentication_total{connect_instance_id=~"", client_id!=""} > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Failed authentication connections for client {{ $labels.client_id }} in connect instance {{ $labels.connect_instance_id }}"
description: "The number of failed authentication connections for client {{ $labels.client_id }} is greater than 0."
```
> [!TIP]
>
> When using the Managed Connector alert templates, replace `` and `` with your actual Connect instance ID and Connector name.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.