Monitoring & Alert Via Prometheus

Metrics are crucial analytical data for system observability. AutoMQ supports exposing various native Apache Kafka® Metrics data through Prometheus. This article outlines the details of the Metrics exposed by AutoMQ.

Metrics collection methods

AutoMQ supports Prometheus-compatible monitoring through the following methods:

Remote Write: AutoMQ pushes metrics directly to a user-provided Prometheus Remote Write endpoint. Configure the endpoint and authentication when you create or update the instance.
Exporter: AutoMQ exposes a Prometheus-compatible scrape endpoint that your monitoring system pulls.

Exporter service discovery depends on the deployment platform:

Kubernetes deployments: Discover the scrape endpoint by using standard Prometheus annotations on Kubernetes resources.
Virtual machine deployments: Discover the hosts by using the host tags provided by the cloud provider.

Both methods provide metrics collection and delivery. Configure dashboards and alerts in your Prometheus-compatible monitoring system. AutoMQ also provides Grafana dashboard templates and Prometheus alert rule templates.

Prometheus Metrics Definitions

For metric names, labels, and definitions, see Prometheus Metrics in the AutoMQ open-source documentation.

Grafana Dashboard Example

As mentioned above, AutoMQ Cloud does not offer managed Grafana dashboard services. Users can refer to the Grafana templates provided by AutoMQ to quickly configure dashboards. The related dashboard templates can be downloaded from this link. The pre-configured Grafana dashboard templates offer metric monitoring across different dimensions:

Cluster Overview: Provides cluster-level monitoring, including node count, data size, cluster traffic, etc. It also includes metric overviews for Topic, Group, and Broker dimensions and offers drill-down capabilities for detailed monitoring.

Broker Metrics: Offers broker-level monitoring, including connection count, partition count, node traffic, node requests, and more.

Topic Metrics: Provides topic-level metric monitoring, including message throughput, total data volume, partition count, and consumption latency.

Group Metrics: Delivers group-level metric monitoring, including consumption rate and consumption latency.

Business Monitoring and Alerts

After sending AutoMQ metrics to Prometheus, configure alerts for cluster capacity, topic writes, and consumer time lag according to your workload.

Kafka Cluster Alert Templates

These Prometheus alert templates cover cluster capacity, topic writes, and consumer freshness. Select the rules your applications need, then adjust the filters, thresholds, and duration to match the workload. Prometheus evaluates the rules; Alertmanager routes notifications. Confirm that your deployment exports the required metrics. Metrics and labels may vary by deployment and version. The examples assume job identifies a cluster. If job identifies a scrape target, use the label that identifies your cluster.

Alert	Example condition	Suitable workload
Broker CPU reaches 80% of the AutoMQ safety watermark	A broker’s `automq_cpu_ratio` exceeds `0.8`.	You want notice before a broker reaches AutoMQ’s CPU safety watermark.
Broker network usage above 80%	A broker’s `automq_network_ratio * 0.8` exceeds `0.8`.	You want to track underlying network utilization.
Topic total write rate drop	The topic total over the current 5 minutes falls below 50% of the 5-minute window from 10 minutes earlier, provided that baseline rate was at least 5 MiB/s.	A topic receives steady writes.
Consumer time lag above 2 minutes	The highest estimated lag among a group’s topic partitions exceeds 120,000 ms.	Consumers normally follow the latest records.
Consumer time lag up 10 minutes from yesterday	The group’s highest estimated lag exceeds its value at the same time yesterday by 600,000 ms.	A group accumulates predictable lag during daily peaks.

Prometheus rule examples

By default, the templates match all nonempty topics and consumer groups. Adjust the topic and group filters and comparison thresholds to match the resources and workload you want to monitor.

groups:
  - name: automq_customer_alerts
    rules:
      - alert: AutoMQBrokerCPUSafetyWatermarkHigh
        expr: max by (job) (automq_cpu_ratio) > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Cluster {{ $labels.job }} reached 80% of AutoMQ's CPU safety watermark on its busiest broker"

      - alert: AutoMQBrokerNetworkUsageHigh
        expr: max by (job) (automq_network_ratio * 0.8) > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Cluster {{ $labels.job }} network usage exceeded 80% on its busiest broker"

      - alert: AutoMQTopicWriteRateDrop
        expr: |
          (
            sum by (job, topic) (max by (job, topic, partition) (
              rate(kafka_network_io_bytes_total{direction="in", topic=~".+"}[5m])
            ))
            < 0.5 *
            sum by (job, topic) (max by (job, topic, partition) (
              rate(kafka_network_io_bytes_total{direction="in", topic=~".+"}[5m] offset 10m)
            ))
          )
          and on (job, topic)
          sum by (job, topic) (max by (job, topic, partition) (
            rate(kafka_network_io_bytes_total{direction="in", topic=~".+"}[5m] offset 10m)
          )) >= 5 * 1024 * 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Topic {{ $labels.topic }} total write rate dropped in cluster {{ $labels.job }}"

      - alert: AutoMQConsumerTimeLagHigh
        expr: |
          max by (job, group, topic) (
            consumer_estimated_time_lag_milliseconds{group=~".+", topic=~".+"} >= 0
          ) > 120000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Consumer group {{ $labels.group }} in cluster {{ $labels.job }} is more than 2 minutes behind on topic {{ $labels.topic }}"

      - alert: AutoMQConsumerTimeLagAboveYesterday
        expr: |
          (
            max by (job, group, topic) (
              consumer_estimated_time_lag_milliseconds{group=~".+", topic=~".+"} >= 0
            )
            -
            max by (job, group, topic) (
              consumer_estimated_time_lag_milliseconds{group=~".+", topic=~".+"} offset 1d >= 0
            )
          ) > 600000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Consumer group {{ $labels.group }} in cluster {{ $labels.job }} exceeds yesterday's lag by 10 minutes on topic {{ $labels.topic }}"

Check the expressions against your metrics before routing notifications. An absent metric or an unavailable yesterday value does not satisfy these comparisons; configure a separate missing-metrics alert if you need to monitor data collection.

Managed Connector Alert Templates

AutoMQ also provides a series of alert templates for Managed Connector scenarios to monitor Connector task status, resource usage, and data processing anomalies. The list of alert rule templates is as follows:

Alert Rule Item	Function and Scenario
TasksFailed	Alert Rule: Connector Task Failure Alert Function and Scenario: Monitors if the number of failed Connector tasks is greater than 0, used to detect task anomalies promptly.
CPUUsage	Alert Rule: High CPU Usage Alert Function and Scenario: Monitors if the CPU usage of a Connector instance exceeds 80%, used to monitor resource pressure.
TotalRecordFailures	Alert Rule: Record Processing Failure Alert Function and Scenario: Monitors if the total number of failed records processed by Connector tasks is greater than 0, used to detect data processing anomalies.
FailedAuthenticationConnections	Alert Rule: Authentication Failure Connection Alert Function and Scenario: Monitors the number of failed authentication connections between Connector and Kafka cluster, used to detect connection configuration issues.

A complete alert template YAML file can be found below for copying and importing.

groups:
  - name: connect_alerts
    rules:
      - alert: TasksFailed
        expr: sum(kafka_connect_worker_metrics_connector_failed_task_count{connect_instance_id=~"<your-connect-instance-id>", connector=~"<your-connector-name>"}) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Connector tasks failed for {{ $labels.connector }} in connect instance {{ $labels.connect_instance_id }}"
          description: "The number of failed tasks for connector {{ $labels.connector }} is greater than 0."

      - alert: CPUUsage
        expr: rate(jvm_cpu_time_seconds_total{connect_instance_id=~"<your-connect-instance-id>"}[5m]) > 0.8
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage {{ printf \"%0.2f\" $value }} for connect instance {{ $labels.connect_instance_id }}"
          description: "The CPU usage of connect instance {{ $labels.connect_instance_id }} is exceeding 80%."

      - alert: TotalRecordFailures
        expr: kafka_task_error_metrics_total_record_failures_total{connect_instance_id=~"<your-connect-instance-id>", connector=~"<your-connector-name>"} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Record failures detected for connector {{ $labels.connector }} task {{ $labels.task }}"
          description: "The total record failures for connector {{ $labels.connector }} task {{ $labels.task }} is greater than 0."

      - alert: FailedAuthenticationConnections
        expr: kafka_connect_metrics_failed_authentication_total{connect_instance_id=~"<your-connect-instance-id>", client_id!=""} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Failed authentication connections for client {{ $labels.client_id }} in connect instance {{ $labels.connect_instance_id }}"
          description: "The number of failed authentication connections for client {{ $labels.client_id }} is greater than 0."

When using the Managed Connector alert templates, replace <your-connect-instance-id> and <your-connector-name> with your actual Connect instance ID and Connector name.

Overview

Getting Started

Deploy on Kubernetes

Subscriptions and Billings

Manage Environments

Manage Identities and Access

Managing AutoMQ

Migrate to AutoMQ

Manage Security

Kafka Connect