Skip to main content
AutoMQ is a cloud-native, redesigned distribution of Kafka that fully utilizes cloud object storage services. It can build the storage layer using object storage from major cloud providers or storage software.

​
Object Storage URI Construction

The URI is a unique identifier used by AutoMQ to represent writes to object storage, adhering to established standards to reconcile differences among various object storage service providers. The URI components are as follows: s3 specifies the protocol; xxx_bucket indicates the specific bucket; region and endpoint are additional parameters used by different vendors to locate the bucket. Ultimately, replace the variables in the configuration templates global.config and global.envs from Deploy Multi-Nodes Cluster on Linux▸ to generate the final configuration files.
ProviderTypeURI Example
AWSCloud Providers3://xxx_bucket?region=us-east-1
AWS-CNCloud Providers3://xxx_bucket?region=cn-northwest-1&endpoint=https://s3.amazonaws.com.cn
ALIYUNCloud Providers3://xxx_bucket?region=oss-cn-shanghai&endpoint=https://oss-cn-shanghai.aliyuncs.com
OCI
Cloud Provider
s3://xxx_bucket?region=us-ashburn-1&endpoint=https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com&pathStyle=true
CEPH
Storage Software
s3://xxx_bucket?region=us-west-2&endpoint=http://127.0.0.1:80
CubeFS
Storage Software
s3://xxx_bucket?region=us-west-2&endpoint=http://10.1.0.240:17410
MinIO
Storage Software
s3://xxx_bucket?region=us-west-2&pathStyle=true&endpoint=http://10.1.0.240:9000
If the s3.wal.path is set, you can also include the batchInterval and maxBytesInBatch parameters in the URI to control the submit interval and the maximum size of bytes per batch, respectively. The default settings are batchInterval=250 milliseconds and maxBytesInBatch=8388608 bytes.

​
Cloud Object Storage Configuration

  • AWS
  • Aliyun
  • GCP
  • Azure
  • OCI
  • Other Cloud Providers
It is recommended to refer to the Policy and Endpoint & Region links in the AWS documentation to get detailed configuration instructions for IAM and S3.
In a production environment, it is advisable to scope IAM Policy permissions to specific buckets to avoid unintended authorizations.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

​
Storage Software Configuration

If you wish to deploy AutoMQ in a private data center, you can also use storage software like MinIO, Ceph, or CubeFS that supports the S3 protocol to provide object storage services.