Skip to content

Managed Kafka

DanubeData runs Apache Kafka for you. A cluster is a Kafka 4.3 broker in KRaft mode (no ZooKeeper), reachable over TLS with a publicly trusted certificate and authenticated with SASL/SCRAM-SHA-512. Any standard Kafka client works.

Kafka is a queue provider next to RabbitMQ: you create it from the same Queues area, API (/api/v1/queue) and CLI (danube queue). Everything in the RabbitMQ pages about MQTT, STOMP and AMQP does not apply to Kafka clusters.

What you get in this version

  • One broker per cluster. There is no high availability yet. If the broker restarts, clients are disconnected until it is back.
  • TLS with a publicly trusted certificate. Clients verify the server against the system trust store. You do not need a CA file.
  • SASL/SCRAM-SHA-512 with an admin user, kafka-<name>-admin, which has full access to the cluster, and up to 20 more users limited by ACLs. See Users and ACLs.
  • A public endpoint kafka-<name>-<project>.queues.danubedata.ro:<port>. The cluster's default firewall allows only that port.
  • Persistent storage sized by the plan.

Not available yet:

  • Stop and start
  • Snapshots and backups
  • Plan changes (resizing a cluster) and changes to settings
  • Deny rules and rules limited to a client address (every ACL rule allows, from any host)
  • Multiple brokers and replication across brokers
  • A topic browser in the console (create and manage topics with Kafka's own tools, below)

Metrics, logs, events and diagnose work. See Metrics below.

The API gives the same "not available for Kafka yet" answer to stop, start, snapshot, plan and settings changes.

Plans

PlanConsole namevCPURAMStorage
kafka-smallDD Isac12 GB25 GB
kafka-mediumDD Golovita24 GB75 GB
kafka-largeDD Merhei48 GB200 GB

See Pricing for what each plan costs. Billing is hourly, like every other resource, and a cluster is billed until you delete it.

Create a cluster

In the console

  1. In the sidebar, open Messaging > Apache Kafka and click Create Kafka cluster (or Create your first Kafka cluster when the list is empty). The page opens with Apache Kafka already chosen. Alternatively, click Create queue on the RabbitMQ list and choose the Apache Kafka broker card.
  2. Under Broker, check that Apache Kafka is selected.
  3. Under Plan, pick a plan. Each shows its vCPU, RAM, disk and monthly price. A plan above your account limit says Upgrade required, with a Request a limit increase link. A Kafka version selector appears only when more than one version is offered; today that is 4.3 alone, so it is hidden.
  4. Location is Falkenstein, Germany. The cluster gets a public TLS endpoint and a firewall, which you can restrict from its Security tab.
  5. Enter a Name and click Create queue. If your project has no payment method yet, you are asked to add one first.

The name cannot be changed later. It uses lowercase letters, numbers and hyphens, from 2 to at most 24 characters (fewer when your project's name is long), and becomes part of the cluster's hostname.

Provisioning takes a few minutes while the public endpoint and its certificate are set up. The Kafka list shows Kafka cluster created. Provisioning will begin shortly., and the cluster's row says the endpoint and its certificate are being set up until it is ready. When the cluster is ready, we email you (subject "Your Kafka cluster '<name>' is ready") with the bootstrap server, the SASL settings and the username. The email never contains the password.

With the API

Bash
curl -X POST https://danubedata.ro/api/v1/queue \
  -H "Authorization: Bearer $DANUBE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "events", "provider": "kafka", "resource_profile": "kafka-small", "version": "4.3"}'

provider is rabbitmq (the default) or kafka. GET /api/v1/queue?provider=kafka lists only Kafka clusters, and every instance reports its provider.

Provisioning is asynchronous: poll GET /api/v1/queue/{id} until status_details.operation.terminal is true, as for every other resource (see Platform Automation).

With the CLI

Bash
danube queue create --name events --provider kafka --profile kafka-small --version=4.3
danube queue ls --provider kafka

Write the version as --version=4.3. danube queue connection-info events prints the connection details once the cluster is ready.

Connection details

In the console, open the cluster (Open cluster in the list's drawer) and use the Connectivity tab. The Connect to your cluster card shows the bootstrap server, security protocol, SASL mechanism and username, and keeps the password hidden until you choose Reveal the password. Below it, Client settings has ready-to-copy snippets for Java (client.properties), kcat and Node.js (KafkaJS), and Download client.properties saves the same settings as a file (shown to members who may see the cluster's credentials). When the cluster has a Schema Registry, the card also has a Schema Registry section with its URL, username and password (separate from the cluster's admin password), and the snippets include its settings.

From the CLI:

Bash
danube queue connection-info events

or GET /api/v1/queue/{id}/connection-info, which returns:

FieldValue
statusprovisioning or ready
bootstrap_serverskafka-<name>-<project>.queues.danubedata.ro:<port>
security_protocolSASL_SSL
sasl_mechanismSCRAM-SHA-512
usernamekafka-<name>-admin
passwordThe admin password
host, portThe two halves of bootstrap_servers
certificateA note that the certificate is publicly trusted

The password is only returned to members whose role may read credentials, and to API tokens with the queue:credentials ability; for everyone else it is null. While the endpoint is still being set up, status is provisioning and bootstrap_servers, host and port are null. The CLI also prints ready-to-use kcat and Java properties snippets.

Treat the admin user like a root credential: it can read, write and delete every topic. Keep it out of source control and out of client-side code, and give your applications their own users.

Connect

The examples read the address and credentials from environment variables:

Bash
export KAFKA_BOOTSTRAP='kafka-events-myproject.queues.danubedata.ro:29092'
export KAFKA_USER='kafka-events-admin'
export KAFKA_PASSWORD='<password>'

The address and port above are examples. Your real bootstrap address, with its port, is shown in the console and in connection-info.

kcat

Bash
kcat -L -b "$KAFKA_BOOTSTRAP" \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanisms=SCRAM-SHA-512 \
  -X sasl.username="$KAFKA_USER" \
  -X sasl.password="$KAFKA_PASSWORD"

-L lists the cluster metadata. Use the same -X options with -P to produce and -C to consume. No CA option is needed.

Java

Put the settings in a properties file and pass it to your client, or to Kafka's command-line tools with --command-config:

properties
bootstrap.servers=<bootstrap address>
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka-events-admin" password="<password>";

No truststore is needed: the JVM's default trust store already trusts the certificate.

Replace <password> with the admin password. If it contains a backslash or a double quote, escape it for the JAAS line and then again for the properties file, because java.util.Properties removes one level of backslashes before JAAS reads the value.

Python (confluent-kafka)

Python
import os
from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "SCRAM-SHA-512",
    "sasl.username": os.environ["KAFKA_USER"],
    "sasl.password": os.environ["KAFKA_PASSWORD"],
})

producer.produce("orders", b"hello")
producer.flush()

Node.js (KafkaJS)

JavaScript
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  brokers: [process.env.KAFKA_BOOTSTRAP],
  ssl: true,
  sasl: {
    mechanism: 'scram-sha-512',
    username: process.env.KAFKA_USER,
    password: process.env.KAFKA_PASSWORD,
  },
});

const producer = kafka.producer();
await producer.connect();
await producer.send({ topic: 'orders', messages: [{ value: 'hello' }] });
await producer.disconnect();

GUI tools (Kafkio, Offset Explorer, Conduktor)

Desktop tools connect with the same settings. In the tool's cluster or connection settings, enter the bootstrap address, then set:

  • Security protocol: SASL_SSL
  • SASL mechanism: SCRAM-SHA-512
  • Username and Password: the admin user and its password

Leave the truststore or CA settings empty. Most of these tools default to SASL plaintext (SASL_PLAINTEXT) with the PLAIN mechanism, and the cluster refuses that combination, so change both fields before you connect.

Schema Registry in GUI tools

Kafka does not tell its clients where a schema registry is, so a tool cannot find your cluster's Schema Registry by itself: set it in the tool's cluster connection, next to the Kafka settings above.

  • URL: the registry's address, https://schema-<name>-<project>.queues.danubedata.ro, shown on the cluster's Connectivity tab.
  • Authentication: HTTP Basic (a username and a password), with the username registry.
  • Password: the registry has its own password, separate from the Kafka admin password. Reveal it in the Schema Registry section of the Connectivity tab.

In Kafkio, open the cluster's connection settings and fill in the Schema Registry section: the URL, Basic authentication, and the username and password above. Offset Explorer and Conduktor have a Schema Registry setting in the cluster's connection or configuration; enter the same URL and choose Basic authentication there. The names of the fields differ between tools and versions.

If your tool reads a Java properties file, choose Download client.properties on the Connectivity tab (shown to members who may see the cluster's credentials) to get the connection settings in one file. The file signs in as the cluster's admin user, a super user that can read, write and manage every topic: use it for your own tools, and give each application its own user with ACLs (see Users and ACLs) instead of sharing it.

Create topics

Topic management is done with Kafka's own tools, authenticated as the admin user. Save the Java properties above as client.properties, then, with the Kafka 4.x command-line tools:

Bash
kafka-topics.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
  --command-config client.properties \
  --create --topic orders --partitions 3 --replication-factor 1

kafka-topics.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
  --command-config client.properties --list

Use --replication-factor 1. The cluster has a single broker, so a higher replication factor is rejected.

Users and ACLs

Every cluster checks each request against ACLs. The admin user, kafka-<name>-admin, is a super user and is never limited. Give each application its own user with only the access it needs, so a leaked password cannot reach everything.

A cluster can have up to 20 users besides its admin. Users can be added, changed and deleted only while the cluster is running.

Names and passwords

You give a user a short name: 1 to 32 lowercase letters, digits and hyphens, unique in the cluster (admin is taken). Clients sign in with the username kafka-<cluster>.<name>, for example kafka-events.billing, using SASL/SCRAM-SHA-512 as for the admin. The whole username is at most 55 characters, so a cluster with a long name allows shorter user names; the user list's meta.max_name_length gives the limit.

The password is generated for you and shown once, in the answer that creates the user. It is never listed again; if it is lost, rotate it. Creating, rotating and deleting users need a role that may read credentials (the owner, Administrators and Editors). An API token needs queue:write and queue:credentials to create or rotate, and queue:delete to delete. Changing a user's rules needs queue:write.

Changes reach the cluster in the background. A user's status is pending until the cluster has applied the latest change, usually within a few minutes, then active. error means the cluster refused it, and status_message says why.

When you rotate a password or delete a user, the change applies to new connections as soon as the user is active again (or gone). Connections that are already open keep working until they next re-authenticate, which every connection does within an hour. Until the rotation is applied, the old password still signs in.

The cluster asks every connection, the admin's included, to re-authenticate within an hour. Clients that support re-authentication (KIP-368: the Java client 2.2 and later, librdkafka 2.2 and later, and the clients built on it) do so without dropping the connection. Older clients, including older kcat and distribution builds of librdkafka, are disconnected once an hour and must reconnect; most do that on their own.

Rules

A user may do only what its rules allow; everything else is denied. Each rule has:

FieldValue
resource_typetopic, group (a consumer group), transactional_id or cluster
nameThe topic, group or transactional id. * with the literal pattern means all of them. Not used for cluster.
patternliteral (exactly this name) or prefix (every name that starts with it). Not used for cluster.
operationsWhat is allowed, from the list below
ResourceOperations
topicread, write, create, delete, alter, describe, describe_configs, alter_configs, all
groupread, describe, delete, describe_configs, alter_configs, all
transactional_iddescribe, write, all
clustercreate, describe, describe_configs, idempotent_write

Changing ACLs or the broker's configuration (alter, alter_configs and all on the cluster) is left to the admin user.

A user can have up to 50 rules. A producer needs write and describe on its topics, which also covers idempotent producing (the default in current clients). A transactional producer also needs write and describe on its transactional id. A consumer needs read and describe on its topics and read on its consumer group.

Example: a producer and a consumer

A producer that writes to every topic starting with orders.:

Bash
curl -X POST https://danubedata.ro/api/v1/queue/$QUEUE_ID/kafka-users \
  -H "Authorization: Bearer $DANUBE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "orders-producer",
    "acls": [
      {"resource_type": "topic", "name": "orders.", "pattern": "prefix", "operations": ["write", "describe"]}
    ]
  }'

A consumer in the group billing that reads the same topics:

Bash
curl -X POST https://danubedata.ro/api/v1/queue/$QUEUE_ID/kafka-users \
  -H "Authorization: Bearer $DANUBE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "billing",
    "acls": [
      {"resource_type": "topic", "name": "orders.", "pattern": "prefix", "operations": ["read", "describe"]},
      {"resource_type": "group", "name": "billing", "pattern": "literal", "operations": ["read"]}
    ]
  }'

Each answer holds the user, with username (kafka-events.orders-producer and kafka-events.billing for a cluster named events) and password. Use them in place of the admin's in the client settings above; nothing else changes.

Manage users with the API

RequestDoes
GET /api/v1/queue/{id}/kafka-usersLists the users and their rules, never the passwords
POST /api/v1/queue/{id}/kafka-usersCreates a user from name and acls, and returns its password
PUT /api/v1/queue/{id}/kafka-users/{userId}Replaces the user's rules with acls (send every rule it should keep)
POST /api/v1/queue/{id}/kafka-users/{userId}/rotate-passwordReturns a new password
DELETE /api/v1/queue/{id}/kafka-users/{userId}Deletes the user. It shows deleting, then leaves the list

Every change is recorded in the project's audit log, without the password.

Schema Registry

A cluster can have a Schema Registry: a Confluent-compatible schema registry (Karapace) for Avro, JSON Schema and Protobuf schemas, which Confluent's serializers and most Kafka tools speak to. It is an add-on at €4.99 a month, billed by the hour while it is enabled. It is available on request: ask support to turn it on for your account.

Enable it

In the console, open the cluster and turn on Schema Registry. From the API, POST /api/v1/queue/{id}/schema-registry. The cluster must be running. The registry is set up in the background and its status reads provisioning until it answers on its public address, usually within a few minutes, then ready.

Each registry has its own HTTPS endpoint, https://schema-<name>-<project>.queues.danubedata.ro, with a publicly trusted certificate. Clients sign in with HTTP Basic authentication: the username is registry, and the password is shown on the cluster page to anyone who may read the cluster's credentials (GET /api/v1/queue/{id}/schema-registry with queue:credentials).

The cluster's firewall does not cover the registry endpoint: it is reachable from anywhere, and TLS and the password protect it. Keep the password secret, and rotate it if it leaks.

Python (confluent-kafka)

Python
import os
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import MessageField, SerializationContext

registry = SchemaRegistryClient({
    "url": "https://schema-events-<project>.queues.danubedata.ro",
    "basic.auth.user.info": "registry:" + os.environ["SCHEMA_REGISTRY_PASSWORD"],
})
schema = '{"type": "record", "name": "Order", "fields": [{"name": "id", "type": "string"}]}'
serialize = AvroSerializer(registry, schema)

producer = Producer({
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "SCRAM-SHA-512",
    "sasl.username": os.environ["KAFKA_USER"],
    "sasl.password": os.environ["KAFKA_PASSWORD"],
})
producer.produce("orders", value=serialize({"id": "42"}, SerializationContext("orders", MessageField.VALUE)))
producer.flush()

Java

Add these to the producer or consumer properties of the Java client to use Confluent's serializers, for example io.confluent.kafka.serializers.KafkaAvroSerializer:

properties
schema.registry.url=https://schema-events-<project>.queues.danubedata.ro
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=registry:<password>

Good to know

  • The registry keeps its schemas in the cluster's _schemas topic. Do not write to it or delete it: that topic is the registry's data.
  • Rotate the password in the console or with POST /api/v1/queue/{id}/schema-registry/rotate-password, which answers the new one. The registry picks it up within a few minutes, without restarting; until then the old password still works.
  • Disable it in the console or with DELETE /api/v1/queue/{id}/schema-registry. Billing stops at once and the endpoint goes away, but the _schemas topic stays: enable the registry again and your subjects and schemas are back, with the same password.
  • Deleting the cluster deletes its registry and its schemas with it.

Metrics

In the console, open the cluster and use the Metrics tab. It shows the cluster's health with the reason for a warning, then its current figures:

  • CPU, as a share of the plan's vCPUs
  • Memory used, out of the broker's limit
  • Disk used, out of the plan's storage, with the percentage used
  • Messages in per second, and bytes in and out per second
  • Partitions, with the number that are under-replicated (highlighted when there are any)
  • Consumer lag: messages produced but not yet consumed, over every consumer group

Charts cover CPU, memory, disk (used against capacity), messages in, bytes in and out, and consumer lag over the window you pick (1 hour to 1 week). A Consumer groups table lists the 50 groups with the most lag, the largest first. A figure shows a dash while it is unknown, for example in the first minutes after the cluster starts. The Kafka list also shows each cluster's CPU, memory and total consumer lag.

From the API, GET /api/v1/queue/{id}/metrics returns the current values, as for RabbitMQ, with no time series:

FieldValue
providerkafka
summarycpu_percent, memory_used_bytes, memory_limit_bytes, disk_used_bytes, disk_capacity_bytes, messages_in_per_sec, bytes_in_per_sec, bytes_out_per_sec, partitions, under_replicated_partitions, consumer_lag_total
summary.consumer_groupsA list of {group, lag}, the largest lag first, at most 50
healthstatus (healthy, warning, critical or unknown) and issues, which say why

A value is null while it is unknown, which is not the same as zero.

Firewall

The cluster's default firewall opens only the Kafka port, to any address. Restrict it like any other firewall: edit its rules on the cluster's Security tab, for example to allow only your VPS instances or your office address. See Firewalls.

Authentication is required on every connection, but the endpoint is on the public internet until you narrow the firewall.

Delete a cluster

Deleting a Kafka cluster deletes its topics and all of its data. There are no snapshots to recover from. Export what you need first, for example by consuming the topics into your own storage.

In the console, open the cluster's Settings tab and choose Delete cluster (it is also in the More menu). From the CLI:

Bash
danube queue rm events