You’re about to learn how to run Apache Kafka on Kubernetes in less than 10 minutes. This is a straightforward guide that has every example and solution you’ll need in case you’re struggling! Which you probably are already, that’s why you’re here!
So let’s understand the basics first!
Understanding Apache Kafka on Kubernetes?
Apache Kafka is a free, open-source platform for distributed event and stream processing, developed in Java. It is made to handle high-demand real-time data streams. The system is built to be fault-tolerant and can support many nodes in a cluster. To run a larger number of nodes effectively, it is important to use containerization and orchestration tools like Kubernetes for better resource management.
What Issues running Kafka on Kubernetes solve?
Digital leaders are using Apache Kafka and Kubernetes to analyze businesses in real-time, respond swiftly to market changes, and keep real-time connections with stakeholders, partners, suppliers, and customers. Kafka Streams helps the processing of Kafka events, allowing for quick reactions to real-time data.
Deploying Apache Kafka on a container orchestration platform such as Kubernetes enables automation, scaling, and deployment of event-driven applications anywhere.
Apache Kafka on Kubernetes serves as an excellent base for cloud-native development, supporting distributed streaming, real-time processing, and high scalability in event-driven applications.
Get exclusive access to all things tech-savvy, and be the first to receive
the latest updates directly in your inbox.
The Kafka Architecture in Simple terms!
- A Kafka cluster consists of one or more servers, known as brokers, that collaborate to manage incoming data streams and facilitate publish-subscribe messaging for Kafka clients, referred to as consumers.
- In a Kafka cluster, each data partition has a single leader broker and may have multiple follower brokers. The leader broker is responsible for all read and write operations for that partition, while each follower broker replicates the leader’s data passively.
- Typically, an open-source service called ZooKeeper is used in a Kafka setup to coordinate the clusters. This service helps select a leader among the brokers and initiates failover in the event of failures.
Step-by-Step Solution: Deploying Kafka on Kubernetes
Prerequisites
Choose an operator/chart: Strimzi operator (Kubernetes-native CRDs) or a Helm chart (Bitnami/Confluent). Strimzi is favored for Kafka on k8s as it offers CRs for Kafka, topics, users, etc.
Kubernetes cluster: Use a regional/multi-zone cluster for high availability (GKE recommends a regional cluster).
Tools: kubectl, helm (optional), gcloud/terraform (for infrastructure provisioning).
Storage: select a StorageClass (SSD) and ensure CSI snapshot support if you intend to snapshot PVs.
Namespaces & quotas: plan for a Kafka namespace, monitoring namespace, etc.
1) Provision cluster (high-level)
Option A — Terraform: set up a regional GKE cluster with node pools for:
- kafka-pool (high memory/disk, SSD)
- zookeeper-pool (if ZooKeeper is used)
- monitoring-pool (Prometheus/Grafana)
Option B — CLI: utilize gcloud container clusters create with –region and –node-pools.
Distribute nodes across zones and provide Kafka nodes with fast disks.
2) Create namespace, resource quotas, and node labels/taints
kubectl create namespace kafka<br>kubectl label node purpose=kafkaOptionally taint nodes and apply tolerations in manifests.
3) Install the operator or Helm chart
Operator (recommended): install Strimzi (the operator manages CRs for Kafka/ZK, users, topics).
kubectl apply -f (retrieve manifest from Strimzi documentation)Helm method: add the chart repository and helm install (if you prefer chart-based deployments like Bitnami).
Wait for operator pods to become ready.
4) Now Deploy Apache Kafka on Kubernetes (Simple Example)
Note: the following is a simplified CR to illustrate intent; modify it for the specific operator/version you are using.
Now we will create the deployment file:
<code><strong>kafka-deployment.yml</strong></code><mark>apiVersion:</mark><mark> </mark><mark>apps/v1</mark><mark><br></mark><mark>kind:</mark><mark> </mark><mark>StatefulSet</mark><mark><br></mark><mark>metadata:</mark><mark><br> </mark><mark>name:</mark><mark> </mark><mark>kafka</mark><mark><br> </mark><mark>namespace:</mark><mark> </mark><mark>kafka</mark><mark><br> </mark><mark>labels:</mark><mark><br> </mark><mark>app:</mark><mark> </mark><mark>kafka-app</mark><mark><br></mark><mark>spec:</mark><mark><br> </mark><mark>serviceName:</mark><mark> </mark><mark>kafka-svc</mark><mark><br> </mark><mark>replicas:</mark><mark> </mark><mark>3</mark><mark><br> </mark><mark>selector:</mark><mark><br> </mark><mark>matchLabels:</mark><mark><br> </mark><mark>app:</mark><mark> </mark><mark>kafka-app</mark><mark><br> </mark><mark>template:</mark><mark><br> </mark><mark>metadata:</mark><mark><br> </mark><mark>labels:</mark><mark><br> </mark><mark>app:</mark><mark> </mark><mark>kafka-app</mark><mark><br> </mark><mark>spec:</mark><mark><br> </mark><mark>containers:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>kafka-container</mark><mark><br> </mark><mark>image:</mark><mark> </mark><mark>doughgle/kafka-kraft</mark><mark><br> </mark><mark>ports:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>containerPort:</mark><mark> </mark><mark>9092</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>containerPort:</mark><mark> </mark><mark>9093</mark><mark><br> </mark><mark>env:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>REPLICAS</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>'3'</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>SERVICE</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>kafka-svc</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>NAMESPACE</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>kafka</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>SHARE_DIR</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>/mnt/kafka</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>CLUSTER_ID</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>bXktY2x1c3Rlci0xMjM0NQ==</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>DEFAULT_REPLICATION_FACTOR</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>'3'</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>DEFAULT_MIN_INSYNC_REPLICAS</mark><mark><br> </mark><mark>value:</mark><mark> </mark><mark>'2'</mark><mark><br> </mark><mark>volumeMounts:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>data</mark><mark><br> </mark><mark>mountPath:</mark><mark> </mark><mark>/mnt/kafka</mark><mark><br> </mark><mark>volumeClaimTemplates:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>metadata:</mark><mark><br> </mark><mark>name:</mark><mark> </mark><mark>data</mark><mark><br> </mark><mark>spec:</mark><mark><br> </mark><mark>accessModes:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>"ReadWriteOnce"</mark><mark><br> </mark><mark>resources:</mark><mark><br> </mark><mark>requests:</mark><mark><br> </mark><mark>storage:</mark><mark> </mark><mark>"1Gi"</mark><mark><br></mark><mark>---</mark><mark><br></mark><mark>apiVersion:</mark><mark> </mark><mark>v1</mark><mark><br></mark><mark>kind:</mark><mark> </mark><mark>Service</mark><mark><br></mark><mark>metadata:</mark><mark><br> </mark><mark>name:</mark><mark> </mark><mark>kafka-svc</mark><mark><br> </mark><mark>namespace:</mark><mark> </mark><mark>kafka</mark><mark><br> </mark><mark>labels:</mark><mark><br> </mark><mark>app:</mark><mark> </mark><mark>kafka-app</mark><mark><br></mark><mark>spec:</mark><mark><br> </mark><mark>type:</mark><mark> </mark><mark>NodePort</mark><mark><br> </mark><mark>ports:</mark><mark><br> </mark><mark>-</mark><mark> </mark><mark>name:</mark><mark> </mark><mark>'9092'</mark><mark><br> </mark><mark>port:</mark><mark> </mark><mark>9092</mark><mark><br> </mark><mark>protocol:</mark><mark> </mark><mark>TCP</mark><mark><br> </mark><mark>targetPort:</mark><mark> </mark><mark>9092</mark><mark><br> </mark><mark>nodePort:</mark><mark> </mark><mark>30092</mark><mark><br> </mark><mark>selector:</mark><mark><br> </mark><mark>app:</mark><mark> </mark><mark>kafka-app</mark>Setting up Zookeeper (or KRaft mode as an alternative)
ZooKeeper is a server for distributed coordination and synchronization of cloud applications. Kafka uses ZooKeeper to track the cluster node status and maintain lists of topics and messages.
Create a YAML file for the ZooKeeper stateful set:
nano zookeeper-stateful-set.yamlPaste the YAML code below:
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
service: zookeeper
name: zookeeper
spec:
serviceName: zookeeper
replicas: 1
selector:
matchLabels:
service: zookeeper
template:
metadata:
labels:
network/kafka-network: "true"
service: zookeeper
spec:
securityContext:
fsGroup: 1000
enableServiceLinks: false
containers:
- name: zookeeper
imagePullPolicy: Always
image: zookeeper
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"
- name: ZOOKEEPER_DATA_DIR
value: "/var/lib/zookeeper/data"
- name: ZOOKEEPER_LOG_DIR
value: "/var/lib/zookeeper/log"
- name: ZOOKEEPER_SERVER_ID
value: "1"
resources: {}
volumeMounts:
- mountPath: /var/lib/zookeeper/data
name: zookeeper-data
- mountPath: /var/lib/zookeeper/log
name: zookeeper-log
hostname: zookeeper
restartPolicy: Always
volumeClaimTemplates:
- metadata:
name: zookeeper-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1024Mi
- metadata:
name: zookeeper-log
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1024MiSave and exit the file.
The code defines the deployment of ZooKeeper with one replica and connects it with the previously defined kafka-network. It also provides the ZooKeeper container image and reserves volume storage for the application’s data and logs.
Now you can test the successfully created service as follows:
$ kubectl get services -n kafka
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
zookeeper-service NodePort 10.100.69.243 <none> 2181:30181/TCP 3m4s
Test Kafka Deployment ( The Final Step)
Test the Kafka deployment with kcat, a generic non-JVM producer and consumer application. Follow the steps below to deploy kcat and use it to test Kafka:
1. Create a YAML file:
nano kcat-deployment.yaml2. Paste the code below to create a kcat deployment:
kind: Deployment
apiVersion: apps/v1
metadata:
name: kcat
labels:
app: kcat
spec:
selector:
matchLabels:
app: kcat
template:
metadata:
labels:
app: kcat
spec:
containers:
- name: kcat
image: edenhill/kcat:1.7.0
command: ["/bin/sh"]
args: ["-c", "trap : TERM INT; sleep 1000 & wait"]Wrapping Up!
That’s everything you need to learn to run Apache Kafka on Kubernetes in the most straightforward way.
It can be complex at first, but with the right setup, it turns into a reliable and scalable solution for handling real-time data.
Kubernetes takes away much of the manual effort by providing automation, high availability, and resilience, while tools like StatefulSets, operators, and monitoring stacks make Kafka management far smoother than traditional deployments. Whether you choose Autopilot for simplicity or Standard mode for full control, Kafka on Kubernetes ensures you’re ready to handle production workloads with confidence.
The bottom line: if your goal is scalability, fault tolerance, and long-term operational ease, deploying Kafka on Kubernetes is the way forward.
FAQ’s
1. Why run Kafka on Kubernetes instead of VMs or bare metal?
Kubernetes offers self-healing, scaling, and automation features that make Kafka easier to manage, especially in high-availability setups.
2. Do I still need ZooKeeper for Kafka on Kubernetes?
It depends on the Kafka version. Traditional Kafka clusters need ZooKeeper, but newer versions (2.8+) can use KRaft mode, which removes the ZooKeeper dependency.
3. What’s the best way to expose Kafka to external clients?
Use a LoadBalancer or Ingress on GKE, but configure advertised.listeners carefully so clients can connect reliably.
4. How do I ensure data persistence for Kafka brokers?
Use PersistentVolumeClaims with a fast storage class (like SSD). This ensures logs and partitions survive pod restarts or node failures
5. How do I monitor Kafka running on Kubernetes?
Set up Prometheus and Grafana with JMX exporters. Track metrics like under-replicated partitions, ISR size, and request latency.
6. Which deployment method should I choose: Helm chart or Operator?
Operators like Strimzi simplify lifecycle management (upgrades, scaling, users, topics). Helm charts are quicker for simple setups but require more manual management.
