Kafka Installation using Docker Compose

Environment

AWS EC2 t3.micro
Ubuntu 24.04 LTS
Docker Engine 28.4.0
Docker Compose v2.39.4

Insatll Docker and Docker Compose

sudo apt update &&\
sudo apt upgrade -y &&\
sudo apt install ca-certificates curl &&\
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc &&\
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null &&\
sudo apt update &&\
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Install Kafka using Docker Compose as a KRaft mode

mkdir kafka-docker-compose &&\
cd kafka-docker-compose

vi docker-compose.yml

version: '3.8'
services:
  kafka:
    image: confluentinc/cp-kafka:latest
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
      - "9093:9093"
    environment:
      KAFKA_HEAP_OPTS: "-Xmx512M -Xms256M"
      KAFKA_KRAFT_MODE: "true"  # Enables KRaft mode.
      KAFKA_PROCESS_ROLES: controller,broker  # Kafka acts as both controller and broker.
      KAFKA_NODE_ID: 1  # Unique ID for the Kafka instance.
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@localhost:9093"  # Controller quorum.
      KAFKA_LISTENERS: PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LOG_DIRS: /var/lib/kafka/data  # Log storage location.
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"  # Enables automatic topic creation.
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1  # Single replica for simplicity.
      KAFKA_LOG_RETENTION_HOURS: 168  # Log retention period (7 days).
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0  # No rebalance delay.
      CLUSTER_ID: "Mk3OEYBSD34fcwNTJENDM2Qk"  # Unique Kafka cluster ID.

sudo docker compose up -d

sudo docker ps

sudo docker exec -it kafka bash

Create a topic.

kafka-topics --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Send a message.

kafka-console-producer --topic test-topic --bootstrap-server localhost:9092

> This is a test

Subscribe messages at a new terminal window.

sudo docker exec -it kafka kafka-console-consumer --topic test-topic --bootstrap-server localhost:9092 --from-beginning