Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation and in this tutorial we gonna to show how to install and configure Kafka on CentOS 7.
Imagine that when an event occurs you want to take an action. for example when a user logs in to a system, an email being sent to him/her.
In such situations, your software as PRODUCER sends login event to Kafka and your email sender script as consumer being notified about user login and sends email.

Here is our environment:

OS: CentOS 7 Linux on VMware
Firewall: firewalld
SElinux: permissive
IP Address: 192.168.147.128

1- Install Java

Kafka has been written in Scala and Java so first we have to install Java:

# yum install java-1.8.0-openjdk

2- Install and configure Kafka

Installing Kafka is pretty simple. just we need to download its package and make service and start it:

# cd /opt/ && wget https://mirror.ibcp.fr/pub/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz
# tar -xzf kafka_2.12-2.5.0.tgz
# mv kafka_2.12-2.5.0 kafka

Now we create service files for Kafka. Kafka uses ZooKeeper so we need to first start a ZooKeeper server:

# vim /lib/systemd/system/zookeeper.service

and we paste the following content in it:

[Unit]
Description=Zookeper daemon
After=network.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

then we start and enable service:

# systemctl enable zookeeper.service
# systemctl start zookeeper.service
# systemctl status zookeeper.service

Now we create Kafka service file and start it:

# vim /lib/systemd/system/kafka.service

and put the following content in it:

[Unit]
Description=Kafka daemon
After=zookeper.service

[Service]
Type=simple
ExecStart=/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

before starting Kafka service we need to put these lines at the end of server.properties file:

listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

then we start Kafka service:

# systemctl enable kafka.service
# systemctl start kafka.service
# systemctl status kafka.service

3- Kafka web UI

By default, Kafka has no web UI for better service management. so we install kafka-webview package as a web UI for Kafka management:

# cd /opt/ && wget https://github.com/SourceLabOrg/kafka-webview/releases/download/v2.5.1/kafka-webview-ui-2.5.1-bin.zip
# unzip kafka-webview-ui-2.5.1-bin.zip
# mv kafka-webview-ui-2.5.1-bin kafka-webview

then we create service file for it:

# vim /lib/systemd/system/kafka-webview.service

and put these content in it:

[Unit]
Description=Kafka Web UI daemon
After=kafka.service

[Service]
Type=simple
ExecStart=/opt/kafka-webview/start.sh
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

finally we start it:

# systemctl enable kafka-webview.service
# systemctl start kafka-webview.service
# systemctl status kafka-webview.service

4- Configure firewall

By default Zookeeper listens on port 2181 and Kafka listens on port 9092. Also web UI listens on port 8080. so we open these ports in firewall:

# firewall-cmd --permanent --add-port=2181/tcp
# firewall-cmd --permanent --add-port=9092/tcp
# firewall-cmd --permanent --add-port=8080/tcp
# firewall-cmd --reload

now we point to this address in browser:

http://192.168.147.128:8080