Kafka SSL Client Authentication in Multi-Tenancy Architecture

gowebing

Apache Kafka is the key product for not only messaging transformations but also real-time data processing, in addition to many other use cases. Architectures hosted inside the cloud claim to be secure in terms of communication and providing general security. But when it comes to the multiple client/consumer communication from a server/producer, Kafka provides in-built support for SSL as well as user-based authentication. In the below article, we will set up such an authentication mechanism step-by-step.

Kafka Server-Client SSL Authentication

The solution is divided into three parts: 

  1. SSL support for one or more brokers: Generate the key and the certificate for each machine in the cluster. You can use Java's KeyTool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.
  2. Kafka Configurations (We used Kafka 2.11-2.3.0).
  3. Running the whole set up.

Instructions to Install This Use Case

SSL support for one or more brokers. We will use Java's key tool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.

We are going to use one Kafka server and two clients (consumers). Also, here, we are using self-signed certificates. Otherwise, we need to have TrustStore and KeyStore JKSs for each server.

Points to note: 

  1. Please create a folder for creating and keeping all cert files.
  2. Please provide identical details and passwords for all. In my case I have used:
Plain Text


Only the CN for client2 I have given a different user for testing purposes. Please generate your certificate carefully, else there will be a problem in the next part.

Shell


Once everything is generated, you can see the generated files:

Plain Text


Kafka Configuration

Chang the server.properties file with below lines:

Properties files


Also add:

Properties files


Then, we can create the necessary new files: client-ssl.properties, client-ssl1.properties, and client-ssl2.properties inside kafka_2.11-2.3.0\config.

Properties files


Properties files


Properties files


First, run Kafka and ZooKeeper:

Shell


Then, open a new terminal and create a new topic:

Shell


After this, check the created certificate:

Shell


It will return the following details at the end. It means your certificate is generated properly.

Shell


We will run the set up for three different scenarios, i.e. without authentication, only server-side authentication, server, and client-side authentication.

Running the Whole Setup

The command for producing using console producer:

.\bin\windows\kafka-console-producer.bat --broker-list <broker host:port> --topic <topic-name> --producer.config config\<config file>

The command for consuming using console consumer:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server <server host:port> --topic <topic-name> --consumer.config config\<config file>

Without Authentication

Producer

Running Kafka producer

Consumer

Running Kafka consumer


Only Server-Side Authentication

Created another topic "test2".

Producer

Creating Kafka producer

Consumer 1

Creating first Kafka consumer

Consumer 2

Creating second Kafka consumer

Note: Please check the used config files.

Server and Client-Side Authentication

For authorization of topic – ‘test2’ only for the user – swarnava.c, use the two following commands:

Shell




Shell




Authorization of current topic


Producer

Creating Kafka producer

Consumer 1 (with User – swarnava.c)

Consumer 1 (with User – swarnava.c)

Consumer 2 (without User – swarnava.c)

Consumer 2 (without User – swarnava.c)


Congrats. You are done. This is the complete implementation of the SSL implementation in Kafka.

 

 

 

 

 

Top