Grafana and Prometheus Setup With Strimzi, a.k.a. Kafka on Kubernetes.
In this article, we will see how quickly we can setup Grafana and Prometheus with Strimzi. So let us quickly go through the steps required for this setup.
1. Start Minikube.
Shell
x
1
$ minikube start -p strimzi-prometheus-grafana --cpus=4 --memory='10g'
2. Download Strimzi. We can follow steps mentioned in Strimzi docs.
Shell
xxxxxxxxxx
1
31
1
#Download the strimzi-x.y.z.zip file from GitHub.
2
#Latest release while writing this article is strimzi-0.20.zip.
3
[chandrashekhar@localhost strimzi-0.20.1]$ pwd
4
/home/chandrashekhar/SSD/Development_SSD/Streams_RH/strimzi-0.20.1
5
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl create ns kafka
6
namespace/kafka created
7
# Let us set context to kafka namespace so that we don't have to provide namespace for each command.
8
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl config set-context --current --namespace=kafka
9
Context "strimzi-prometheus-grafana" modified.
10
[chandrashekhar@localhost strimzi-0.20.1]$ ls -ltr
11
total 12
12
drwxr-xr-x. 6 chandrashekhar chandrashekhar 4096 Dec 14 20:11 install
13
drwxr-xr-x. 12 chandrashekhar chandrashekhar 4096 Dec 14 20:11 examples
14
drwxr-xr-x. 3 chandrashekhar chandrashekhar 4096 Dec 14 20:11 docs
15
[chandrashekhar@localhost strimzi-0.20.1]$ sed -i 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml
16
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f install/cluster-operator/
17
serviceaccount/strimzi-cluster-operator created
18
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
19
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
20
--------------
21
--------------
22
configmap/strimzi-cluster-operator created
23
deployment.apps/strimzi-cluster-operator created
24
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods
25
NAME READY STATUS RESTARTS AGE
26
strimzi-cluster-operator-545b7f4bf5-gktrg 0/1 ContainerCreating 0 2m51s
27
# wait for couple of minutes as it will download the images.
28
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods
29
NAME READY STATUS RESTARTS AGE
30
strimzi-cluster-operator-545b7f4bf5-gktrg 1/1 Running 0 5m48s
31
[chandrashekhar@localhost strimzi-0.20.1]$
3. Setup the Kafka Cluster now. You can find kafka-ephemeral.yaml with kafkaExporter and metrics configured.
Shell
xxxxxxxxxx
1
13
1
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f kafka-ephemeral.yaml
2
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods
3
NAME READY STATUS RESTARTS AGE
4
my-cluster-entity-operator-7f95fc64b5-wtmz5 3/3 Running 2 4m37s
5
my-cluster-kafka-0 1/1 Running 0 5m18s
6
my-cluster-kafka-1 1/1 Running 0 5m18s
7
my-cluster-kafka-2 1/1 Running 0 5m18s
8
my-cluster-kafka-exporter-56b65ff8b9-wxzc2 1/1 Running 0 45s
9
my-cluster-zookeeper-0 1/1 Running 0 13m
10
my-cluster-zookeeper-1 1/1 Running 0 13m
11
my-cluster-zookeeper-2 1/1 Running 0 13m
12
strimzi-cluster-operator-545b7f4bf5-gktrg 1/1 Running 0 29m
13
[chandrashekhar@localhost strimzi-0.20.1]$
4. Install the Prometheus operator and the Prometheus server instance. Some important modified YAML files can be found in my personnel GitHub link.
Shell
x
1
[chandrashekhar@localhost strimzi-0.20.1]$ curl -s https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml | sed -e 's/namespace: .*/namespace: kafka/' > prometheus-operator-deployment.yaml
2
# First remove line 1811 to 1829 from prometheus-operator-deployment.yaml.
3
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f prometheus-operator-deployment.yaml
4
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
5
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
6
---------
7
---------
8
service/prometheus-operator created
9
[chandrashekhar@localhost strimzi-0.20.1]$
10
11
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods|grep prometheus
12
prometheus-operator-79cd654746-qr4bf 1/1 Running 0 100s
13
[chandrashekhar@localhost strimzi-0.20.1]$ find . -name prometheus-additional.yaml
14
./examples/metrics/prometheus-additional-properties/prometheus-additional.yaml
15
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl create secret generic additional-scrape-configs --from-file=./examples/metrics/prometheus-additional-properties/prometheus-additional.yaml -n kafka
16
secret/additional-scrape-configs created
17
[chandrashekhar@localhost examples]$ cd ..
18
[chandrashekhar@localhost strimzi-0.20.1]$ find . -name strimzi-pod-monitor.yaml
19
./examples/metrics/prometheus-install/strimzi-pod-monitor.yaml
20
[chandrashekhar@localhost strimzi-0.20.1]$
21
[chandrashekhar@localhost strimzi-0.20.1]$ vi ./examples/metrics/prometheus-install/strimzi-pod-monitor.yaml
22
# Modify value of namespaceSelector.matchNames property to kafka in strimzi-pod-monitor.yaml. It exists in mutiple locations.
23
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f ./examples/metrics/prometheus-install/strimzi-pod-monitor.yaml
24
podmonitor.monitoring.coreos.com/cluster-operator-metrics created
25
podmonitor.monitoring.coreos.com/entity-operator-metrics created
26
podmonitor.monitoring.coreos.com/bridge-metrics created
27
podmonitor.monitoring.coreos.com/kafka-resources-metrics created
28
[chandrashekhar@localhost strimzi-0.20.1]$ find . -name prometheus-additional.yaml
29
./examples/metrics/prometheus-additional-properties/prometheus-additional.yaml
30
31
[chandrashekhar@localhost strimzi-0.20.1]$ find . -name prometheus-rules.yaml
32
./examples/metrics/prometheus-install/prometheus-rules.yaml
33
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f ./examples/metrics/prometheus-install/prometheus-rules.yaml
34
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
35
36
[chandrashekhar@localhost strimzi-0.20.1]$ find . -name prometheus.yaml
37
./examples/metrics/prometheus-install/prometheus.yaml
38
# Make sure that namespace in this prometheus.yaml is set to "kafka" namespace.
39
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f ./examples/metrics/prometheus-install/prometheus.yaml
40
clusterrole.rbac.authorization.k8s.io/prometheus-server created
41
serviceaccount/prometheus-server created
42
clusterrolebinding.rbac.authorization.k8s.io/prometheus-server created
43
prometheus.monitoring.coreos.com/prometheus created
44
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods|grep prometheus
45
prometheus-operator-79cd654746-qr4bf 1/1 Running 0 11m
46
prometheus-prometheus-0 2/2 Running 1 3m21s
47
[chandrashekhar@localhost strimzi-0.20.1]$
48
5. Check if Strimzi metrics are available.
Shell
xxxxxxxxxx
1
19
1
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods|grep exporter
2
my-cluster-kafka-exporter-56b65ff8b9-wxzc2 1/1 Running 0 25m
3
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl port-forward my-cluster-kafka-exporter-56b65ff8b9-wxzc2 9404:9404 -n kafka
4
Forwarding from 127.0.0.1:9404 -> 9404
5
Forwarding from [::1]:9404 -> 9404
6
7
# In a different shell execute
8
[chandrashekhar@localhost strimzi-0.20.1]$ curl http://localhost:9404/metrics|grep kafka_
9
% Total % Received % Xferd Average Speed Time Time Time Current
10
Dload Upload Total Spent Left Speed
11
100 5571 100 5571 0 0 81926 0 --:--:-- --:--:-- --:--:-- 81926
12
# HELP kafka_brokers Number of Brokers in the Kafka Cluster.
13
# TYPE kafka_brokers gauge
14
kafka_brokers 3
15
# HELP kafka_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which kafka_exporter was built.
16
# TYPE kafka_exporter_build_info gauge
17
kafka_exporter_build_info{branch="HEAD",goversion="go1.10.3",revision="830660212e6c109e69dcb1cb58f5159fe3b38903",version="1.2.0"} 1
18
[chandrashekhar@localhost strimzi-0.20.1]$
19
6. Now setup Grafana.
Shell
x
1
[chandrashekhar@localhost strimzi-0.20.1]$ find . -name grafana.yaml
2
./examples/metrics/grafana-install/grafana.yaml
3
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f ./examples/metrics/grafana-install/grafana.yaml
4
deployment.apps/grafana created
5
service/grafana created
6
# Prometheus service which will be used when we set up Grafana dashboards.
7
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get svc|grep prometheus
8
prometheus-operated ClusterIP None <none> 9090/TCP 10h
9
prometheus-operator ClusterIP None <none> 8080/TCP 10h
10
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get pods|grep grafana
11
grafana-7b7bfd6ffc-llf88 1/1 Running 1 6m32s
12
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl get svc|grep grafana
13
grafana ClusterIP 10.98.232.101 <none> 3000/TCP 8m8s
14
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl port-forward prometheus-prometheus-0 9090:9090
15
Forwarding from 127.0.0.1:9090 -> 9090
16
Forwarding from [::1]:9090 -> 9090
17
18
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl port-forward grafana-7b7bfd6ffc-llf88 3000:3000
19
Forwarding from 127.0.0.1:3000 -> 3000
20
Forwarding from [::1]:3000 -> 3000
21
22
7. Connect Grafana with Prometheus as a datasource. Note that in the previous step, we found that the Prometheus service is prometheus-operated on port 9090. Grafana will use this URL to scrap the Prometheus server.
8. Import Dashboard: In the "examples/metrics/grafana-dashboards" file, there are samples available for the Grafana Dashboard; we can reuse them.
Shell
xxxxxxxxxx
1
14
1
[chandrashekhar@localhost grafana-dashboards]$ pwd
2
/home/chandrashekhar/SSD/Development_SSD/Streams_RH/strimzi-0.20.1/examples/metrics/grafana-dashboards
3
[chandrashekhar@localhost grafana-dashboards]$ ls -ltr
4
total 352
5
-rw-r--r--. 1 chandrashekhar chandrashekhar 35654 Dec 14 20:11 strimzi-zookeeper.json
6
-rw-r--r--. 1 chandrashekhar chandrashekhar 39175 Dec 14 20:11 strimzi-operators.json
7
-rw-r--r--. 1 chandrashekhar chandrashekhar 17099 Dec 14 20:11 strimzi-kafka-mirror-maker-2.json
8
-rw-r--r--. 1 chandrashekhar chandrashekhar 68748 Dec 14 20:11 strimzi-kafka.json
9
-rw-r--r--. 1 chandrashekhar chandrashekhar 34571 Dec 14 20:11 strimzi-kafka-exporter.json
10
-rw-r--r--. 1 chandrashekhar chandrashekhar 27385 Dec 14 20:11 strimzi-kafka-connect.json
11
-rw-r--r--. 1 chandrashekhar chandrashekhar 73193 Dec 14 20:11 strimzi-kafka-bridge.json
12
-rw-r--r--. 1 chandrashekhar chandrashekhar 50404 Dec 14 20:11 strimzi-cruise-control.json
13
[chandrashekhar@localhost grafana-dashboards]$
14
I used strimzi-zookeeper.json, strimzi-kafka.json, and strimzi-kafka-exporter.json for this demonstration.
Import Dashboard 1
Import Dashboard 2
9. The below allows you to send/receive traffic in Kafka Brokers:
Shell
x
34
1
#create topic
2
[chandrashekhar@localhost strimzi-0.20.1]$ less topic.yaml
3
apiVersion: kafka.strimzi.io/v1beta1
4
kind: KafkaTopic
5
metadata:
6
name: my-topic
7
labels:
8
strimzi.io/cluster: "my-cluster"
9
spec:
10
partitions: 3
11
replicas: 2
12
13
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl apply -f topic.yaml
14
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl exec my-cluster-kafka-0 -- bin/kafka-topics.sh --list --bootstrap-server my-cluster-kafka-bootstrap:9092
15
my-topic
16
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl -n kafka exec --stdin --tty my-cluster-kafka-0 -- bash
17
[kafka@my-cluster-kafka-0 bin]$ ./kafka-console-producer.sh --bootstrap-server 0.0.0.0:9092 --topic my-topic
18
>hey
19
>csp
20
>1
21
>2
22
>3
23
>4
24
# In different terminal we can start consumer thread.
25
[chandrashekhar@localhost strimzi-0.20.1]$ kubectl -n kafka exec --stdin --tty my-cluster-kafka-0 -- bin/kafka-console-consumer.sh --bootstrap-server 0.0.0.0:9092 --group poc-grafana --topic my-topic
26
27
hey
28
csp
29
1
30
2
31
3
32
4
10. Finally, we can view all dashboards with runtime changes.
That's it guys. I hope you find it helpful. There is another article for Prometheus integrations with Java applications which you may find helpful too.
