Creating a Highly Available K3s Cluster
K3s is a lightweight Kubernetes distribution, which is easy to install. It is a fully compliant Kubernetes distribution with an embedded SQLite database as the default datastore and supports external datastore such as PostgreSQL, MySQL, and etcd. K3s includes a local storage provider, a service load balancer, a Helm controller, and the Traefik ingress controller. It also automates and manages complex cluster operations such as distributing certificates. K3s are easier to use, and more lightweight, with a binary size of less than 100 MB. In other words, you can run a highly available, certified Kubernetes distribution designed for production workloads on nanodes as well.
However, for a highly available K3s cluster with two master nodes, you can use an external datastore and an external load balancer for balancing the TCP traffic on 80/http, 443/https, 3306/mysql, and 6443/Kubernetes API.
K3s Highly Available Architecture
Infrastructure Requirements
- Two Linux nodes, typically virtual machines. You can deploy a K3s cluster on just about any flavor of Linux. However, K3s are officially tested on Ubuntu 16.04 and Ubuntu 18.04. Nanode costs just 5$ a month. In this tutorial these nodes are referred as master 1 and master 2.
- An external database to store the cluster data. MySQL is recommended. For more information, see installing MySQL. In this tutorial this external database is referred as mysql.example.com.
- A load balancer or NodeBalancer to direct traffic to the two nodes. Add the configurations for 80/http, 443/https, 3306/mysql, and 6443/Kubernetes API. For more information see, adding a TCP NodeBalancer. A NodeBalancer costs 10$ a month.
- A DNS record to map a URL to the NodeBalancer. This is the application server URL, and downstream Kubernetes clusters need to reach it. For more information, see adding a DNS record. In this tutorial the URL is referred as k3s.example.com.
Before You Begin
- Ensure that all the nodes are in the same region.
- Ensure that mysql.example.com and k3s.example.com resolve to the IP address of the NodeBalancer.
- Ensure that you can connect to the external database mysql.example.com from master 1 and master 2.
Creating a Kubernetes Cluster
1. Connect to master 1, and run the following commands to start the K3s server and connect it to the external datastore:
$export K3S_DATASTORE_ENDPOINT="mysql://user:password@tcp(mysql.example.com:3306/rancher
$curl -sfL https://get.k3s.io |
INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 -t agent-secret --tls-san
k3s.example.com --node-taint k3s-controlplane=true:NoExecute" sh -s -
server
Where:
- user: the user that you created on mysql.example.com
- password: the password that you set for the user
- rancher: the database that you created on mysql.example.com
- k3s.example.com: the URL to access the application server
- agent-secret: the secret token to use for authentication
2. Connect to master 2, and run the following commands to start the K3s server and connect it to the external datastore:
$export K3S_DATASTORE_ENDPOINT="mysql://user:password@tcp(mysql.example.com:3306/rancher
$curl -sfL https://get.k3s.io |
INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 -t agent-secret --tls-san
k3s.example.com --node-taint k3s-controlplane=true:NoExecute" sh -s -
server
3. To confirm that K3s has been set up successfully, run the following command on either of the K3s server nodes:
$sudo k3s kubectl get nodes
The output is similar to:
xxxxxxxxxx
NAME STATUS ROLES AGE VERSION
master Ready master 168m v1.18.2+k3s1
master2 Ready master 61m v1.18.2+k3s1
Result: You have successfully set up a highly available K3s Kubernetes cluster.
Note: Because K3s server nodes are schedulable by default, the minimum number of nodes for an HA K3s server cluster is two server nodes and zero agent nodes. However, you can always add agent nodes designated to run your apps and services to your cluster.
