Use KubeKey To Set Up a Kubernetes and KubeSphere Cluster With NFS Storage

Introduction

In my previous articles, I talked about how to use KubeKey to set up and scale a Kubernetes cluster. As you may already know, KubeKey can do way more than that. You can also use KubeKey to install KubeSphere, a container platform running on top of Kubernetes with streamlined DevOps workflows, unified multi-cluster management, and more. Besides, KubeKey is able to install cloud-native add-ons by Chart or YAML files.

Among other things, storage represents one of, if not the most important element as you set up a Kubernetes cluster. Kubernetes itself supports multiple storage solutions, such as NFS-client, Ceph CSI, and GlusterFS (in-tree). In this article, I am going to show you how to use KubeKey to create a Kubernetes and KubeSphere cluster with NFS-client Provisioner providing external storage.

What Is NFS?

Network File System (NFS) provides you with remote access to files and directories on a server machine. It allows multiple clients to access shared resources over a network.

You can install the NFS-client Provisioner as a storage plugin for your Kubernetes cluster while you must configure an NFS server beforehand. Here is how it works: an NFS client mounts a directory on the server machine so that files residing on the NFS server are accessible to the NFS client. The directory, which you create on the server and share with your client, is also called an exported directory. To ensure that your client can access the directory smoothly, you must grant access to them with several commands which I will show you later.

At the same time, a component called nfs-common (for Debian-based Linux distributions) or nfs-utils (for CentOS-based Linux distributions) needs to be installed on all the client machines to provide essential NFS client libraries and utilities.

How Does KubeKey Use NFS-client Configurations?

As I said above, you can use KubeKey to install different add-ons by Chart or YAML files. The configuration of these add-ons must be offered in the configuration file (config-sample.yaml by default) created by KubeKey. To let KubeKey apply these configurations during installation, there are generally two ways:

  1. Input necessary parameters under the addons field directly in config-sample.yaml.
  2. Create a separate configuration file for your add-on to list all the necessary parameters and provide the path of the file in config-sample.yaml so that KubeKey can reference it.

In this article, I will use the second way for demonstration. The general steps are:

  1. Set up an NFS server.
  2. Install nfs-common or nfs-utils on all the client machines in your cluster and create a separate configuration file for NFS-client Provisioner on one of the client machines that servers as the taskbox for installation.
  3. Download KubeKey on the taskbox.
  4. Use KubeKey to create your cluster configuration file (config-sample.yaml) and edit it.
  5. Install Kubernetes, KubeSphere, and NFS-client Provisioner including the storage class.

Prepare Hosts

Here is my node information for your reference:

Host IP Hostname Role System
192.168.0.2 server NFS server Ubuntu 16.04, 4 Cores, 4 G Memory, 100 G Disk
192.168.0.3 client1 master, etcd Ubuntu 16.04, 8 Cores, 8 G Memory, 50 G Disk
192.168.0.4 client2 worker Ubuntu 16.04, 8 Cores, 8 G Memory, 50 G Disk
192.168.0.5 client3 worker Ubuntu 16.04, 8 Cores, 8 G Memory, 50 G Disk

The server machine, where the NFS server will be installed, provides external storage services for three client machines in the cluster.

For more information about requirements for nodes, networks, and dependencies, see one of my previous posts.

Install and Configure an NFS Server

Step 1: Install the NFS Kernel Server

To set up your server machine, you must install the NFS kernel server on it.

Run the following command so that you will be using the latest package on Ubuntu for installation.

XML
 




x


1
sudo apt-get update



Install the NFS kernel server.

XML
 




xxxxxxxxxx
1


1
sudo apt install nfs-kernel-server



Step 2: Create an Exported Directory

Your NFS client will mount a directory on the server machine which has been exported by the NFS server.

Run the following command as root to specify a mount folder name (for example, /mnt/demo). This is also the directory that will be shared with your client machines.

XML
 




xxxxxxxxxx
1


1
sudo mkdir -p /mnt/demo



To ensure that all the clients can access the directory, remove permissions of the folder.

XML
 




xxxxxxxxxx
1


1
sudo chown nobody:nogroup /mnt/demo


XML
 




xxxxxxxxxx
1


1
sudo chmod 777 /mnt/demo



Step 3: Grant Your Client Machine Access To the NFS Server

Run the following command to edit the /etc/exports file with nano.

XML
 




xxxxxxxxxx
1


1
sudo nano /etc/exports



Add your client information to the file. Note that if you have multiple directories that you want to share with your clients, you need to add them all to the file. Here is the syntax:

XML
 




xxxxxxxxxx
1


1
/mnt/demo clientIP(rw,sync,no_subtree_check)



If you have multiple client machines, add each line for all of them. Alternatively, specify a subnet in the file so that all the clients within it can access the NFS server. For example:

XML
 




xxxxxxxxxx
1


1
/mnt/demo 192.168.0.0/24(rw,sync,no_subtree_check)



Note:

Step 4: Apply the Configuration

Run the following command to export your shared directory.

XML
 




xxxxxxxxxx
1


1
sudo exportfs -a



To make your configuration effective, restart the NFS kernel server.

XML
 




xxxxxxxxxx
1


 
1
sudo systemctl restart nfs-kernel-server


Configure the Client Machine

Now that we have our server machine ready, we need to install nfs-common on all of our clients. It provides necessary NFS functions while you do not need to install any server components.

Likewise, execute the following command to make sure you are using the latest package.

XML
 




xxxxxxxxxx
1


1
sudo apt-get update



Install nfs-common on all the clients.

XML
 




xxxxxxxxxx
1


1
sudo apt-get install nfs-common



Go to one of the client machines (taskbox) where you want to download KubeKey later (for example, client1). Create a configuration file that contains all the necessary parameters of your NFS server which will be referenced by KubeKey during installation.

XML
 




xxxxxxxxxx
1


 
1
vi nfs-client.yaml



Here is my configuration for your reference:

XML
 




xxxxxxxxxx
1


1
nfs:
2
  server: "192.168.0.2"    # This is the server IP address. Replace it with your own.
3
  path: "/mnt/demo"    # Replace the exported directory with your own.
4
storageClass:
5
  defaultClass: false



Note

Download KubeKey and Install a Cluster

Download KubeKey from its GitHub Release Page or use the following command to download KubeKey version 1.0.1. You only need to download KubeKey to one of your machines that serves as the taskbox for installation.

XML
 




xxxxxxxxxx
1


1
curl -sfL https://get-kk.kubesphere.io | VERSION=v1.0.1 sh -



The above command downloads KubeKey and unzips the file. Your folder now contains a file called kk. Make it executable.

XML
 




xxxxxxxxxx
1


1
chmod +x kk



Specify a Kubernetes version and a KubeSphere version that you want to install. For more information about supported Kubernetes versions, see this list.

XML
 




xxxxxxxxxx
1


1
./kk create config --with-kubernetes v1.17.9 --with-kubesphere v3.0.0



A default file config-sample.yaml will be created if you do not customize the name. Edit the file.

XML
 




xxxxxxxxxx
1


1
vi config-sample.yaml


YAML
 




xxxxxxxxxx
1
40


 
1
...
2
metadata:
3
  name: sample
4
spec:
5
  hosts:
6
  - {name: client1, address: 192.168.0.3, internalAddress: 192.168.0.3, user: ubuntu, password: Testing123}
7
  - {name: client2, address: 192.168.0.4, internalAddress: 192.168.0.4, user: ubuntu, password: Testing123}
8
  - {name: client3, address: 192.168.0.5, internalAddress: 192.168.0.5, user: ubuntu, password: Testing123}
9
  roleGroups:
10
    etcd:
11
    - client1
12
    master:
13
    - client1
14
    worker:
15
    - client2
16
    - client3
17
  controlPlaneEndpoint:
18
    domain: lb.kubesphere.local
19
    address: ""
20
    port: "6443"
21
  kubernetes:
22
    version: v1.17.9
23
    imageRepo: kubesphere
24
    clusterName: cluster.local
25
  network:
26
    plugin: calico
27
    kubePodsCIDR: 10.233.64.0/18
28
    kubeServiceCIDR: 10.233.0.0/18
29
  registry:
30
    registryMirrors: []
31
    insecureRegistries: []
32
  addons:
33
  - name: nfs-client
34
    namespace: kube-system
35
    sources:
36
      chart:
37
        name: nfs-client-provisioner
38
        repo: https://charts.kubesphere.io/main
39
        values: /home/ubuntu/nfs-client.yaml # Use the path of your own NFS-client configuration file.
40
...             



Pay special attention to the field of addons. For more information about each parameter in this file, you can see one of my previous posts, or have a look at this example file and explanations for the addons field. Note that you can also enable pluggable components of KubeSphere in this file, such as DevOps, service mesh, and App Store.

Save the file and execute the following command to install Kubernetes and KubeSphere:

XML
 




xxxxxxxxxx
1


1
./kk create cluster -f config-sample.yaml



When the installation finishes, you can inspect installation logs with the following command:

XML
 




xxxxxxxxxx
1


1
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f



You can see the following message if the installation is successful:

YAML
 




xxxxxxxxxx
1
19


1
#####################################################
2
###              Welcome to KubeSphere!           ###
3
#####################################################
4
   
5
Console: http://192.168.0.3:30880
6
Account: admin
7
Password: P@88w0rd
8
   
9
NOTES:
10
  1. After you log into the console, please check the
11
     monitoring status of service components in
12
     "Cluster Management". If any service is not
13
     ready, please wait patiently until all components 
14
     are up and running.
15
  2. Please change the default password after login.
16
   
17
#####################################################
18
https://kubesphere.io             20xx-xx-xx xx:xx:xx
19
#####################################################



Verify Installation

You can verify that NFS-client has been successfully installed either from the command line or from the KubeSphere web console.

Command-Line

Run the following command to check your storage class.

XML
 




xxxxxxxxxx
1


1
kubectl get sc



I did not set nfs-client as the default storage class so KubeKey installed OpenEBS for me as well. Expected output:

XML
 




xxxxxxxxxx
1


1
NAME              PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
2
local (default)   openebs.io/local                                  Delete          WaitForFirstConsumer   false                  16m
3
nfs-client        cluster.local/nfs-client-nfs-client-provisioner   Delete          Immediate              true                   16m



Run the following command to check the statuses of Pods.

XML
 




xxxxxxxxxx
1


1
kubectl get pod -n kube-system



Note that nfs-client is installed in the namespace kube-system. Expected output (exclude irrelevant Pods):

XML
 




xxxxxxxxxx
1


1
NAME                                                 READY   STATUS    RESTARTS   AGE
2
nfs-client-nfs-client-provisioner-6fc95f4f79-92lsh   1/1     Running   0          16m



KubeSphere Console

The ks-console Service is being exposed through a NodePort. Log in to the console at <node IP>:30880 with the default account and password (admin/P@88w0rd). You may need to open the port in your security groups and configure relevant port forwarding rules depending on your environment.

Click Platform in the top left corner and go to Cluster Management. In Storage Classes under Storage, you can see two storage classes:

Go to Pods under Application Workloads, the Pod of nfs-client is also functioning well in the kube-system namespace.

You can create a PVC in a project and verify the persistent volume bound to it can be mounted to a workload successfully.

For more information about how to create a PVC on the KubeSphere console, see Volumes.

 

 

 

 

Top