cloud-native-setup/clr-k8s-examples

How to setup the cluster

Sample multi-node vagrant setup

To be able to test this tool, you can create a 3-node vagrant setup. In this tutorial, we will talk about using libvirt, but you can use any hypervisor that you are familiar with.

• Install vagrant on the distro you are using. Steps can be found at Vagrant docs
• vagrant up --provider=libvirt

Now you have a 3 node cluster up and running. Each of them have 2 vCPU, 4GB Memory, 2x10GB disks, 1 additional private network.

Setup the nodes in the cluster

Run setup_system.sh once on each and every node (master and workers) to ensure k8s works on it.

This script ensures the following

• Installs the bundles the Clearlinux needs to support Kubernetes, CRIO and Kata
• Customizes the system to ensure correct defaults are setup (IP Forwarding, Swap off,...)
• Ensures all the dependencies are loaded on boot (kernel modules)

Bring up the master

Run create_stack.sh on the master node. This sets up the master and also uses kubelet config via kubeadm.yaml to propagate cluster wide kubelet configuration to all workers. Customize it if you need to setup other cluster wide properties.

Join Workers to the cluster

kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert-hash <hash> --cri-socket=/run/crio/crio.sock

Note: Remember to append --cri-socket=/run/crio/crio.sock to the join command generated by the master.

On workers just use the join command that the master spits out. There nothing else you need to run on the worker. All the other k8s customizations are pushed in from master via the values setup in the kubeadm.yaml file.

So if you want to customize the kubelet on the master or the workers (things like resource reservations etc), update this file (when the cluster is created). The master will push this configuration automatically to every worker node that joins in.

Running Kata Workloads

The cluster is setup out of the box to support Kata via runtime class. Clearlinux will also setup kata automatically on all nodes. So running a workload with runtime class set to "kata" will launch the POD/Deployment with Kata.

An example is

kubectl apply -f tests/test-deploy-kata.yaml

Making Kata the default runtime

Today in crio.conf runc is the default runtime when a user does not specify runtimeClass in the pod spec. If you want to run a cluster where kata is used by default, except for workloads we know for sure will not work with kata, use the admission webhook and sample admission controller we created by running -

kubectl apply -f admit-kata/

The admission webhook is setup to exclude certian namespaces from being run with Kata using filters on namespace labels.

    namespaceSelector:
      matchExpressions:
        -  {key: "kata", operator: NotIn, values: ["false"]}

The rook operators for example are marked as such

apiVersion: v1
kind: Namespace
metadata:
  name: rook-ceph-system
  labels:
    kata: "false"

Pods not explicitly excluded by the namespace filter are dynamically tagged to run with Kata with some exceptions -

• hostNetwork: true
• rook-ceph and rook-ceph-system namespaces (buggy)

Other pod properties will be added as exceptions in future.

Accessing control plane services

Pre-req

You need to have credentials of the cluster, on the computer you will be accessing the control plane services from. If it is not under $HOME/.kube, set KUBECONFIG environment variable for kubectl to find.

Dashboard

kubectl proxy # starts serving on 127.0.0.1:8001

Dashboard is available at this URL http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

Kibana

Start proxy same as above. Kibana is available at this URL http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana

Grafana

kubectl -n monitoring port-forward svc/grafana 3000

Grafana is available at this URL http://localhost:3000 . Default credentials are admin/admin. Upon entering you will be asked to chose a new password.

Cleaning up the cluster (Hard reset to a clean state)

Run reset_stack.sh on all the nodes