# How to setup the cluster ## Prerequisite This setup currently will work with Kubernetes 1.14 & above. Any version of Kubernetes before that might work, but is not guaranteed. ## Sample multi-node vagrant setup To be able to test this tool, you can create a 3-node vagrant setup. In this tutorial, we will talk about using [libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), but you can use any hypervisor that you are familiar with. ## Install vagrant Follow instructions in the [Vagrant docs](https://www.vagrantup.com/intro/getting-started/install.html#installing-vagrant) Or, follow our [detailed steps](vagrant.md) Now you have a 3 node cluster up and running. Each of them have 4 vCPU, 8GB Memory, 2x10GB disks, 1 additional private network. Customize the setup using environment variables. E.g., `NODES=2 MEMORY=16384 CPUS=8 vagrant up --provider=libvirt` To login to the master node and change to this directory ```bash vagrant ssh clr-01 cd clr-k8s-examples ``` ## Setup the nodes in the cluster Run [`setup_system.sh`](setup_system.sh) once on each and every node (master and workers) to ensure Kubernetes works on it. This script ensures the following * Installs the bundles the Clearlinux needs to support Kubernetes, CRIO and Kata * Customizes the system to ensure correct defaults are setup (IP Forwarding, Swap off,...) * Ensures all the dependencies are loaded on boot (kernel modules) > NOTE: This step is done automatically if using vagrant. The [`setup_system.sh`](setup_system.sh) script uses the runtime specified in the `RUNNER` environment variable and defaults to `crio`. To use the `containerd` runtime, set the `RUNNER` environment variable to `containerd`. ### Specify a version of Clear Linux To specify a particular version of Clear Linux to use, set the CLRK8S_CLR_VER environment variable to the desired version before starting setup_system.sh (e.g. `CLRK8S_CLR_VER=31400 ./setup_system.sh`) ### Configuration for high numbers of pods per node In order to enable running greater than 110 pods per node, set the environment variable `HIGH_POD_COUNT` to any non-empty value. > NOTE: Use this configuration when utilizing the [metrics](../metrics) tooling in this repo. ### Enabling experimental firecracker support > EXPERIMENTAL: Optionally run [`setup_kata_firecracker.sh`](setup_kata_firecracker.sh) to be able to use firecracker VMM with Kata. The firecracker setup switches the setup to use a sparse file backed loop device for devicemapper storage. This should not be used for production. > NOTE: This step is done automatically if using vagrant. ## Bring up the master Run [`create_stack.sh`](create_stack.sh) on the master node. This sets up the master and also uses kubelet config via [`kubeadm.yaml`](kubeadm.yaml) to propagate cluster wide kubelet configuration to all workers. Customize it if you need to setup other cluster wide properties. There are different flavors to install, run `./create_stack.sh help` to get more information. > NOTE: Before running [`create_stack.sh`](create_stack.sh) script, make sure to export the necessary environment variables if needed to be changed. By default it will use `CLRK8S_CNI` to be canal, and `CLRK8S_RUNNER` to be crio. Cilium is tested only in the Vagrant. ```bash # default shows help ./create_stack.sh ``` In order to enable running greater than 110 pods per node, set the environment variable `HIGH_POD_COUNT` to any non-empty value. ## Join Workers to the cluster ```bash kubeadm join : --token --discovery-token-ca-cert-hash --cri-socket=/run/crio/crio.sock ``` Note: Remember to append `--cri-socket=/run/crio/crio.sock` to the join command generated by the master. On workers just use the join command that the master spits out. There nothing else you need to run on the worker. All the other Kubernetes customizations are pushed in from master via the values setup in the `kubeadm.yaml` file. So if you want to customize the kubelet on the master or the workers (things like resource reservations etc), update this file (when the cluster is created). The master will push this configuration automatically to every worker node that joins in. ## Running Kata Workloads The cluster is setup out of the box to support Kata via runtime class. Clearlinux will also setup kata automatically on all nodes. So running a workload with runtime class set to "kata" will launch the POD/Deployment with Kata. An example is `kubectl apply -f tests/deploy-svc-ing/test-deploy-kata-qemu.yaml` ### Running Kata Workloads with Firecracker > EXPERIMENTAL: If firecracker setup has been enabled, runtime class set to "kata-fc" will launch the POD/Deployment with firecracker as the isolation mechanism for Kata. An example is `kubectl apply -f tests/deploy-svc-ing/test-deploy-kata-fc.yaml` ## Making Kata the default runtime using admission controller If you want to run a cluster where kata is used by default, except for workloads we know for sure will not work with kata, using [admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) and sample admission controller, follow `admit-kata` [README.md](admit-kata/README.md) ## Accessing control plane services ### Pre-req You need to have credentials of the cluster, on the computer you will be accessing the control plane services from. If it is not under `$HOME/.kube`, set `KUBECONFIG` environment variable for `kubectl` to find. ### Dashboard ```bash kubectl proxy # starts serving on 127.0.0.1:8001 ``` Dashboard is available at this URL http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy ### Kibana Start proxy same as above. Kibana is available at this URL http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana ### Grafana ```bash kubectl -n monitoring port-forward svc/grafana 3000 ``` Grafana is available at this URL http://localhost:3000 . Default credentials are `admin/admin`. Upon entering you will be asked to chose a new password. ## Cleaning up the cluster (Hard reset to a clean state) Run `reset_stack.sh` on all the nodes