diff --git a/metrics/README.md b/metrics/README.md index e1db5c0..96e0aba 100644 --- a/metrics/README.md +++ b/metrics/README.md @@ -15,10 +15,10 @@ is below: | ---- | ----------- | | collectd | `collectd` based statistics/metrics gathering daemonset code | | lib | General library helper functions for forming and launching workloads, and storing results in a uniform manner to aid later analysis | +| lib/cpu-load* | Routines to enable CPU load generation on a cluster | | report | Rmarkdown based report generator, used to produce a PDF comparison report of 1 or more sets of results | | scaling | Tests to measure scaling, such as linear or parallel launching of pods | - ## Results storage and analysis The tools generate JSON formatted results files via the `lib/json.bash` functions. The `metrics_json_save()` @@ -150,3 +150,11 @@ The `collectd` statistics are only configured and gathered if the environment va ### privileged statistics pods The privileged statistics pods `YAML` can be found in the `scaling/stats.yaml` file. An example of how to invoke and use this daemonset to extract statistics can be found in the `scaling/k8s_scale.sh` file. + +## Configuring constant 'loads' + +The framework includes some tooling to assist in setting up constant pre-defined 'loads' across the cluster to aid evaluation of their impacts on the scaling metrics. + +### CPU load generator + +Details of how to configure a constant CPU load are detailed in the [cpu-load documentation](lib/cpu-load.md). diff --git a/metrics/lib/cpu-load.bash b/metrics/lib/cpu-load.bash new file mode 100644 index 0000000..c503b7d --- /dev/null +++ b/metrics/lib/cpu-load.bash @@ -0,0 +1,90 @@ +#!/bin/bash +# +# Copyright (c) 2019 Intel Corporation +# +# SPDX-License-Identifier: Apache-2.0 + +# Helper routines for setting up a constant CPU load on the cluster/nodes + +CPULOAD_DIR=${THIS_FILE%/*} + +# Default to testing all cores +SMF_CPU_LOAD_NODES_NCPU=${SMF_CPU_LOAD_NODES_NCPU:-0} +# Default to 100% load (yes, this might kill your node) +SMF_CPU_LOAD_NODES_PERCENT=${SMF_CPU_LOAD_NODES_PERCENT:-} +# Default to not setting any limits or requests, so no cpuset limiting and +# no cpu core pinning +SMF_CPU_LOAD_NODES_LIMIT=${SMF_CPU_LOAD_NODES_LIMIT:-} +SMF_CPU_LOAD_NODES_REQUEST=${SMF_CPU_LOAD_NODES_REQUEST:-} + +cpu_load_post_deploy_sleep=${cpu_load_post_deploy_sleep:-30} + +cpu_per_node_daemonset=cpu-load +clean_up_cpu_per_node=false + +# Use a DaemonSet to place one cpu stressor on each node. +cpu_per_node_init() { + info "Generating per-node CPU load daemonset" + + local ds_template=${CPULOAD_DIR}/cpu_load_daemonset.yaml.in + local ds_yaml=${ds_template%\.in} + + # Grab a copy of the template + cp -f ${ds_template} ${ds_yaml} + + # If a setting is not used (defined), then delete its relevant + # lines from the YAML. Note, the YAML is constructed when necessary + # with comments on the correct lines to ensure all necessary lines are + # deleted + if [ -z "$SMF_CPU_LOAD_NODES_NCPU" ]; then + sed -i '/CPU_NCPU/d' ${ds_yaml} + fi + + if [ -z "${SMF_CPU_LOAD_NODES_PERCENT}" ]; then + sed -i '/CPU_PERCENT/d' ${ds_yaml} + fi + + if [ -z "${SMF_CPU_LOAD_NODES_LIMIT}" ]; then + sed -i '/CPU_LIMIT/d' ${ds_yaml} + fi + + if [ -z "${SMF_CPU_LOAD_NODES_REQUEST}" ]; then + sed -i '/CPU_REQUEST/d' ${ds_yaml} + fi + + # And then finally replace all the remaining defined parts with the + # real values. + sed -i \ + -e "s|@CPU_NCPU@|${SMF_CPU_LOAD_NODES_NCPU}|g" \ + -e "s|@CPU_PERCENT@|${SMF_CPU_LOAD_NODES_PERCENT}|g" \ + -e "s|@CPU_LIMIT@|${SMF_CPU_LOAD_NODES_LIMIT}|g" \ + -e "s|@CPU_REQUEST@|${SMF_CPU_LOAD_NODES_REQUEST}|g" \ + ${ds_yaml} + + # Launch the daemonset... + info "Deploying cpu-load-per-node daemonset" + kubectl apply -f ${ds_yaml} + kubectl rollout status --timeout=${wait_time}s daemonset/${cpu_per_node_daemonset} + clean_up_cpu_per_node=yes + info "cpu-load-per-node daemonset Deployed" + if [ -n "$cpu_load_post_deploy_sleep" ]; then + info "Sleeping ${cpu_load_post_deploy_sleep}s for cpu-load to settle" + sleep ${cpu_load_post_deploy_sleep} + fi +} + +cpu_load_init() { + info "Check if we need CPU load generators..." + # This is defaulted of off (not defined), unless the high level test requests it. + if [ -n "$SMF_CPU_LOAD_NODES" ]; then + info "Initialising per-node CPU load" + cpu_per_node_init + fi +} + +cpu_load_shutdown() { + if [ "$clean_up_cpu_per_node" = "yes" ]; then + info "Cleaning up cpu per node load daemonset" + kubectl delete daemonset --wait=true --timeout=${delete_wait_time}s "${cpu_per_node_daemonset}" || true + fi +} diff --git a/metrics/lib/cpu-load.md b/metrics/lib/cpu-load.md new file mode 100644 index 0000000..1593f43 --- /dev/null +++ b/metrics/lib/cpu-load.md @@ -0,0 +1,81 @@ +# `cpu-load` stack stresser + +The `cpu-load` stress functionality of the scaling framework allows you to optionally add a constant CPU stress +load to cluster under test whilst the tests are running. This aids impact analysis of CPU load. + +The `cpu-load` functionality utilises the [`stress-ng`](https://kernel.ubuntu.com/git/cking/stress-ng.git/) tool +to generate the CPU load. Some of the configuration parameters are taken directoy from the `stress-ng` command line. + +## Configuration + +`cpu-load` is configured via a number of environment variables: + +| Tool | Description | +| ---- | ----------- | +| collectd | `collectd` based statistics/metrics gathering daemonset code | +| lib | General library helper functions for forming and launching workloads, and storing results in a uniform manner to aid later analysis | +| report | Rmarkdown based report generator, used to produce a PDF comparison report of 1 or more sets of results | +| scaling | Tests to measure scaling, such as linear or parallel launching of pods | + +| Variable | Description | Default | +| -------- | ----------- | ------- | +| `SMF_CPU_LOAD_NODES` | Set to non-empty to deploy `cpu-load` stressor | unset (off) | +| `SMF_CPU_LOAD_NODES_NCPU` | Number of stressor threads to launch per node | 0 (one per cpu) | +| `SMF_CPU_LOAD_NODES_PERCENT` | Percentage of CPU to load | unset (100%) | +| `SMF_CPU_LOAD_NODES_LIMIT` | k8s cpu resource limit to set | unset (none) | +| `SMF_CPU_LOAD_NODES_REQUEST` | k8s cpu resource request to set | unset (none) | +| `cpu_load_post_deploy_sleep` | Seconds to sleep for `cpu-load` deployment to settle | 30 | + +`SMF_CPU_LOAD_NODES` must be set to a non-empty string to enable the `cpu-load` functionality. `cpu-load` uses +a daemonSet to deploy one `stress-ng` single container pod to each active node in the cluster. + + +Any of the `SMF_CPU_LOAD_NODES_*` variables can be set, or unset, and the daemonSet pods will be configured +appropriately. + +## Examples + +The combinations of settings available allow a lot of flexibility. Below are some common example setups: + +### 50% CPU load on all cores of all nodes (`stress-ng`) + +Here we allow `stress-ng` to spawn workers to cover all the CPUs on each node, but ask it to restrict its +bandwidth use to 50% of the CPU. We do not use the k8s limits. + +```bash +export SMF_CPU_LOAD_NODES=true +#export SMF_CPU_LOAD_NODES_NCPU= +export SMF_CPU_LOAD_NODES_PERCENT=50 +#export SMF_CPU_LOAD_NODES_LIMIT=999m +#export SMF_CPU_LOAD_NODES_REQUEST=999m +``` + +### 50% CPU load on 1 un-pinned core of all nodes (k8s `limits`) + +Here we set `stress-ng` to run a single worker thread at 100% CPU, but use the k8s resource limits to restrict +actual CPU usage to 50%. Because the k8s limit and request are not whole interger units, if the static policy is +in place on the k8s cluster, the pods will be classified as Guaranteed QoS, but will *not* get pinned to a specific +cpuset. + +```bash +export SMF_CPU_LOAD_NODES=true +export SMF_CPU_LOAD_NODES_NCPU=1 +export SMF_CPU_LOAD_NODES_PERCENT=100 +export SMF_CPU_LOAD_NODES_LIMIT=500m +export SMF_CPU_LOAD_NODES_REQUEST=500m +``` + +### 50% CPU load pinned to 1 core, on all nodes + +Here we set `stress-ng` to run a single worker thread at 50% CPU, and use the k8s resource limits to classify the +pod as Guaranteed, and as we are using whole integer units of CPU resource requests, if the static policy manager is +in play, the thread will be pinned to a single cpu cpuset. + +```bash +export SMF_CPU_LOAD_NODES=true +export SMF_CPU_LOAD_NODES_NCPU=1 +export SMF_CPU_LOAD_NODES_PERCENT=50 +export SMF_CPU_LOAD_NODES_LIMIT=1 +export SMF_CPU_LOAD_NODES_REQUEST=1 +``` + diff --git a/metrics/lib/cpu_load_daemonset.yaml.in b/metrics/lib/cpu_load_daemonset.yaml.in new file mode 100644 index 0000000..a9a7521 --- /dev/null +++ b/metrics/lib/cpu_load_daemonset.yaml.in @@ -0,0 +1,30 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: cpu-load +spec: + selector: + matchLabels: + name: cpu-load-pods + template: + metadata: + labels: + name: cpu-load-pods + spec: + hostNetwork: true + terminationGracePeriodSeconds: 0 + containers: + - name: cpu-load + imagePullPolicy: IfNotPresent + image: polinux/stress-ng + command: ["stress-ng"] + args: # comment fields here so we can *delete* sections on demand + - "--cpu" + - "@CPU_NCPU@" + - "-l" #CPU_PERCENT + - "@CPU_PERCENT@" #CPU_PERCENT + resources: + limits: + cpu: @CPU_LIMIT@ + requests: + cpu: @CPU_REQUEST@