metrics: add cpu-load generator code and docs

Add library code that can generate a variety of cpu loads across the cluster. Configuration is via environment variables, documented in the .md file. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2026-09-05 13:21:33 +00:00 · 2019-11-15 12:23:38 -06:00
parent 39c7cc643a
commit c846e9753d
4 changed files with 210 additions and 1 deletions
@@ -15,10 +15,10 @@ is below:
 | ---- | ----------- |
 | collectd | `collectd` based statistics/metrics gathering daemonset code |
 | lib | General library helper functions for forming and launching workloads, and storing results in a uniform manner to aid later analysis |
+| lib/cpu-load* | Routines to enable CPU load generation on a cluster |
 | report | Rmarkdown based report generator, used to produce a PDF comparison report of 1 or more sets of results |
 | scaling | Tests to measure scaling, such as linear or parallel launching of pods |

-
 ## Results storage and analysis

 The tools generate JSON formatted results files via the `lib/json.bash` functions. The `metrics_json_save()`
@@ -150,3 +150,11 @@ The `collectd` statistics are only configured and gathered if the environment va
 ### privileged statistics pods

 The privileged statistics pods `YAML` can be found in the `scaling/stats.yaml` file. An example of how to invoke and use this daemonset to extract statistics can be found in the `scaling/k8s_scale.sh` file.
+
+## Configuring constant 'loads'
+
+The framework includes some tooling to assist in setting up constant pre-defined 'loads' across the cluster to aid evaluation of their impacts on the scaling metrics.
+
+### CPU load generator
+
+Details of how to configure a constant CPU load are detailed in the [cpu-load documentation](lib/cpu-load.md).
@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# Copyright (c) 2019 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Helper routines for setting up a constant CPU load on the cluster/nodes
+
+CPULOAD_DIR=${THIS_FILE%/*}
+
+# Default to testing all cores
+SMF_CPU_LOAD_NODES_NCPU=${SMF_CPU_LOAD_NODES_NCPU:-0}
+# Default to 100% load (yes, this might kill your node)
+SMF_CPU_LOAD_NODES_PERCENT=${SMF_CPU_LOAD_NODES_PERCENT:-}
+# Default to not setting any limits or requests, so no cpuset limiting and
+# no cpu core pinning
+SMF_CPU_LOAD_NODES_LIMIT=${SMF_CPU_LOAD_NODES_LIMIT:-}
+SMF_CPU_LOAD_NODES_REQUEST=${SMF_CPU_LOAD_NODES_REQUEST:-}
+
+cpu_load_post_deploy_sleep=${cpu_load_post_deploy_sleep:-30}
+
+cpu_per_node_daemonset=cpu-load
+clean_up_cpu_per_node=false
+
+# Use a DaemonSet to place one cpu stressor on each node.
+cpu_per_node_init() {
+	info "Generating per-node CPU load daemonset"
+
+	local ds_template=${CPULOAD_DIR}/cpu_load_daemonset.yaml.in
+	local ds_yaml=${ds_template%\.in}
+
+	# Grab a copy of the template
+	cp -f ${ds_template} ${ds_yaml}
+
+	# If a setting is not used (defined), then delete its relevant
+	# lines from the YAML. Note, the YAML is constructed when necessary
+	# with comments on the correct lines to ensure all necessary lines are
+	# deleted
+	if [ -z "$SMF_CPU_LOAD_NODES_NCPU" ]; then
+		sed -i '/CPU_NCPU/d' ${ds_yaml}
+	fi
+
+	if [ -z "${SMF_CPU_LOAD_NODES_PERCENT}" ]; then
+		sed -i '/CPU_PERCENT/d' ${ds_yaml}
+	fi
+
+	if [ -z "${SMF_CPU_LOAD_NODES_LIMIT}" ]; then
+		sed -i '/CPU_LIMIT/d' ${ds_yaml}
+	fi
+
+	if [ -z "${SMF_CPU_LOAD_NODES_REQUEST}" ]; then
+		sed -i '/CPU_REQUEST/d' ${ds_yaml}
+	fi
+
+	# And then finally replace all the remaining defined parts with the
+	# real values.
+	sed -i \
+		-e "s|@CPU_NCPU@|${SMF_CPU_LOAD_NODES_NCPU}|g" \
+		-e "s|@CPU_PERCENT@|${SMF_CPU_LOAD_NODES_PERCENT}|g" \
+		-e "s|@CPU_LIMIT@|${SMF_CPU_LOAD_NODES_LIMIT}|g" \
+		-e "s|@CPU_REQUEST@|${SMF_CPU_LOAD_NODES_REQUEST}|g" \
+		${ds_yaml}
+
+	# Launch the daemonset...
+	info "Deploying cpu-load-per-node daemonset"
+	kubectl apply -f ${ds_yaml}
+	kubectl rollout status --timeout=${wait_time}s daemonset/${cpu_per_node_daemonset}
+	clean_up_cpu_per_node=yes
+	info "cpu-load-per-node daemonset Deployed"
+	if [ -n "$cpu_load_post_deploy_sleep" ]; then
+		info "Sleeping ${cpu_load_post_deploy_sleep}s for cpu-load to settle"
+		sleep ${cpu_load_post_deploy_sleep}
+	fi
+}
+
+cpu_load_init() {
+	info "Check if we need CPU load generators..."
+	# This is defaulted of off (not defined), unless the high level test requests it.
+	if [ -n "$SMF_CPU_LOAD_NODES" ]; then
+		info "Initialising per-node CPU load"
+		cpu_per_node_init
+	fi
+}
+
+cpu_load_shutdown() {
+	if [ "$clean_up_cpu_per_node" = "yes" ]; then
+		info "Cleaning up cpu per node load daemonset"
+		kubectl delete daemonset --wait=true --timeout=${delete_wait_time}s "${cpu_per_node_daemonset}" || true
+	fi
+}
@@ -0,0 +1,81 @@
+# `cpu-load` stack stresser
+
+The `cpu-load` stress functionality of the scaling framework allows you to optionally add a constant CPU stress
+load to cluster under test whilst the tests are running. This aids impact analysis of CPU load.
+
+The `cpu-load` functionality utilises the [`stress-ng`](https://kernel.ubuntu.com/git/cking/stress-ng.git/) tool
+to generate the CPU load. Some of the configuration parameters are taken directoy from the `stress-ng` command line.
+
+## Configuration
+
+`cpu-load` is configured via a number of environment variables:
+
+| Tool | Description |
+| ---- | ----------- |
+| collectd | `collectd` based statistics/metrics gathering daemonset code |
+| lib | General library helper functions for forming and launching workloads, and storing results in a uniform manner to aid later analysis |
+| report | Rmarkdown based report generator, used to produce a PDF comparison report of 1 or more sets of results |
+| scaling | Tests to measure scaling, such as linear or parallel launching of pods |
+
+| Variable | Description | Default |
+| -------- | ----------- | ------- |
+| `SMF_CPU_LOAD_NODES` | Set to non-empty to deploy `cpu-load` stressor | unset (off) |
+| `SMF_CPU_LOAD_NODES_NCPU` | Number of stressor threads to launch per node | 0 (one per cpu) |
+| `SMF_CPU_LOAD_NODES_PERCENT` | Percentage of CPU to load | unset (100%) |
+| `SMF_CPU_LOAD_NODES_LIMIT` | k8s cpu resource limit to set | unset (none) |
+| `SMF_CPU_LOAD_NODES_REQUEST` | k8s cpu resource request to set | unset (none) |
+| `cpu_load_post_deploy_sleep` | Seconds to sleep for `cpu-load` deployment to settle | 30 |
+
+`SMF_CPU_LOAD_NODES` must be set to a non-empty string to enable the `cpu-load` functionality. `cpu-load` uses
+a daemonSet to deploy one `stress-ng` single container pod to each active node in the cluster.
+
+
+Any of the `SMF_CPU_LOAD_NODES_*` variables can be set, or unset, and the daemonSet pods will be configured
+appropriately.
+
+## Examples
+
+The combinations of settings available allow a lot of flexibility. Below are some common example setups:
+
+### 50% CPU load on all cores of all nodes (`stress-ng`)
+
+Here we allow `stress-ng` to spawn workers to cover all the CPUs on each node, but ask it to restrict its
+bandwidth use to 50% of the CPU. We do not use the k8s limits.
+
+```bash
+export SMF_CPU_LOAD_NODES=true
+#export SMF_CPU_LOAD_NODES_NCPU=
+export SMF_CPU_LOAD_NODES_PERCENT=50
+#export SMF_CPU_LOAD_NODES_LIMIT=999m
+#export SMF_CPU_LOAD_NODES_REQUEST=999m
+```
+
+### 50% CPU load on 1 un-pinned core of all nodes (k8s `limits`)
+
+Here we set `stress-ng` to run a single worker thread at 100% CPU, but use the k8s resource limits to restrict
+actual CPU usage to 50%. Because the k8s limit and request are not whole interger units, if the static policy is
+in place on the k8s cluster, the pods will be classified as Guaranteed QoS, but will *not* get pinned to a specific
+cpuset.
+
+```bash
+export SMF_CPU_LOAD_NODES=true
+export SMF_CPU_LOAD_NODES_NCPU=1
+export SMF_CPU_LOAD_NODES_PERCENT=100
+export SMF_CPU_LOAD_NODES_LIMIT=500m
+export SMF_CPU_LOAD_NODES_REQUEST=500m
+```
+
+### 50% CPU load pinned to 1 core, on all nodes
+
+Here we set `stress-ng` to run a single worker thread at 50% CPU, and use the k8s resource limits to classify the
+pod as Guaranteed, and as we are using whole integer units of CPU resource requests, if the static policy manager is
+in play, the thread will be pinned to a single cpu cpuset.
+
+```bash
+export SMF_CPU_LOAD_NODES=true
+export SMF_CPU_LOAD_NODES_NCPU=1
+export SMF_CPU_LOAD_NODES_PERCENT=50
+export SMF_CPU_LOAD_NODES_LIMIT=1
+export SMF_CPU_LOAD_NODES_REQUEST=1
+```
+
@@ -0,0 +1,30 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: cpu-load
+spec:
+  selector:
+    matchLabels:
+      name: cpu-load-pods
+  template:
+    metadata:
+      labels:
+        name: cpu-load-pods
+    spec:
+      hostNetwork: true
+      terminationGracePeriodSeconds: 0
+      containers:
+        - name: cpu-load
+          imagePullPolicy: IfNotPresent
+          image: polinux/stress-ng
+          command: ["stress-ng"]
+          args:                 # comment fields here so we can *delete* sections on demand
+            - "--cpu"
+            - "@CPU_NCPU@"
+            - "-l"              #CPU_PERCENT
+            - "@CPU_PERCENT@"   #CPU_PERCENT
+          resources:
+            limits:
+              cpu: @CPU_LIMIT@
+            requests:
+              cpu: @CPU_REQUEST@