dlan17/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'pm-cpufreq'

Browse Source 
Merge cpufreq updates for 6.16-rc1:

 - Refactor cpufreq_online(), add and use cpufreq policy locking guards,
   use __free() in policy reference counting, and clean up core cpufreq
   code on top of that (Rafael Wysocki).

 - Fix boost handling on CPU suspend/resume and sysfs updates (Viresh
   Kumar).

 - Fix des_perf clamping with max_perf in amd_pstate_update() (Dhananjay
   Ugwekar).

 - Add offline, online and suspend callbacks to the amd-pstate driver,
   rename and use the existing amd_pstate_epp callbacks in it (Dhananjay
   Ugwekar).

 - Add support for the "Requested CPU Min frequency" BIOS option to the
   amd-pstate driver (Dhananjay Ugwekar).

 - Reset amd-pstate driver mode after running selftests (Swapnil
   Sapkal).

 - Add helper for governor checks to the schedutil cpufreq governor and
   move cpufreq-specific EAS checks to cpufreq (Rafael Wysocki).

 - Populate the cpu_capacity sysfs entries from the intel_pstate driver
   after registering asym capacity support (Ricardo Neri).

 - Add support for enabling Energy-aware scheduling (EAS) to the
   intel_pstate driver when operating in the passive mode on a hybrid
   platform (Rafael Wysocki).

 - Avoid shadowing ret in amd_pstate_ut_check_driver() (Nathan
   Chancellor).

 - Drop redundant cpus_read_lock() from store_local_boost() in the
   cpufreq core (Seyediman Seyedarab).

 - Replace sscanf() with kstrtouint() in the cpufreq code and use a
   symbol instead of a raw number in it (Bowen Yu).

 - Add support for autonomous CPU performance state selection to the
   CPPC cpufreq driver (Lifeng Zheng).

* pm-cpufreq: (31 commits)
  cpufreq: CPPC: Add support for autonomous selection
  cpufreq: Update sscanf() to kstrtouint()
  cpufreq: Replace magic number
  cpufreq: drop redundant cpus_read_lock() from store_local_boost()
  cpufreq/amd-pstate: Avoid shadowing ret in amd_pstate_ut_check_driver()
  cpufreq: intel_pstate: Document hybrid processor support
  cpufreq: intel_pstate: EAS: Increase cost for CPUs using L3 cache
  cpufreq: intel_pstate: EAS support for hybrid platforms
  cpufreq: Drop policy locking from cpufreq_policy_is_good_for_eas()
  cpufreq: intel_pstate: Populate the cpu_capacity sysfs entries
  arch_topology: Relocate cpu_scale to topology.[h|c]
  cpufreq/sched: Move cpufreq-specific EAS checks to cpufreq
  cpufreq/sched: schedutil: Add helper for governor checks
  amd-pstate-ut: Reset amd-pstate driver mode after running selftests
  cpufreq/amd-pstate: Add support for the "Requested CPU Min frequency" BIOS option
  cpufreq/amd-pstate: Add offline, online and suspend callbacks for amd_pstate_driver
  cpufreq: Force sync policy boost with global boost on sysfs update
  cpufreq: Preserve policy's boost state after resume
  cpufreq: Introduce policy_set_boost()
  cpufreq: Don't unnecessarily call set_boost()
  ...
This commit is contained in:
Rafael J. Wysocki
2025-05-26 20:19:40 +02:00
parent e481e10ab5 922607a2b4
commit f34dc28343
16 changed files with 836 additions and 418 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/ABI/testing/sysfs-devices-system-cpu
+54
View File
@@ -268,6 +268,60 @@ Description: Discover CPUs in the same CPU frequency coordination domain
This file is only present if the acpi-cpufreq or the cppc-cpufreq
drivers are in use.
What: /sys/devices/system/cpu/cpuX/cpufreq/auto_select
Date: May 2025
Contact: linux-pm@vger.kernel.org
Description: Autonomous selection enable
Read/write interface to control autonomous selection enable
Read returns autonomous selection status:
0: autonomous selection is disabled
1: autonomous selection is enabled
Write 'y' or '1' or 'on' to enable autonomous selection.
Write 'n' or '0' or 'off' to disable autonomous selection.
This file is only present if the cppc-cpufreq driver is in use.
What: /sys/devices/system/cpu/cpuX/cpufreq/auto_act_window
Date: May 2025
Contact: linux-pm@vger.kernel.org
Description: Autonomous activity window
This file indicates a moving utilization sensitivity window to
the platform's autonomous selection policy.
Read/write an integer represents autonomous activity window (in
microseconds) from/to this file. The max value to write is
1270000000 but the max significand is 127. This means that if 128
is written to this file, 127 will be stored. If the value is
greater than 130, only the first two digits will be saved as
significand.
Writing a zero value to this file enable the platform to
determine an appropriate Activity Window depending on the workload.
Writing to this file only has meaning when Autonomous Selection is
enabled.
This file is only present if the cppc-cpufreq driver is in use.
What: /sys/devices/system/cpu/cpuX/cpufreq/energy_performance_preference_val
Date: May 2025
Contact: linux-pm@vger.kernel.org
Description: Energy performance preference
Read/write an 8-bit integer from/to this file. This file
represents a range of values from 0 (performance preference) to
0xFF (energy efficiency preference) that influences the rate of
performance increase/decrease and the result of the hardware's
energy efficiency and performance optimization policies.
Writing to this file only has meaning when Autonomous Selection is
enabled.
This file is only present if the cppc-cpufreq driver is in use.
What: /sys/devices/system/cpu/cpu*/cache/index3/cache_disable_{0,1}
Date: August 2008
Documentation/admin-guide/pm/intel_pstate.rst
+102 -2
View File
@@ -329,6 +329,106 @@ information listed above is the same for all of the processors supporting the
HWP feature, which is why ``intel_pstate`` works with all of them.]
Support for Hybrid Processors
=============================
Some processors supported by ``intel_pstate`` contain two or more types of CPU
cores differing by the maximum turbo P-state, performance vs power characteristics,
cache sizes, and possibly other properties. They are commonly referred to as
hybrid processors. To support them, ``intel_pstate`` requires HWP to be enabled
and it assumes the HWP performance units to be the same for all CPUs in the
system, so a given HWP performance level always represents approximately the
same physical performance regardless of the core (CPU) type.
Hybrid Processors with SMT
--------------------------
On systems where SMT (Simultaneous Multithreading), also referred to as
HyperThreading (HT) in the context of Intel processors, is enabled on at least
one core, ``intel_pstate`` assigns performance-based priorities to CPUs. Namely,
the priority of a given CPU reflects its highest HWP performance level which
causes the CPU scheduler to generally prefer more performant CPUs, so the less
performant CPUs are used when the other ones are fully loaded. However, SMT
siblings (that is, logical CPUs sharing one physical core) are treated in a
special way such that if one of them is in use, the effective priority of the
other ones is lowered below the priorities of the CPUs located in the other
physical cores.
This approach maximizes performance in the majority of cases, but unfortunately
it also leads to excessive energy usage in some important scenarios, like video
playback, which is not generally desirable. While there is no other viable
choice with SMT enabled because the effective capacity and utilization of SMT
siblings are hard to determine, hybrid processors without SMT can be handled in
more energy-efficient ways.
.. _CAS:
Capacity-Aware Scheduling Support
---------------------------------
The capacity-aware scheduling (CAS) support in the CPU scheduler is enabled by
``intel_pstate`` by default on hybrid processors without SMT. CAS generally
causes the scheduler to put tasks on a CPU so long as there is a sufficient
amount of spare capacity on it, and if the utilization of a given task is too
high for it, the task will need to go somewhere else.
Since CAS takes CPU capacities into account, it does not require CPU
prioritization and it allows tasks to be distributed more symmetrically among
the more performant and less performant CPUs. Once placed on a CPU with enough
capacity to accommodate it, a task may just continue to run there regardless of
whether or not the other CPUs are fully loaded, so on average CAS reduces the
utilization of the more performant CPUs which causes the energy usage to be more
balanced because the more performant CPUs are generally less energy-efficient
than the less performant ones.
In order to use CAS, the scheduler needs to know the capacity of each CPU in
the system and it needs to be able to compute scale-invariant utilization of
CPUs, so ``intel_pstate`` provides it with the requisite information.
First of all, the capacity of each CPU is represented by the ratio of its highest
HWP performance level, multiplied by 1024, to the highest HWP performance level
of the most performant CPU in the system, which works because the HWP performance
units are the same for all CPUs. Second, the frequency-invariance computations,
carried out by the scheduler to always express CPU utilization in the same units
regardless of the frequency it is currently running at, are adjusted to take the
CPU capacity into account. All of this happens when ``intel_pstate`` has
registered itself with the ``CPUFreq`` core and it has figured out that it is
running on a hybrid processor without SMT.
Energy-Aware Scheduling Support
-------------------------------
If ``CONFIG_ENERGY_MODEL`` has been set during kernel configuration and
``intel_pstate`` runs on a hybrid processor without SMT, in addition to enabling
`CAS <CAS_>`_ it registers an Energy Model for the processor. This allows the
Energy-Aware Scheduling (EAS) support to be enabled in the CPU scheduler if
``schedutil`` is used as the ``CPUFreq`` governor which requires ``intel_pstate``
to operate in the `passive mode <Passive Mode_>`_.
The Energy Model registered by ``intel_pstate`` is artificial (that is, it is
based on abstract cost values and it does not include any real power numbers)
and it is relatively simple to avoid unnecessary computations in the scheduler.
There is a performance domain in it for every CPU in the system and the cost
values for these performance domains have been chosen so that running a task on
a less performant (small) CPU appears to be always cheaper than running that
task on a more performant (big) CPU. However, for two CPUs of the same type,
the cost difference depends on their current utilization, and the CPU whose
current utilization is higher generally appears to be a more expensive
destination for a given task. This helps to balance the load among CPUs of the
same type.
Since EAS works on top of CAS, high-utilization tasks are always migrated to
CPUs with enough capacity to accommodate them, but thanks to EAS, low-utilization
tasks tend to be placed on the CPUs that look less expensive to the scheduler.
Effectively, this causes the less performant and less loaded CPUs to be
preferred as long as they have enough spare capacity to run the given task
which generally leads to reduced energy usage.
The Energy Model created by ``intel_pstate`` can be inspected by looking at
the ``energy_model`` directory in ``debugfs`` (typlically mounted on
``/sys/kernel/debug/``).
User Space Interface in ``sysfs``
=================================
@@ -697,8 +797,8 @@ of them have to be prepended with the ``intel_pstate=`` prefix.
Limits`_ for details).
``no_cas``
Do not enable capacity-aware scheduling (CAS) which is enabled by
default on hybrid systems.
Do not enable `capacity-aware scheduling <CAS_>`_ which is enabled by
default on hybrid systems without SMT.
Diagnostics and Tuning
======================
drivers/base/arch_topology.c
-52
View File
@@ -154,14 +154,6 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq,
per_cpu(arch_freq_scale, i) = scale;
}
DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE;
EXPORT_PER_CPU_SYMBOL_GPL(cpu_scale);
void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity)
{
per_cpu(cpu_scale, cpu) = capacity;
}
DEFINE_PER_CPU(unsigned long, hw_pressure);
/**
@@ -207,53 +199,9 @@ void topology_update_hw_pressure(const struct cpumask *cpus,
}
EXPORT_SYMBOL_GPL(topology_update_hw_pressure);
static ssize_t cpu_capacity_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
struct cpu *cpu = container_of(dev, struct cpu, dev);
return sysfs_emit(buf, "%lu\n", topology_get_cpu_scale(cpu->dev.id));
}
static void update_topology_flags_workfn(struct work_struct *work);
static DECLARE_WORK(update_topology_flags_work, update_topology_flags_workfn);
static DEVICE_ATTR_RO(cpu_capacity);
static int cpu_capacity_sysctl_add(unsigned int cpu)
{
struct device *cpu_dev = get_cpu_device(cpu);
if (!cpu_dev)
return -ENOENT;
device_create_file(cpu_dev, &dev_attr_cpu_capacity);
return 0;
}
static int cpu_capacity_sysctl_remove(unsigned int cpu)
{
struct device *cpu_dev = get_cpu_device(cpu);
if (!cpu_dev)
return -ENOENT;
device_remove_file(cpu_dev, &dev_attr_cpu_capacity);
return 0;
}
static int register_cpu_capacity_sysctl(void)
{
cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "topology/cpu-capacity",
cpu_capacity_sysctl_add, cpu_capacity_sysctl_remove);
return 0;
}
subsys_initcall(register_cpu_capacity_sysctl);
static int update_topology;
int topology_update_cpu_topology(void)
drivers/base/topology.c
+52
View File
@@ -208,3 +208,55 @@ static int __init topology_sysfs_init(void)
}
device_initcall(topology_sysfs_init);
DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE;
EXPORT_PER_CPU_SYMBOL_GPL(cpu_scale);
void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity)
{
per_cpu(cpu_scale, cpu) = capacity;
}
static ssize_t cpu_capacity_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
struct cpu *cpu = container_of(dev, struct cpu, dev);
return sysfs_emit(buf, "%lu\n", topology_get_cpu_scale(cpu->dev.id));
}
static DEVICE_ATTR_RO(cpu_capacity);
static int cpu_capacity_sysctl_add(unsigned int cpu)
{
struct device *cpu_dev = get_cpu_device(cpu);
if (!cpu_dev)
return -ENOENT;
device_create_file(cpu_dev, &dev_attr_cpu_capacity);
return 0;
}
static int cpu_capacity_sysctl_remove(unsigned int cpu)
{
struct device *cpu_dev = get_cpu_device(cpu);
if (!cpu_dev)
return -ENOENT;
device_remove_file(cpu_dev, &dev_attr_cpu_capacity);
return 0;
}
static int register_cpu_capacity_sysctl(void)
{
cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "topology/cpu-capacity",
cpu_capacity_sysctl_add, cpu_capacity_sysctl_remove);
return 0;
}
subsys_initcall(register_cpu_capacity_sysctl);
drivers/cpufreq/amd-pstate-ut.c
+13 -8
View File
@@ -242,25 +242,30 @@ static int amd_pstate_set_mode(enum amd_pstate_mode mode)
static int amd_pstate_ut_check_driver(u32 index)
{
enum amd_pstate_mode mode1, mode2 = AMD_PSTATE_DISABLE;
enum amd_pstate_mode orig_mode = amd_pstate_get_status();
int ret;
for (mode1 = AMD_PSTATE_DISABLE; mode1 < AMD_PSTATE_MAX; mode1++) {
int ret = amd_pstate_set_mode(mode1);
ret = amd_pstate_set_mode(mode1);
if (ret)
return ret;
for (mode2 = AMD_PSTATE_DISABLE; mode2 < AMD_PSTATE_MAX; mode2++) {
if (mode1 == mode2)
continue;
ret = amd_pstate_set_mode(mode2);
if (ret) {
pr_err("%s: failed to update status for %s->%s\n", __func__,
amd_pstate_get_mode_string(mode1),
amd_pstate_get_mode_string(mode2));
return ret;
}
if (ret)
goto out;
}
}
return 0;
out:
if (ret)
pr_warn("%s: failed to update status for %s->%s: %d\n", __func__,
amd_pstate_get_mode_string(mode1),
amd_pstate_get_mode_string(mode2), ret);
amd_pstate_set_mode(orig_mode);
return ret;
}
static int __init amd_pstate_ut_init(void)
drivers/cpufreq/amd-pstate.c
+93 -33
View File
@@ -389,7 +389,8 @@ static inline int amd_pstate_cppc_enable(struct cpufreq_policy *policy)
static int msr_init_perf(struct amd_cpudata *cpudata)
{
union perf_cached perf = READ_ONCE(cpudata->perf);
u64 cap1, numerator;
u64 cap1, numerator, cppc_req;
u8 min_perf;
int ret = rdmsrl_safe_on_cpu(cpudata->cpu, MSR_AMD_CPPC_CAP1,
&cap1);
@@ -400,6 +401,22 @@ static int msr_init_perf(struct amd_cpudata *cpudata)
if (ret)
return ret;
ret = rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, &cppc_req);
if (ret)
return ret;
WRITE_ONCE(cpudata->cppc_req_cached, cppc_req);
min_perf = FIELD_GET(AMD_CPPC_MIN_PERF_MASK, cppc_req);
/*
* Clear out the min_perf part to check if the rest of the MSR is 0, if yes, this is an
* indication that the min_perf value is the one specified through the BIOS option
*/
cppc_req &= ~(AMD_CPPC_MIN_PERF_MASK);
if (!cppc_req)
perf.bios_min_perf = min_perf;
perf.highest_perf = numerator;
perf.max_limit_perf = numerator;
perf.min_limit_perf = FIELD_GET(AMD_CPPC_LOWEST_PERF_MASK, cap1);
@@ -554,6 +571,10 @@ static void amd_pstate_update(struct amd_cpudata *cpudata, u8 min_perf,
if (!policy)
return;
/* limit the max perf when core performance boost feature is disabled */
if (!cpudata->boost_supported)
max_perf = min_t(u8, perf.nominal_perf, max_perf);
des_perf = clamp_t(u8, des_perf, min_perf, max_perf);
policy->cur = perf_to_freq(perf, cpudata->nominal_freq, des_perf);
@@ -563,10 +584,6 @@ static void amd_pstate_update(struct amd_cpudata *cpudata, u8 min_perf,
des_perf = 0;
}
/* limit the max perf when core performance boost feature is disabled */
if (!cpudata->boost_supported)
max_perf = min_t(u8, perf.nominal_perf, max_perf);
if (trace_amd_pstate_perf_enabled() && amd_pstate_sample(cpudata)) {
trace_amd_pstate_perf(min_perf, des_perf, max_perf, cpudata->freq,
cpudata->cur.mperf, cpudata->cur.aperf, cpudata->cur.tsc,
@@ -580,20 +597,26 @@ static int amd_pstate_verify(struct cpufreq_policy_data *policy_data)
{
/*
* Initialize lower frequency limit (i.e.policy->min) with
* lowest_nonlinear_frequency which is the most energy efficient
* frequency. Override the initial value set by cpufreq core and
* amd-pstate qos_requests.
* lowest_nonlinear_frequency or the min frequency (if) specified in BIOS,
* Override the initial value set by cpufreq core and amd-pstate qos_requests.
*/
if (policy_data->min == FREQ_QOS_MIN_DEFAULT_VALUE) {
struct cpufreq_policy *policy __free(put_cpufreq_policy) =
cpufreq_cpu_get(policy_data->cpu);
struct amd_cpudata *cpudata;
union perf_cached perf;
if (!policy)
return -EINVAL;
cpudata = policy->driver_data;
policy_data->min = cpudata->lowest_nonlinear_freq;
perf = READ_ONCE(cpudata->perf);
if (perf.bios_min_perf)
policy_data->min = perf_to_freq(perf, cpudata->nominal_freq,
perf.bios_min_perf);
else
policy_data->min = cpudata->lowest_nonlinear_freq;
}
cpufreq_verify_within_cpu_limits(policy_data);
@@ -808,19 +831,16 @@ static void amd_pstate_init_prefcore(struct amd_cpudata *cpudata)
sched_set_itmt_core_prio((int)READ_ONCE(cpudata->prefcore_ranking), cpudata->cpu);
}
static void amd_pstate_update_limits(unsigned int cpu)
static void amd_pstate_update_limits(struct cpufreq_policy *policy)
{
struct cpufreq_policy *policy __free(put_cpufreq_policy) = cpufreq_cpu_get(cpu);
struct amd_cpudata *cpudata;
u32 prev_high = 0, cur_high = 0;
bool highest_perf_changed = false;
unsigned int cpu = policy->cpu;
if (!amd_pstate_prefcore)
return;
if (!policy)
return;
if (amd_get_highest_perf(cpu, &cur_high))
return;
@@ -1024,6 +1044,10 @@ free_cpudata1:
static void amd_pstate_cpu_exit(struct cpufreq_policy *policy)
{
struct amd_cpudata *cpudata = policy->driver_data;
union perf_cached perf = READ_ONCE(cpudata->perf);
/* Reset CPPC_REQ MSR to the BIOS value */
amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false);
freq_qos_remove_request(&cpudata->req[1]);
freq_qos_remove_request(&cpudata->req[0]);
@@ -1305,6 +1329,12 @@ static ssize_t amd_pstate_show_status(char *buf)
return sysfs_emit(buf, "%s\n", amd_pstate_mode_string[cppc_state]);
}
int amd_pstate_get_status(void)
{
return cppc_state;
}
EXPORT_SYMBOL_GPL(amd_pstate_get_status);
int amd_pstate_update_status(const char *buf, size_t size)
{
int mode_idx;
@@ -1419,7 +1449,6 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy)
struct amd_cpudata *cpudata;
union perf_cached perf;
struct device *dev;
u64 value;
int ret;
/*
@@ -1484,12 +1513,6 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy)
cpudata->epp_default = AMD_CPPC_EPP_BALANCE_PERFORMANCE;
}
if (cpu_feature_enabled(X86_FEATURE_CPPC)) {
ret = rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, &value);
if (ret)
return ret;
WRITE_ONCE(cpudata->cppc_req_cached, value);
}
ret = amd_pstate_set_epp(policy, cpudata->epp_default);
if (ret)
return ret;
@@ -1509,6 +1532,11 @@ static void amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy)
struct amd_cpudata *cpudata = policy->driver_data;
if (cpudata) {
union perf_cached perf = READ_ONCE(cpudata->perf);
/* Reset CPPC_REQ MSR to the BIOS value */
amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false);
kfree(cpudata);
policy->driver_data = NULL;
}
@@ -1559,21 +1587,38 @@ static int amd_pstate_epp_set_policy(struct cpufreq_policy *policy)
return 0;
}
static int amd_pstate_epp_cpu_online(struct cpufreq_policy *policy)
static int amd_pstate_cpu_online(struct cpufreq_policy *policy)
{
pr_debug("AMD CPU Core %d going online\n", policy->cpu);
return amd_pstate_cppc_enable(policy);
}
static int amd_pstate_epp_cpu_offline(struct cpufreq_policy *policy)
{
return 0;
}
static int amd_pstate_epp_suspend(struct cpufreq_policy *policy)
static int amd_pstate_cpu_offline(struct cpufreq_policy *policy)
{
struct amd_cpudata *cpudata = policy->driver_data;
union perf_cached perf = READ_ONCE(cpudata->perf);
/*
* Reset CPPC_REQ MSR to the BIOS value, this will allow us to retain the BIOS specified
* min_perf value across kexec reboots. If this CPU is just onlined normally after this, the
* limits, epp and desired perf will get reset to the cached values in cpudata struct
*/
return amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false);
}
static int amd_pstate_suspend(struct cpufreq_policy *policy)
{
struct amd_cpudata *cpudata = policy->driver_data;
union perf_cached perf = READ_ONCE(cpudata->perf);
int ret;
/*
* Reset CPPC_REQ MSR to the BIOS value, this will allow us to retain the BIOS specified
* min_perf value across kexec reboots. If this CPU is just resumed back without kexec,
* the limits, epp and desired perf will get reset to the cached values in cpudata struct
*/
ret = amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false);
if (ret)
return ret;
/* invalidate to ensure it's rewritten during resume */
cpudata->cppc_req_cached = 0;
@@ -1584,6 +1629,17 @@ static int amd_pstate_epp_suspend(struct cpufreq_policy *policy)
return 0;
}
static int amd_pstate_resume(struct cpufreq_policy *policy)
{
struct amd_cpudata *cpudata = policy->driver_data;
union perf_cached perf = READ_ONCE(cpudata->perf);
int cur_perf = freq_to_perf(perf, cpudata->nominal_freq, policy->cur);
/* Set CPPC_REQ to last sane value until the governor updates it */
return amd_pstate_update_perf(policy, perf.min_limit_perf, cur_perf, perf.max_limit_perf,
0U, false);
}
static int amd_pstate_epp_resume(struct cpufreq_policy *policy)
{
struct amd_cpudata *cpudata = policy->driver_data;
@@ -1609,6 +1665,10 @@ static struct cpufreq_driver amd_pstate_driver = {
.fast_switch = amd_pstate_fast_switch,
.init = amd_pstate_cpu_init,
.exit = amd_pstate_cpu_exit,
.online = amd_pstate_cpu_online,
.offline = amd_pstate_cpu_offline,
.suspend = amd_pstate_suspend,
.resume = amd_pstate_resume,
.set_boost = amd_pstate_set_boost,
.update_limits = amd_pstate_update_limits,
.name = "amd-pstate",
@@ -1621,9 +1681,9 @@ static struct cpufreq_driver amd_pstate_epp_driver = {
.setpolicy = amd_pstate_epp_set_policy,
.init = amd_pstate_epp_cpu_init,
.exit = amd_pstate_epp_cpu_exit,
.offline = amd_pstate_epp_cpu_offline,
.online = amd_pstate_epp_cpu_online,
.suspend = amd_pstate_epp_suspend,
.offline = amd_pstate_cpu_offline,
.online = amd_pstate_cpu_online,
.suspend = amd_pstate_suspend,
.resume = amd_pstate_epp_resume,
.update_limits = amd_pstate_update_limits,
.set_boost = amd_pstate_set_boost,
drivers/cpufreq/amd-pstate.h
+3
View File
@@ -30,6 +30,7 @@
* @lowest_perf: the absolute lowest performance level of the processor
* @min_limit_perf: Cached value of the performance corresponding to policy->min
* @max_limit_perf: Cached value of the performance corresponding to policy->max
* @bios_min_perf: Cached perf value corresponding to the "Requested CPU Min Frequency" BIOS option
*/
union perf_cached {
struct {
@@ -39,6 +40,7 @@ union perf_cached {
u8 lowest_perf;
u8 min_limit_perf;
u8 max_limit_perf;
u8 bios_min_perf;
};
u64 val;
};
@@ -119,6 +121,7 @@ enum amd_pstate_mode {
AMD_PSTATE_MAX,
};
const char *amd_pstate_get_mode_string(enum amd_pstate_mode mode);
int amd_pstate_get_status(void);
int amd_pstate_update_status(const char *buf, size_t size);
#endif /* _LINUX_AMD_PSTATE_H */
drivers/cpufreq/cppc_cpufreq.c
+109
View File
@@ -808,10 +808,119 @@ static ssize_t show_freqdomain_cpus(struct cpufreq_policy *policy, char *buf)
return cpufreq_show_cpus(cpu_data->shared_cpu_map, buf);
}
static ssize_t show_auto_select(struct cpufreq_policy *policy, char *buf)
{
bool val;
int ret;
ret = cppc_get_auto_sel(policy->cpu, &val);
/* show "<unsupported>" when this register is not supported by cpc */
if (ret == -EOPNOTSUPP)
return sysfs_emit(buf, "<unsupported>\n");
if (ret)
return ret;
return sysfs_emit(buf, "%d\n", val);
}
static ssize_t store_auto_select(struct cpufreq_policy *policy,
const char *buf, size_t count)
{
bool val;
int ret;
ret = kstrtobool(buf, &val);
if (ret)
return ret;
ret = cppc_set_auto_sel(policy->cpu, val);
if (ret)
return ret;
return count;
}
static ssize_t show_auto_act_window(struct cpufreq_policy *policy, char *buf)
{
u64 val;
int ret;
ret = cppc_get_auto_act_window(policy->cpu, &val);
/* show "<unsupported>" when this register is not supported by cpc */
if (ret == -EOPNOTSUPP)
return sysfs_emit(buf, "<unsupported>\n");
if (ret)
return ret;
return sysfs_emit(buf, "%llu\n", val);
}
static ssize_t store_auto_act_window(struct cpufreq_policy *policy,
const char *buf, size_t count)
{
u64 usec;
int ret;
ret = kstrtou64(buf, 0, &usec);
if (ret)
return ret;
ret = cppc_set_auto_act_window(policy->cpu, usec);
if (ret)
return ret;
return count;
}
static ssize_t show_energy_performance_preference_val(struct cpufreq_policy *policy, char *buf)
{
u64 val;
int ret;
ret = cppc_get_epp_perf(policy->cpu, &val);
/* show "<unsupported>" when this register is not supported by cpc */
if (ret == -EOPNOTSUPP)
return sysfs_emit(buf, "<unsupported>\n");
if (ret)
return ret;
return sysfs_emit(buf, "%llu\n", val);
}
static ssize_t store_energy_performance_preference_val(struct cpufreq_policy *policy,
const char *buf, size_t count)
{
u64 val;
int ret;
ret = kstrtou64(buf, 0, &val);
if (ret)
return ret;
ret = cppc_set_epp(policy->cpu, val);
if (ret)
return ret;
return count;
}
cpufreq_freq_attr_ro(freqdomain_cpus);
cpufreq_freq_attr_rw(auto_select);
cpufreq_freq_attr_rw(auto_act_window);
cpufreq_freq_attr_rw(energy_performance_preference_val);
static struct freq_attr *cppc_cpufreq_attr[] = {
&freqdomain_cpus,
&auto_select,
&auto_act_window,
&energy_performance_preference_val,
NULL,
};
drivers/cpufreq/cpufreq.c
+221 -256
View File
@@ -255,51 +255,6 @@ void cpufreq_cpu_put(struct cpufreq_policy *policy)
}
EXPORT_SYMBOL_GPL(cpufreq_cpu_put);
/**
* cpufreq_cpu_release - Unlock a policy and decrement its usage counter.
* @policy: cpufreq policy returned by cpufreq_cpu_acquire().
*/
void cpufreq_cpu_release(struct cpufreq_policy *policy)
{
if (WARN_ON(!policy))
return;
lockdep_assert_held(&policy->rwsem);
up_write(&policy->rwsem);
cpufreq_cpu_put(policy);
}
/**
* cpufreq_cpu_acquire - Find policy for a CPU, mark it as busy and lock it.
* @cpu: CPU to find the policy for.
*
* Call cpufreq_cpu_get() to get a reference on the cpufreq policy for @cpu and
* if the policy returned by it is not NULL, acquire its rwsem for writing.
* Return the policy if it is active or release it and return NULL otherwise.
*
* The policy returned by this function has to be released with the help of
* cpufreq_cpu_release() in order to release its rwsem and balance its usage
* counter properly.
*/
struct cpufreq_policy *cpufreq_cpu_acquire(unsigned int cpu)
{
struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
if (!policy)
return NULL;
down_write(&policy->rwsem);
if (policy_is_inactive(policy)) {
cpufreq_cpu_release(policy);
return NULL;
}
return policy;
}
/*********************************************************************
* EXTERNALLY AFFECTING FREQUENCY CHANGES *
*********************************************************************/
@@ -636,6 +591,22 @@ static ssize_t show_local_boost(struct cpufreq_policy *policy, char *buf)
return sysfs_emit(buf, "%d\n", policy->boost_enabled);
}
static int policy_set_boost(struct cpufreq_policy *policy, bool enable)
{
int ret;
if (policy->boost_enabled == enable)
return 0;
policy->boost_enabled = enable;
ret = cpufreq_driver->set_boost(policy, enable);
if (ret)
policy->boost_enabled = !policy->boost_enabled;
return ret;
}
static ssize_t store_local_boost(struct cpufreq_policy *policy,
const char *buf, size_t count)
{
@@ -651,21 +622,11 @@ static ssize_t store_local_boost(struct cpufreq_policy *policy,
if (!policy->boost_supported)
return -EINVAL;
if (policy->boost_enabled == enable)
ret = policy_set_boost(policy, enable);
if (!ret)
return count;
policy->boost_enabled = enable;
cpus_read_lock();
ret = cpufreq_driver->set_boost(policy, enable);
cpus_read_unlock();
if (ret) {
policy->boost_enabled = !policy->boost_enabled;
return ret;
}
return count;
return ret;
}
static struct freq_attr local_boost = __ATTR(boost, 0644, show_local_boost, store_local_boost);
@@ -845,7 +806,7 @@ static ssize_t show_scaling_governor(struct cpufreq_policy *policy, char *buf)
static ssize_t store_scaling_governor(struct cpufreq_policy *policy,
const char *buf, size_t count)
{
char str_governor[16];
char str_governor[CPUFREQ_NAME_LEN];
int ret;
ret = sscanf(buf, "%15s", str_governor);
@@ -956,9 +917,9 @@ static ssize_t store_scaling_setspeed(struct cpufreq_policy *policy,
if (!policy->governor || !policy->governor->store_setspeed)
return -EINVAL;
ret = sscanf(buf, "%u", &freq);
if (ret != 1)
return -EINVAL;
ret = kstrtouint(buf, 0, &freq);
if (ret)
return ret;
policy->governor->store_setspeed(policy, freq);
@@ -1025,17 +986,16 @@ static ssize_t show(struct kobject *kobj, struct attribute *attr, char *buf)
{
struct cpufreq_policy *policy = to_policy(kobj);
struct freq_attr *fattr = to_attr(attr);
ssize_t ret = -EBUSY;
if (!fattr->show)
return -EIO;
down_read(&policy->rwsem);
if (likely(!policy_is_inactive(policy)))
ret = fattr->show(policy, buf);
up_read(&policy->rwsem);
guard(cpufreq_policy_read)(policy);
return ret;
if (likely(!policy_is_inactive(policy)))
return fattr->show(policy, buf);
return -EBUSY;
}
static ssize_t store(struct kobject *kobj, struct attribute *attr,
@@ -1043,17 +1003,16 @@ static ssize_t store(struct kobject *kobj, struct attribute *attr,
{
struct cpufreq_policy *policy = to_policy(kobj);
struct freq_attr *fattr = to_attr(attr);
ssize_t ret = -EBUSY;
if (!fattr->store)
return -EIO;
down_write(&policy->rwsem);
if (likely(!policy_is_inactive(policy)))
ret = fattr->store(policy, buf, count);
up_write(&policy->rwsem);
guard(cpufreq_policy_write)(policy);
return ret;
if (likely(!policy_is_inactive(policy)))
return fattr->store(policy, buf, count);
return -EBUSY;
}
static void cpufreq_sysfs_release(struct kobject *kobj)
@@ -1211,7 +1170,8 @@ static int cpufreq_add_policy_cpu(struct cpufreq_policy *policy, unsigned int cp
if (cpumask_test_cpu(cpu, policy->cpus))
return 0;
down_write(&policy->rwsem);
guard(cpufreq_policy_write)(policy);
if (has_target())
cpufreq_stop_governor(policy);
@@ -1222,7 +1182,7 @@ static int cpufreq_add_policy_cpu(struct cpufreq_policy *policy, unsigned int cp
if (ret)
pr_err("%s: Failed to start governor\n", __func__);
}
up_write(&policy->rwsem);
return ret;
}
@@ -1242,9 +1202,10 @@ static void handle_update(struct work_struct *work)
container_of(work, struct cpufreq_policy, update);
pr_debug("handle_update for cpu %u called\n", policy->cpu);
down_write(&policy->rwsem);
guard(cpufreq_policy_write)(policy);
refresh_frequency_limits(policy);
up_write(&policy->rwsem);
}
static int cpufreq_notifier_min(struct notifier_block *nb, unsigned long freq,
@@ -1270,11 +1231,11 @@ static void cpufreq_policy_put_kobj(struct cpufreq_policy *policy)
struct kobject *kobj;
struct completion *cmp;
down_write(&policy->rwsem);
cpufreq_stats_free_table(policy);
kobj = &policy->kobj;
cmp = &policy->kobj_unregister;
up_write(&policy->rwsem);
scoped_guard(cpufreq_policy_write, policy) {
cpufreq_stats_free_table(policy);
kobj = &policy->kobj;
cmp = &policy->kobj_unregister;
}
kobject_put(kobj);
/*
@@ -1350,7 +1311,6 @@ static struct cpufreq_policy *cpufreq_policy_alloc(unsigned int cpu)
init_waitqueue_head(&policy->transition_wait);
INIT_WORK(&policy->update, handle_update);
policy->cpu = cpu;
return policy;
err_min_qos_notifier:
@@ -1419,35 +1379,17 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy)
kfree(policy);
}
static int cpufreq_online(unsigned int cpu)
static int cpufreq_policy_online(struct cpufreq_policy *policy,
unsigned int cpu, bool new_policy)
{
struct cpufreq_policy *policy;
bool new_policy;
unsigned long flags;
unsigned int j;
int ret;
pr_debug("%s: bringing CPU%u online\n", __func__, cpu);
guard(cpufreq_policy_write)(policy);
/* Check if this CPU already has a policy to manage it */
policy = per_cpu(cpufreq_cpu_data, cpu);
if (policy) {
WARN_ON(!cpumask_test_cpu(cpu, policy->related_cpus));
if (!policy_is_inactive(policy))
return cpufreq_add_policy_cpu(policy, cpu);
/* This is the only online CPU for the policy. Start over. */
new_policy = false;
down_write(&policy->rwsem);
policy->cpu = cpu;
policy->governor = NULL;
} else {
new_policy = true;
policy = cpufreq_policy_alloc(cpu);
if (!policy)
return -ENOMEM;
down_write(&policy->rwsem);
}
policy->cpu = cpu;
policy->governor = NULL;
if (!new_policy && cpufreq_driver->online) {
/* Recover policy->cpus using related_cpus */
@@ -1470,7 +1412,7 @@ static int cpufreq_online(unsigned int cpu)
if (ret) {
pr_debug("%s: %d: initialization failed\n", __func__,
__LINE__);
goto out_free_policy;
goto out_clear_policy;
}
/*
@@ -1621,33 +1563,6 @@ static int cpufreq_online(unsigned int cpu)
goto out_destroy_policy;
}
up_write(&policy->rwsem);
kobject_uevent(&policy->kobj, KOBJ_ADD);
/* Callback for handling stuff after policy is ready */
if (cpufreq_driver->ready)
cpufreq_driver->ready(policy);
/* Register cpufreq cooling only for a new policy */
if (new_policy && cpufreq_thermal_control_enabled(cpufreq_driver))
policy->cdev = of_cpufreq_cooling_register(policy);
/* Let the per-policy boost flag mirror the cpufreq_driver boost during init */
if (cpufreq_driver->set_boost && policy->boost_supported &&
policy->boost_enabled != cpufreq_boost_enabled()) {
policy->boost_enabled = cpufreq_boost_enabled();
ret = cpufreq_driver->set_boost(policy, policy->boost_enabled);
if (ret) {
/* If the set_boost fails, the online operation is not affected */
pr_info("%s: CPU%d: Cannot %s BOOST\n", __func__, policy->cpu,
str_enable_disable(policy->boost_enabled));
policy->boost_enabled = !policy->boost_enabled;
}
}
pr_debug("initialization complete\n");
return 0;
out_destroy_policy:
@@ -1662,14 +1577,72 @@ out_exit_policy:
if (cpufreq_driver->exit)
cpufreq_driver->exit(policy);
out_free_policy:
out_clear_policy:
cpumask_clear(policy->cpus);
up_write(&policy->rwsem);
cpufreq_policy_free(policy);
return ret;
}
static int cpufreq_online(unsigned int cpu)
{
struct cpufreq_policy *policy;
bool new_policy;
int ret;
pr_debug("%s: bringing CPU%u online\n", __func__, cpu);
/* Check if this CPU already has a policy to manage it */
policy = per_cpu(cpufreq_cpu_data, cpu);
if (policy) {
WARN_ON(!cpumask_test_cpu(cpu, policy->related_cpus));
if (!policy_is_inactive(policy))
return cpufreq_add_policy_cpu(policy, cpu);
/* This is the only online CPU for the policy. Start over. */
new_policy = false;
} else {
new_policy = true;
policy = cpufreq_policy_alloc(cpu);
if (!policy)
return -ENOMEM;
}
ret = cpufreq_policy_online(policy, cpu, new_policy);
if (ret) {
cpufreq_policy_free(policy);
return ret;
}
kobject_uevent(&policy->kobj, KOBJ_ADD);
/* Callback for handling stuff after policy is ready */
if (cpufreq_driver->ready)
cpufreq_driver->ready(policy);
/* Register cpufreq cooling only for a new policy */
if (new_policy && cpufreq_thermal_control_enabled(cpufreq_driver))
policy->cdev = of_cpufreq_cooling_register(policy);
/*
* Let the per-policy boost flag mirror the cpufreq_driver boost during
* initialization for a new policy. For an existing policy, maintain the
* previous boost value unless global boost is disabled.
*/
if (cpufreq_driver->set_boost && policy->boost_supported &&
(new_policy || !cpufreq_boost_enabled())) {
ret = policy_set_boost(policy, cpufreq_boost_enabled());
if (ret) {
/* If the set_boost fails, the online operation is not affected */
pr_info("%s: CPU%d: Cannot %s BOOST\n", __func__, policy->cpu,
str_enable_disable(cpufreq_boost_enabled()));
}
}
pr_debug("initialization complete\n");
return 0;
}
/**
* cpufreq_add_dev - the cpufreq interface for a CPU device.
* @dev: CPU device.
@@ -1757,11 +1730,10 @@ static int cpufreq_offline(unsigned int cpu)
return 0;
}
down_write(&policy->rwsem);
guard(cpufreq_policy_write)(policy);
__cpufreq_offline(cpu, policy);
up_write(&policy->rwsem);
return 0;
}
@@ -1778,33 +1750,29 @@ static void cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif)
if (!policy)
return;
down_write(&policy->rwsem);
scoped_guard(cpufreq_policy_write, policy) {
if (cpu_online(cpu))
__cpufreq_offline(cpu, policy);
if (cpu_online(cpu))
__cpufreq_offline(cpu, policy);
remove_cpu_dev_symlink(policy, cpu, dev);
remove_cpu_dev_symlink(policy, cpu, dev);
if (!cpumask_empty(policy->real_cpus))
return;
if (!cpumask_empty(policy->real_cpus)) {
up_write(&policy->rwsem);
return;
/*
* Unregister cpufreq cooling once all the CPUs of the policy
* are removed.
*/
if (cpufreq_thermal_control_enabled(cpufreq_driver)) {
cpufreq_cooling_unregister(policy->cdev);
policy->cdev = NULL;
}
/* We did light-weight exit earlier, do full tear down now */
if (cpufreq_driver->offline && cpufreq_driver->exit)
cpufreq_driver->exit(policy);
}
/*
* Unregister cpufreq cooling once all the CPUs of the policy are
* removed.
*/
if (cpufreq_thermal_control_enabled(cpufreq_driver)) {
cpufreq_cooling_unregister(policy->cdev);
policy->cdev = NULL;
}
/* We did light-weight exit earlier, do full tear down now */
if (cpufreq_driver->offline && cpufreq_driver->exit)
cpufreq_driver->exit(policy);
up_write(&policy->rwsem);
cpufreq_policy_free(policy);
}
@@ -1874,27 +1842,26 @@ static unsigned int cpufreq_verify_current_freq(struct cpufreq_policy *policy, b
*/
unsigned int cpufreq_quick_get(unsigned int cpu)
{
struct cpufreq_policy *policy;
unsigned int ret_freq = 0;
struct cpufreq_policy *policy __free(put_cpufreq_policy) = NULL;
unsigned long flags;
read_lock_irqsave(&cpufreq_driver_lock, flags);
if (cpufreq_driver && cpufreq_driver->setpolicy && cpufreq_driver->get) {
ret_freq = cpufreq_driver->get(cpu);
unsigned int ret_freq = cpufreq_driver->get(cpu);
read_unlock_irqrestore(&cpufreq_driver_lock, flags);
return ret_freq;
}
read_unlock_irqrestore(&cpufreq_driver_lock, flags);
policy = cpufreq_cpu_get(cpu);
if (policy) {
ret_freq = policy->cur;
cpufreq_cpu_put(policy);
}
if (policy)
return policy->cur;
return ret_freq;
return 0;
}
EXPORT_SYMBOL(cpufreq_quick_get);
@@ -1906,15 +1873,13 @@ EXPORT_SYMBOL(cpufreq_quick_get);
*/
unsigned int cpufreq_quick_get_max(unsigned int cpu)
{
struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
unsigned int ret_freq = 0;
struct cpufreq_policy *policy __free(put_cpufreq_policy);
if (policy) {
ret_freq = policy->max;
cpufreq_cpu_put(policy);
}
policy = cpufreq_cpu_get(cpu);
if (policy)
return policy->max;
return ret_freq;
return 0;
}
EXPORT_SYMBOL(cpufreq_quick_get_max);
@@ -1926,15 +1891,13 @@ EXPORT_SYMBOL(cpufreq_quick_get_max);
*/
__weak unsigned int cpufreq_get_hw_max_freq(unsigned int cpu)
{
struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
unsigned int ret_freq = 0;
struct cpufreq_policy *policy __free(put_cpufreq_policy);
if (policy) {
ret_freq = policy->cpuinfo.max_freq;
cpufreq_cpu_put(policy);
}
policy = cpufreq_cpu_get(cpu);
if (policy)
return policy->cpuinfo.max_freq;
return ret_freq;
return 0;
}
EXPORT_SYMBOL(cpufreq_get_hw_max_freq);
@@ -1954,19 +1917,18 @@ static unsigned int __cpufreq_get(struct cpufreq_policy *policy)
*/
unsigned int cpufreq_get(unsigned int cpu)
{
struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
unsigned int ret_freq = 0;
struct cpufreq_policy *policy __free(put_cpufreq_policy);
if (policy) {
down_read(&policy->rwsem);
if (cpufreq_driver->get)
ret_freq = __cpufreq_get(policy);
up_read(&policy->rwsem);
policy = cpufreq_cpu_get(cpu);
if (!policy)
return 0;
cpufreq_cpu_put(policy);
}
guard(cpufreq_policy_read)(policy);
return ret_freq;
if (cpufreq_driver->get)
return __cpufreq_get(policy);
return 0;
}
EXPORT_SYMBOL(cpufreq_get);
@@ -2025,9 +1987,9 @@ void cpufreq_suspend(void)
for_each_active_policy(policy) {
if (has_target()) {
down_write(&policy->rwsem);
cpufreq_stop_governor(policy);
up_write(&policy->rwsem);
scoped_guard(cpufreq_policy_write, policy) {
cpufreq_stop_governor(policy);
}
}
if (cpufreq_driver->suspend && cpufreq_driver->suspend(policy))
@@ -2068,9 +2030,9 @@ void cpufreq_resume(void)
pr_err("%s: Failed to resume driver: %s\n", __func__,
cpufreq_driver->name);
} else if (has_target()) {
down_write(&policy->rwsem);
ret = cpufreq_start_governor(policy);
up_write(&policy->rwsem);
scoped_guard(cpufreq_policy_write, policy) {
ret = cpufreq_start_governor(policy);
}
if (ret)
pr_err("%s: Failed to start governor for CPU%u's policy\n",
@@ -2438,15 +2400,9 @@ int cpufreq_driver_target(struct cpufreq_policy *policy,
unsigned int target_freq,
unsigned int relation)
{
int ret;
guard(cpufreq_policy_write)(policy);
down_write(&policy->rwsem);
ret = __cpufreq_driver_target(policy, target_freq, relation);
up_write(&policy->rwsem);
return ret;
return __cpufreq_driver_target(policy, target_freq, relation);
}
EXPORT_SYMBOL_GPL(cpufreq_driver_target);
@@ -2618,31 +2574,6 @@ EXPORT_SYMBOL_GPL(cpufreq_unregister_governor);
* POLICY INTERFACE *
*********************************************************************/
/**
* cpufreq_get_policy - get the current cpufreq_policy
* @policy: struct cpufreq_policy into which the current cpufreq_policy
* is written
* @cpu: CPU to find the policy for
*
* Reads the current cpufreq policy.
*/
int cpufreq_get_policy(struct cpufreq_policy *policy, unsigned int cpu)
{
struct cpufreq_policy *cpu_policy;
if (!policy)
return -EINVAL;
cpu_policy = cpufreq_cpu_get(cpu);
if (!cpu_policy)
return -EINVAL;
memcpy(policy, cpu_policy, sizeof(*policy));
cpufreq_cpu_put(cpu_policy);
return 0;
}
EXPORT_SYMBOL(cpufreq_get_policy);
DEFINE_PER_CPU(unsigned long, cpufreq_pressure);
/**
@@ -2793,6 +2724,21 @@ static int cpufreq_set_policy(struct cpufreq_policy *policy,
return ret;
}
static void cpufreq_policy_refresh(struct cpufreq_policy *policy)
{
guard(cpufreq_policy_write)(policy);
/*
* BIOS might change freq behind our back
* -> ask driver for current freq and notify governors about a change
*/
if (cpufreq_driver->get && has_target() &&
(cpufreq_suspended || WARN_ON(!cpufreq_verify_current_freq(policy, false))))
return;
refresh_frequency_limits(policy);
}
/**
* cpufreq_update_policy - Re-evaluate an existing cpufreq policy.
* @cpu: CPU to re-evaluate the policy for.
@@ -2804,23 +2750,13 @@ static int cpufreq_set_policy(struct cpufreq_policy *policy,
*/
void cpufreq_update_policy(unsigned int cpu)
{
struct cpufreq_policy *policy = cpufreq_cpu_acquire(cpu);
struct cpufreq_policy *policy __free(put_cpufreq_policy);
policy = cpufreq_cpu_get(cpu);
if (!policy)
return;
/*
* BIOS might change freq behind our back
* -> ask driver for current freq and notify governors about a change
*/
if (cpufreq_driver->get && has_target() &&
(cpufreq_suspended || WARN_ON(!cpufreq_verify_current_freq(policy, false))))
goto unlock;
refresh_frequency_limits(policy);
unlock:
cpufreq_cpu_release(policy);
cpufreq_policy_refresh(policy);
}
EXPORT_SYMBOL(cpufreq_update_policy);
@@ -2829,7 +2765,7 @@ EXPORT_SYMBOL(cpufreq_update_policy);
* @cpu: CPU to update the policy limits for.
*
* Invoke the driver's ->update_limits callback if present or call
* cpufreq_update_policy() for @cpu.
* cpufreq_policy_refresh() for @cpu.
*/
void cpufreq_update_limits(unsigned int cpu)
{
@@ -2840,9 +2776,9 @@ void cpufreq_update_limits(unsigned int cpu)
return;
if (cpufreq_driver->update_limits)
cpufreq_driver->update_limits(cpu);
cpufreq_driver->update_limits(policy);
else
cpufreq_update_policy(cpu);
cpufreq_policy_refresh(policy);
}
EXPORT_SYMBOL_GPL(cpufreq_update_limits);
@@ -2876,8 +2812,10 @@ static int cpufreq_boost_trigger_state(int state)
unsigned long flags;
int ret = 0;
if (cpufreq_driver->boost_enabled == state)
return 0;
/*
* Don't compare 'cpufreq_driver->boost_enabled' with 'state' here to
* make sure all policies are in sync with global boost flag.
*/
write_lock_irqsave(&cpufreq_driver_lock, flags);
cpufreq_driver->boost_enabled = state;
@@ -2888,12 +2826,9 @@ static int cpufreq_boost_trigger_state(int state)
if (!policy->boost_supported)
continue;
policy->boost_enabled = state;
ret = cpufreq_driver->set_boost(policy, state);
if (ret) {
policy->boost_enabled = !policy->boost_enabled;
ret = policy_set_boost(policy, state);
if (ret)
goto err_reset_state;
}
}
cpus_read_unlock();
@@ -3118,6 +3053,36 @@ static int __init cpufreq_core_init(void)
return 0;
}
static bool cpufreq_policy_is_good_for_eas(unsigned int cpu)
{
struct cpufreq_policy *policy __free(put_cpufreq_policy);
policy = cpufreq_cpu_get(cpu);
if (!policy) {
pr_debug("cpufreq policy not set for CPU: %d\n", cpu);
return false;
}
return sugov_is_governor(policy);
}
bool cpufreq_ready_for_eas(const struct cpumask *cpu_mask)
{
unsigned int cpu;
/* Do not attempt EAS if schedutil is not being used. */
for_each_cpu(cpu, cpu_mask) {
if (!cpufreq_policy_is_good_for_eas(cpu)) {
pr_debug("rd %*pbl: schedutil is mandatory for EAS\n",
cpumask_pr_args(cpu_mask));
return false;
}
}
return true;
}
module_param(off, int, 0444);
module_param_string(default_governor, default_governor, CPUFREQ_NAME_LEN, 0444);
core_initcall(cpufreq_core_init);
drivers/cpufreq/intel_pstate.c
+150 -31
View File
@@ -221,6 +221,7 @@ struct global_params {
* @sched_flags: Store scheduler flags for possible cross CPU update
* @hwp_boost_min: Last HWP boosted min performance
* @suspended: Whether or not the driver has been suspended.
* @pd_registered: Set when a perf domain is registered for this CPU.
* @hwp_notify_work: workqueue for HWP notifications.
*
* This structure stores per CPU instance data for all CPUs.
@@ -260,6 +261,9 @@ struct cpudata {
unsigned int sched_flags;
u32 hwp_boost_min;
bool suspended;
#ifdef CONFIG_ENERGY_MODEL
bool pd_registered;
#endif
struct delayed_work hwp_notify_work;
};
@@ -303,6 +307,7 @@ static bool hwp_is_hybrid;
static struct cpufreq_driver *intel_pstate_driver __read_mostly;
#define INTEL_PSTATE_CORE_SCALING 100000
#define HYBRID_SCALING_FACTOR_ADL 78741
#define HYBRID_SCALING_FACTOR_MTL 80000
#define HYBRID_SCALING_FACTOR_LNL 86957
@@ -311,7 +316,7 @@ static int hybrid_scaling_factor;
static inline int core_get_scaling(void)
{
return 100000;
return INTEL_PSTATE_CORE_SCALING;
}
#ifdef CONFIG_ACPI
@@ -948,12 +953,124 @@ static struct cpudata *hybrid_max_perf_cpu __read_mostly;
*/
static DEFINE_MUTEX(hybrid_capacity_lock);
#ifdef CONFIG_ENERGY_MODEL
#define HYBRID_EM_STATE_COUNT 4
static int hybrid_active_power(struct device *dev, unsigned long *power,
unsigned long *freq)
{
/*
* Create "utilization bins" of 0-40%, 40%-60%, 60%-80%, and 80%-100%
* of the maximum capacity such that two CPUs of the same type will be
* regarded as equally attractive if the utilization of each of them
* falls into the same bin, which should prevent tasks from being
* migrated between them too often.
*
* For this purpose, return the "frequency" of 2 for the first
* performance level and otherwise leave the value set by the caller.
*/
if (!*freq)
*freq = 2;
/* No power information. */
*power = EM_MAX_POWER;
return 0;
}
static int hybrid_get_cost(struct device *dev, unsigned long freq,
unsigned long *cost)
{
struct pstate_data *pstate = &all_cpu_data[dev->id]->pstate;
struct cpu_cacheinfo *cacheinfo = get_cpu_cacheinfo(dev->id);
/*
* The smaller the perf-to-frequency scaling factor, the larger the IPC
* ratio between the given CPU and the least capable CPU in the system.
* Regard that IPC ratio as the primary cost component and assume that
* the scaling factors for different CPU types will differ by at least
* 5% and they will not be above INTEL_PSTATE_CORE_SCALING.
*
* Add the freq value to the cost, so that the cost of running on CPUs
* of the same type in different "utilization bins" is different.
*/
*cost = div_u64(100ULL * INTEL_PSTATE_CORE_SCALING, pstate->scaling) + freq;
/*
* Increase the cost slightly for CPUs able to access L3 to avoid
* touching it in case some other CPUs of the same type can do the work
* without it.
*/
if (cacheinfo) {
unsigned int i;
/* Check if L3 cache is there. */
for (i = 0; i < cacheinfo->num_leaves; i++) {
if (cacheinfo->info_list[i].level == 3) {
*cost += 2;
break;
}
}
}
return 0;
}
static bool hybrid_register_perf_domain(unsigned int cpu)
{
static const struct em_data_callback cb
= EM_ADV_DATA_CB(hybrid_active_power, hybrid_get_cost);
struct cpudata *cpudata = all_cpu_data[cpu];
struct device *cpu_dev;
/*
* Registering EM perf domains without enabling asymmetric CPU capacity
* support is not really useful and one domain should not be registered
* more than once.
*/
if (!hybrid_max_perf_cpu || cpudata->pd_registered)
return false;
cpu_dev = get_cpu_device(cpu);
if (!cpu_dev)
return false;
if (em_dev_register_perf_domain(cpu_dev, HYBRID_EM_STATE_COUNT, &cb,
cpumask_of(cpu), false))
return false;
cpudata->pd_registered = true;
return true;
}
static void hybrid_register_all_perf_domains(void)
{
unsigned int cpu;
for_each_online_cpu(cpu)
hybrid_register_perf_domain(cpu);
}
static void hybrid_update_perf_domain(struct cpudata *cpu)
{
if (cpu->pd_registered)
em_adjust_cpu_capacity(cpu->cpu);
}
#else /* !CONFIG_ENERGY_MODEL */
static inline bool hybrid_register_perf_domain(unsigned int cpu) { return false; }
static inline void hybrid_register_all_perf_domains(void) {}
static inline void hybrid_update_perf_domain(struct cpudata *cpu) {}
#endif /* CONFIG_ENERGY_MODEL */
static void hybrid_set_cpu_capacity(struct cpudata *cpu)
{
arch_set_cpu_capacity(cpu->cpu, cpu->capacity_perf,
hybrid_max_perf_cpu->capacity_perf,
cpu->capacity_perf,
cpu->pstate.max_pstate_physical);
hybrid_update_perf_domain(cpu);
topology_set_cpu_scale(cpu->cpu, arch_scale_cpu_capacity(cpu->cpu));
pr_debug("CPU%d: perf = %u, max. perf = %u, base perf = %d\n", cpu->cpu,
cpu->capacity_perf, hybrid_max_perf_cpu->capacity_perf,
@@ -1042,6 +1159,11 @@ static void hybrid_refresh_cpu_capacity_scaling(void)
guard(mutex)(&hybrid_capacity_lock);
__hybrid_refresh_cpu_capacity_scaling();
/*
* Perf domains are not registered before setting hybrid_max_perf_cpu,
* so register them all after setting up CPU capacity scaling.
*/
hybrid_register_all_perf_domains();
}
static void hybrid_init_cpu_capacity_scaling(bool refresh)
@@ -1069,7 +1191,7 @@ static void hybrid_init_cpu_capacity_scaling(bool refresh)
hybrid_refresh_cpu_capacity_scaling();
/*
* Disabling ITMT causes sched domains to be rebuilt to disable asym
* packing and enable asym capacity.
* packing and enable asym capacity and EAS.
*/
sched_clear_itmt_support();
}
@@ -1147,6 +1269,14 @@ static void hybrid_update_capacity(struct cpudata *cpu)
}
hybrid_set_cpu_capacity(cpu);
/*
* If the CPU was offline to start with and it is going online for the
* first time, a perf domain needs to be registered for it if hybrid
* capacity scaling has been enabled already. In that case, sched
* domains need to be rebuilt to take the new perf domain into account.
*/
if (hybrid_register_perf_domain(cpu->cpu))
em_rebuild_sched_domains();
unlock:
mutex_unlock(&hybrid_capacity_lock);
@@ -1356,9 +1486,11 @@ static void intel_pstate_update_policies(void)
cpufreq_update_policy(cpu);
}
static void __intel_pstate_update_max_freq(struct cpudata *cpudata,
struct cpufreq_policy *policy)
static void __intel_pstate_update_max_freq(struct cpufreq_policy *policy,
struct cpudata *cpudata)
{
guard(cpufreq_policy_write)(policy);
if (hwp_active)
intel_pstate_get_hwp_cap(cpudata);
@@ -1368,42 +1500,34 @@ static void __intel_pstate_update_max_freq(struct cpudata *cpudata,
refresh_frequency_limits(policy);
}
static void intel_pstate_update_limits(unsigned int cpu)
static bool intel_pstate_update_max_freq(struct cpudata *cpudata)
{
struct cpufreq_policy *policy = cpufreq_cpu_acquire(cpu);
struct cpudata *cpudata;
struct cpufreq_policy *policy __free(put_cpufreq_policy);
policy = cpufreq_cpu_get(cpudata->cpu);
if (!policy)
return;
return false;
cpudata = all_cpu_data[cpu];
__intel_pstate_update_max_freq(policy, cpudata);
__intel_pstate_update_max_freq(cpudata, policy);
return true;
}
/* Prevent the driver from being unregistered now. */
mutex_lock(&intel_pstate_driver_lock);
static void intel_pstate_update_limits(struct cpufreq_policy *policy)
{
struct cpudata *cpudata = all_cpu_data[policy->cpu];
cpufreq_cpu_release(policy);
__intel_pstate_update_max_freq(policy, cpudata);
hybrid_update_capacity(cpudata);
mutex_unlock(&intel_pstate_driver_lock);
}
static void intel_pstate_update_limits_for_all(void)
{
int cpu;
for_each_possible_cpu(cpu) {
struct cpufreq_policy *policy = cpufreq_cpu_acquire(cpu);
if (!policy)
continue;
__intel_pstate_update_max_freq(all_cpu_data[cpu], policy);
cpufreq_cpu_release(policy);
}
for_each_possible_cpu(cpu)
intel_pstate_update_max_freq(all_cpu_data[cpu]);
mutex_lock(&hybrid_capacity_lock);
@@ -1843,13 +1967,8 @@ static void intel_pstate_notify_work(struct work_struct *work)
{
struct cpudata *cpudata =
container_of(to_delayed_work(work), struct cpudata, hwp_notify_work);
struct cpufreq_policy *policy = cpufreq_cpu_acquire(cpudata->cpu);
if (policy) {
__intel_pstate_update_max_freq(cpudata, policy);
cpufreq_cpu_release(policy);
if (intel_pstate_update_max_freq(cpudata)) {
/*
* The driver will not be unregistered while this function is
* running, so update the capacity without acquiring the driver
include/linux/arch_topology.h
-8
View File
@@ -14,14 +14,6 @@ int topology_update_cpu_topology(void);
struct device_node;
bool topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu);
DECLARE_PER_CPU(unsigned long, cpu_scale);
static inline unsigned long topology_get_cpu_scale(int cpu)
{
return per_cpu(cpu_scale, cpu);
}
void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
DECLARE_PER_CPU(unsigned long, capacity_freq_ref);
include/linux/cpufreq.h
+18 -4
View File
@@ -170,6 +170,12 @@ struct cpufreq_policy {
struct notifier_block nb_max;
};
DEFINE_GUARD(cpufreq_policy_write, struct cpufreq_policy *,
down_write(&_T->rwsem), up_write(&_T->rwsem))
DEFINE_GUARD(cpufreq_policy_read, struct cpufreq_policy *,
down_read(&_T->rwsem), up_read(&_T->rwsem))
/*
* Used for passing new cpufreq policy data to the cpufreq driver's ->verify()
* callback for sanitization. That callback is only expected to modify the min
@@ -235,9 +241,6 @@ void disable_cpufreq(void);
u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy);
struct cpufreq_policy *cpufreq_cpu_acquire(unsigned int cpu);
void cpufreq_cpu_release(struct cpufreq_policy *policy);
int cpufreq_get_policy(struct cpufreq_policy *policy, unsigned int cpu);
void refresh_frequency_limits(struct cpufreq_policy *policy);
void cpufreq_update_policy(unsigned int cpu);
void cpufreq_update_limits(unsigned int cpu);
@@ -395,7 +398,7 @@ struct cpufreq_driver {
unsigned int (*get)(unsigned int cpu);
/* Called to update policy limits on firmware notifications. */
void (*update_limits)(unsigned int cpu);
void (*update_limits)(struct cpufreq_policy *policy);
/* optional */
int (*bios_limit)(int cpu, unsigned int *limit);
@@ -647,6 +650,15 @@ module_exit(__governor##_exit)
struct cpufreq_governor *cpufreq_default_governor(void);
struct cpufreq_governor *cpufreq_fallback_governor(void);
#ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL
bool sugov_is_governor(struct cpufreq_policy *policy);
#else
static inline bool sugov_is_governor(struct cpufreq_policy *policy)
{
return false;
}
#endif
static inline void cpufreq_policy_apply_limits(struct cpufreq_policy *policy)
{
if (policy->max < policy->cur)
@@ -1225,6 +1237,8 @@ void cpufreq_generic_init(struct cpufreq_policy *policy,
struct cpufreq_frequency_table *table,
unsigned int transition_latency);
bool cpufreq_ready_for_eas(const struct cpumask *cpu_mask);
static inline void cpufreq_register_em_with_opp(struct cpufreq_policy *policy)
{
dev_pm_opp_of_register_em(get_cpu_device(policy->cpu),
include/linux/topology.h
+9
View File
@@ -332,4 +332,13 @@ sched_numa_hop_mask(unsigned int node, unsigned int hops)
!IS_ERR_OR_NULL(mask); \
__hops++)
DECLARE_PER_CPU(unsigned long, cpu_scale);
static inline unsigned long topology_get_cpu_scale(int cpu)
{
return per_cpu(cpu_scale, cpu);
}
void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
#endif /* _LINUX_TOPOLOGY_H */
kernel/sched/cpufreq_schedutil.c
+7 -2
View File
@@ -630,7 +630,7 @@ static const struct kobj_type sugov_tunables_ktype = {
/********************** cpufreq governor interface *********************/
struct cpufreq_governor schedutil_gov;
static struct cpufreq_governor schedutil_gov;
static struct sugov_policy *sugov_policy_alloc(struct cpufreq_policy *policy)
{
@@ -909,7 +909,7 @@ static void sugov_limits(struct cpufreq_policy *policy)
WRITE_ONCE(sg_policy->limits_changed, true);
}
struct cpufreq_governor schedutil_gov = {
static struct cpufreq_governor schedutil_gov = {
.name = "schedutil",
.owner = THIS_MODULE,
.flags = CPUFREQ_GOV_DYNAMIC_SWITCHING,
@@ -927,4 +927,9 @@ struct cpufreq_governor *cpufreq_default_governor(void)
}
#endif
bool sugov_is_governor(struct cpufreq_policy *policy)
{
return policy->governor == &schedutil_gov;
}
cpufreq_governor_init(schedutil_gov);
kernel/sched/sched.h
-2
View File
@@ -3509,8 +3509,6 @@ static inline bool sched_energy_enabled(void)
return static_branch_unlikely(&sched_energy_present);
}
extern struct cpufreq_governor schedutil_gov;
#else /* ! (CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL) */
#define perf_domain_span(pd) NULL
kernel/sched/topology.c
+5 -20
View File
@@ -212,8 +212,6 @@ static bool sched_energy_update;
static bool sched_is_eas_possible(const struct cpumask *cpu_mask)
{
bool any_asym_capacity = false;
struct cpufreq_policy *policy;
struct cpufreq_governor *gov;
int i;
/* EAS is enabled for asymmetric CPU capacity topologies. */
@@ -248,25 +246,12 @@ static bool sched_is_eas_possible(const struct cpumask *cpu_mask)
return false;
}
/* Do not attempt EAS if schedutil is not being used. */
for_each_cpu(i, cpu_mask) {
policy = cpufreq_cpu_get(i);
if (!policy) {
if (sched_debug()) {
pr_info("rd %*pbl: Checking EAS, cpufreq policy not set for CPU: %d",
cpumask_pr_args(cpu_mask), i);
}
return false;
}
gov = policy->governor;
cpufreq_cpu_put(policy);
if (gov != &schedutil_gov) {
if (sched_debug()) {
pr_info("rd %*pbl: Checking EAS, schedutil is mandatory\n",
cpumask_pr_args(cpu_mask));
}
return false;
if (!cpufreq_ready_for_eas(cpu_mask)) {
if (sched_debug()) {
pr_info("rd %*pbl: Checking EAS: cpufreq is not ready\n",
cpumask_pr_args(cpu_mask));
}
return false;
}
return true;