mirror of
https://github.com/clearlinux/clear-linux-documentation.git
synced 2026-09-05 13:21:27 +00:00
Edited text for grammar, flow, and conciseness.
Signed-off-by: MCamp859 <mary.camp@ptiglobal.net>
This commit is contained in:
MCamp859
@@ -4,88 +4,68 @@ Restarting system services after an OS update
|
||||
#############################################
|
||||
|
||||
The software life cycle describes how software is created, developed, and
|
||||
deployed, but also includes how to replace or update software. A good OS
|
||||
deployed, and includes how to replace or update software. A good OS
|
||||
provides tools for the entire software life cycle. These tools must include
|
||||
ways to remove software components properly when replaced with something else.
|
||||
|
||||
Most of the work on software update code in |CL| has primarily focused on
|
||||
adding new software to the system. Aside from recommending to users to reboot
|
||||
their system once in a while to ensure the new software is running, we did not
|
||||
provide any tools to do this easily, until now.
|
||||
adding new software to the system. We recommend that users reboot their system
|
||||
once in a while, but we did not provide any tools to restart services easily,
|
||||
until now.
|
||||
|
||||
User challenges
|
||||
***************
|
||||
|
||||
Users might find it difficult to determine which services to restart. Users
|
||||
can either evaluate each system and reboot manually, or figure out which
|
||||
services to restart based on documentation like the |CL| Release Notes.
|
||||
Since neither option solves the issue for many users, we established
|
||||
improvement in this area was needed.
|
||||
It can be difficult to determine which services to restart. You can either
|
||||
evaluate each system and reboot manually, or figure out which services to
|
||||
restart based on documentation like the |CL| release notes. Since neither
|
||||
option solves the issue completely, the |CL| team created a solution.
|
||||
|
||||
Over the years, several OSes have approached the problem and come up with
|
||||
partial solutions. Some solutions are based on three steps:
|
||||
Over the years, several OSes approached the problem and created partial
|
||||
solutions such as the following:
|
||||
|
||||
#. Mark updates requiring a reboot, such as kernel updates.
|
||||
#. Inform the user of those updates.
|
||||
#. Ask usersto restart.
|
||||
* Automatically restart services during an upgrade.
|
||||
* Evaluate services using these steps:
|
||||
|
||||
Other variants automatically restart services during an upgrade. These
|
||||
solutions are acceptable since there is nothing wrong with regularly
|
||||
restarting services when your OS does not update automatically. Users are
|
||||
already logged into a terminal, aware of updates happening, and ready to
|
||||
respond to messages and notices about additional actions to take after an OS
|
||||
update.
|
||||
* Mark updates requiring a reboot, such as kernel updates.
|
||||
* Inform the user of those updates.
|
||||
* Ask the user to restart.
|
||||
|
||||
However, |CL| updates software automatically and users never see any notices
|
||||
from the updater. Users can still review the journal, which stores updater
|
||||
notices.
|
||||
Both solutions are acceptable for many OSes. However, |CL| updates software
|
||||
automatically and users do not see notices from the updater unless they review
|
||||
the journal.
|
||||
|
||||
This scenario requires a completely different solution. Thus, we arrive at the
|
||||
following solution requirements:
|
||||
|CL| requires a completely different solution, with the following
|
||||
requirements:
|
||||
|
||||
* Eliminate the guesswork about what to restart and under what circumstances.
|
||||
* Cannot restart everything. Many service deamons do not yet support an automatic
|
||||
* Cannot restart everything. Many service daemons do not support an automatic
|
||||
background restart.
|
||||
* Fit into the |CL| architectural perspective: be small, quick, and lean.
|
||||
|
||||
Enter :command:`clr-service-restart`
|
||||
************************************
|
||||
:command:`clr-service-restart` functionality
|
||||
********************************************
|
||||
|
||||
After exploring possible solutions, we devised a simple and efficient method
|
||||
of catching the most typical reasons to restart a service daemon:
|
||||
The most typical reasons to restart a service daemon include:
|
||||
|
||||
* The new version replaced the executable file itself.
|
||||
* The new version replaces a library component used by a service daemon.
|
||||
* A new version replaces the executable file itself.
|
||||
* A new version replaces a library component used by a service daemon.
|
||||
|
||||
Both cases are extremely common. Thus, any solution that can detect these situations will provide good results for
|
||||
our purposes. There are other cases to consider, for instance, when
|
||||
configuration files or shared data files change on disk, but this is less
|
||||
common. Our method focuses on restarting daemons when it is really needed,
|
||||
especially in the case of security updates.
|
||||
Our method restarts daemons for both cases by reading various files in the
|
||||
:file:`procfs` filesystem provided by the kernel.
|
||||
|
||||
The kernel actually provides us with this exact information in the
|
||||
:file:`procfs` filesystem. Both the deleted executable file case and the
|
||||
upgraded library component file case are detectable by reading
|
||||
various files in the :file:`procfs` filesystem.
|
||||
The second part of the problem is to determine whether or not running
|
||||
processes are part of a system service. (The tool focuses on system services
|
||||
because most system services are background tasks with no direct user
|
||||
interaction.) Fortunately, :command:`systemd` provides a simple way to:
|
||||
|
||||
The second part of the problem is to determine whether or not actual running
|
||||
processes are part of a system service. The reason we're only interested
|
||||
in system services is because that is where the risk is large for
|
||||
security issues, and most system services are background tasks that
|
||||
have no direct user interaction. We did not consider restarting things such
|
||||
as desktop tasks because it would ruin any sort of user experience quickly.
|
||||
* Determine which active tasks are within the system domain.
|
||||
* Determine which service daemon maps to the task.
|
||||
|
||||
Fortunately, :command:`systemd` provides us with a simple way of determining
|
||||
what tasks are active and entirely within the system domain, and we can
|
||||
directly determine what service daemon they are part of. This approach
|
||||
eliminates any daemon that is not started to begin with.
|
||||
We combined both solutions into a low-overhead tool that shows which system
|
||||
daemons need a restart, as shown below:
|
||||
|
||||
The combination of these two partial solutions effectively gives us a nice
|
||||
tool, with very little overhead, that shows which system daemons
|
||||
are in need of a restart:
|
||||
|
||||
Figure 1: Invoking :command:`clr-service-restart' to show all system
|
||||
services in need of a restart
|
||||
Figure 1: Invoking :command:`clr-service-restart`
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
@@ -100,133 +80,105 @@ are in need of a restart:
|
||||
/usr/bin/systemctl --no-ask-password try-restart NetworkManager.service
|
||||
....
|
||||
|
||||
Making it `safe`
|
||||
****************
|
||||
:command:`clr-service-restart` implements a whitelist to identify which
|
||||
daemons can be restarted. The system administrator maintains a customized
|
||||
overlay of the default |CL| OS whitelist. When a software update occurs,
|
||||
:command:`clr-service-restart` consults the whitelist to see if a service
|
||||
daemon is allowed to be restarted or not. See the options section below for
|
||||
details.
|
||||
|
||||
Now that we know exactly what daemons should be restarted, we can go
|
||||
ahead and restart all of them, right? Not so fast! You definitely
|
||||
don't want to do this automatically. Restarting the gdm.service automatically
|
||||
on all the PCs in the office will not get you brownie points.
|
||||
|
||||
Before doing the restart, we need some selection mechanism to figure out what
|
||||
is OK to restart, and what is not. Since we want to restart the important
|
||||
processes automatically and in the background, we want to be conservative
|
||||
and start with low-risk daemons that we know are restartable without
|
||||
any significant impact. So, we need a whitelist.
|
||||
|
||||
We've implemented this whitelist in |CL| by making
|
||||
:command:`clr-service-restart` a one-stop tool to manipulate this
|
||||
whitelist. By default, the |CL| OS provides a basic set of whitelist entries
|
||||
that permit :command:`clr-service-restart`
|
||||
to perform automatic restarts of service daemons if they are whitelisted
|
||||
already. On top of that, the system administrator can maintain
|
||||
their own overlay customization of the whitelist, by allowing them to either
|
||||
'allow' or 'disallow' restarting of system services as well. When a software
|
||||
update occurs, the whitelist is consulted to see if a service daemon is
|
||||
allowed to be restarted or not.
|
||||
|
||||
This gives local administrators full control over the process. Either the
|
||||
local administrator can turn restarting all off, or they can add more
|
||||
services to the list if needed.
|
||||
|
||||
Options for :command:`clr-service-restart`
|
||||
==========================================
|
||||
******************************************
|
||||
|
||||
The :option:`allow` option tells :command:`clr-service-restart` that on this
|
||||
machine, the tallow daemon may be restarted automatically after an OS
|
||||
software update. The :command:`clr-service-restart` daemon creates the
|
||||
appropriate symlink in :file:`/etc/clr-service-restart` to keep a record
|
||||
of this.
|
||||
The :option:`allow` option identifies a daemon to restart after an OS software
|
||||
update. The :command:`clr-service-restart` daemon creates a symlink in
|
||||
:file:`/etc/clr-service-restart` as a record. The example below tells
|
||||
:command:`clr-service-restart` to restart the :option:`tallow` daemon after an
|
||||
OS software update.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ sudo clr-service-restart allow tallow.service
|
||||
|
||||
The :option:`disallow` command tells :command:`clr-service-restart` that
|
||||
although the OS defaults permit :option:`rngd` to be restarted, it should not
|
||||
be done on this particular system. A symlink is created in
|
||||
:file:`/etc/clr-service-restart` that points to :file:`/dev/null` to record
|
||||
this. Note that you don't need to add the :option:'.service' part to refer to
|
||||
the correct unit name.
|
||||
The :option:`disallow` option tells :command:`clr-service-restart` not to
|
||||
restart the specified daemon even if the OS defaults permit the daemon to be
|
||||
restarted. The :command:`clr-service-restart` daemon creates a symlink in
|
||||
:file:`/etc/clr-service-restart` that points to :file:`/dev/null` as a record.
|
||||
The example below tells :command:`clr-service-restart` not to restart the
|
||||
:option:`rngd` daemon after an OS software update.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ sudo clr-service-restart disallow rngd
|
||||
|
||||
The :option:`default` command makes :command:`clr-service-restart` revert
|
||||
back to the OS provided defaults and delete any symlink in
|
||||
:file:`/etc/clr-service-restart`. In the following example, it causes
|
||||
:option:`rngd` to be restarted automatically again, because it is whitelisted
|
||||
for automatic service restarts by default in |CL|.
|
||||
The :option:`default` option makes :command:`clr-service-restart` revert back
|
||||
to the OS defaults and delete any symlink in :file:`/etc/clr-service-restart`.
|
||||
The example below tells :command:`clr-service-restart` to restart
|
||||
:option:`rngd` automatically again, because :option:`rngd` is whitelisted for
|
||||
automatic service restarts by default in |CL|.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ sudo clr-service-restart default rngd
|
||||
|
||||
Checking in on :command:`clr-service-restart`
|
||||
*********************************************
|
||||
Monitoring options for :command:`clr-service-restart`
|
||||
=====================================================
|
||||
|
||||
:command:`clr-service-restart` works in the background and is invoked by
|
||||
:command:`swupd` automatically. The journal output shows
|
||||
that services are being restarted once in a while after an OS software update.
|
||||
:command:`clr-service-restart` works in the background and is invoked with
|
||||
:command:`swupd` automatically. You can review the journal output to verify
|
||||
that services are restarted after an OS software update.
|
||||
|
||||
This still means that there are system services that are not being restarted,
|
||||
so you may want check on the current situation and potentially take action.
|
||||
To enable this, :command:`clr-service-restart` can be invoked with two
|
||||
additional options, which are described below.
|
||||
To monitor :command:`clr-service-restart` use one or both options described
|
||||
below.
|
||||
|
||||
:option:`-n`
|
||||
|
||||
This option makes :command:`clr-service-restart` perform no restarts. Instead
|
||||
it displays the services that could potentially be restarted. When used,
|
||||
:command:`clr-service-restart` outputs a list of messages showing which
|
||||
service needs a restart, what unit it is, and why it needs a restart. Also,
|
||||
it shows the command that it would have used to restart this unit.
|
||||
:command:`clr-service-restart` outputs a list of messages showing:
|
||||
|
||||
* Which service needs a restart.
|
||||
* What unit it is.
|
||||
* Why it needs a restart.
|
||||
* Which command is required to restart this unit.
|
||||
|
||||
:option:`-a`
|
||||
|
||||
This option makes :command:`clr-service-restart` consider all system
|
||||
services, not just the ones that are whitelisted. Because the default
|
||||
whitelist in |CL| is relatively short at the moment, you may want to use this
|
||||
option to restart everything that it can find when you log in on the system.
|
||||
This option makes :command:`clr-service-restart` consider all system services,
|
||||
not only the ones that are whitelisted. Because the default whitelist in |CL|
|
||||
is relatively short, you can use this option to restart all impacted services
|
||||
when you log in on the system.
|
||||
|
||||
If you pass both options (:option:`-a` and :option:`-n`), :command:`clr-
|
||||
service-restart` displays a complete list of system services that need a
|
||||
restart. Use both options to verify that all desired daemons have been
|
||||
restarted.
|
||||
|
||||
Of course, if you pass both options (:option:`-a` and :option:`-n`), you will
|
||||
see the complete list of system services that need a restart. This is
|
||||
very handy to answer the question whether you may have missed restarting
|
||||
another daemon or not, without actually doing this inadvertently.
|
||||
|
||||
Telemetry
|
||||
*********
|
||||
|
||||
The |CL| team has attempted to minimize any disruption caused by this tool,
|
||||
but we realize that it can cause problems, either in the form of a short
|
||||
service outage when a daemon is being restarted, or worse, when a daemon
|
||||
fails to properly restart.
|
||||
:command:`clr-service-restart` may cause problems such as a short service
|
||||
outage when a daemon is being restarted, or if a daemon fails to properly
|
||||
restart. To minimize issues, :command:`clr-service-restart` creates a
|
||||
telemetry record and sends it to the optional |CL| telemetry service if both
|
||||
conditions below are met:
|
||||
|
||||
For that reason, we have created a whitelist to ensure units that we do not
|
||||
trust to restart are left alone, however, if something bad does happen, we
|
||||
want to have a way of catching it early.
|
||||
* If a unit fails to automatically restart after an OS update.
|
||||
* If that unit resides in the system location :file:`/usr/lib/systemd/system`.
|
||||
|
||||
We've implemented this as follows:
|
||||
If we find a unit that fails to automatically restart after an OS update,
|
||||
and that unit resides in the system location :file:`/usr/lib/systemd/system`,
|
||||
then we create a telemetry record and
|
||||
send that record to the optional |CL| telemetry service.
|
||||
|
||||
If you don't have the |CL| telemetrics bundle installed, nothing will happen
|
||||
with this data - it is discarded. However, if you do have the
|
||||
telemetrics bundle installed, and you opted in to send telemetry, then
|
||||
the name of the system unit is sent to our telemetry service. This allows
|
||||
us to quickly correct mistakes and remove services that are not as
|
||||
safe to restart as we thought.
|
||||
If you do not install the |CL| telemetrics bundle, the data is discarded. If
|
||||
you install the telemetrics bundle and you opt to send telemetry, then the
|
||||
system unit name is sent to the |CL| telemetry service. We evaluate the report
|
||||
and update the whitelist to remove services that are not safe to restart.
|
||||
|
||||
Conclusion
|
||||
**********
|
||||
|
||||
As always, we enjoy coming up with simple and efficient solutions that make
|
||||
our |CL| users' lives easier, and we love to share them with a bigger
|
||||
audience. We've made a github project of :command:`clr-service-restart` and
|
||||
you are invited to look at the code, share your thoughts, and work with us on
|
||||
improving the project. You can find the project at:
|
||||
The |CL| team enjoys coming up with simple and efficient solutions to make
|
||||
your work easier. We made a github project of :command:`clr-service-restart`
|
||||
and we invite you to look at the code, share your thoughts, and work with us
|
||||
on improving the project. You can find the project at:
|
||||
|
||||
https://github.com/clearlinux/clr-service-restart
|
||||
Reference in New Issue
Block a user