The current implementation is not setup to optimally load manifests and attempt deltas.
This changes the flow to be as follows:
1.) Load current Manifest.MoM
2.) Load server Manifest.MoM
3.) Recurse and load all current manifests, since we know exactly which are needed already
4.) Link current and server Manifest.MoMs, setting versions and peers
5.) Load server manifests recursively adding their includes
6.) Recurse the server manifests to find and load any new included bundles
7.) Link current and server Manifest.MoMs again to account for new bundles
When loading the server manifests, deltas will attempt to be applied to
current manifests before doing a full download. Following this flow enables
swupd to load manifests in a logical order, and short circuit trying to
reload them later on as manifests are recursed for includes again later.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The current version must be passed to add_subscriptions, else it will call
load_manifest() with equal current and server versions and never enter the
try_delta_manifest code path.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
We must not run these with --no-block because order must be preserved,
and critical components may need to be reexec'd, making this a core piece
of the swupd update process.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
For various reasons - partial downloads, power failure, user exiting -
the /var/lib/swupd state directory may become corrupt. When this happens,
manifests can be rendered unusable, causing various errors in swupd such as
signature failures, or errors reading the manifest. This patch attempts to
avoid such things causing updates or verifies to fail by cleaning up the
Manifests artifacts of what it could not use, and re-downloading them to
try again.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This introduces a -t/--time option to update, verify, and bundle-add commands, displaying raw elapsed time and CPU process time for integral swupd operations within the given code paths.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The system clock may be terribly off, especially on new hardware that has not
yet been calibrated. Updates rely on the certificate and system time being
sane to verify validity, so if a mismatch is found the certificate will
be deemed invalid and the update stopped. This patch attempts to fix the
system time to something sane using the time from the swupd binary itself,
which should not have been touched by any user except root. If the time is
normal and verification fails, the cert cannot be trusted.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This sets swupd signature verification to mandatory, failing and exiting
for any verify error, though --nosigcheck enables user to bypass all
signature-based security checks.
The signed MoM serves as the top level chain of trust, and it is used to
extend content trust down to the individual file level. When the signature
of the top-level MoM is invalid or cannot be verified for any reason, we
warn and abort the operation.
Passing --nosigcheck will allow this to proceed, explicitly accepting the
unverifiable MoM and outputting a log entry to the Journal. This is not
recommended and unsupported by upstream once the chain of security is broken,
because it may imply update content was or may be installed that was not
generated by the official upstream.
Signed-off-by: Brad T. Peters <brad.t.peters@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This is a new sub-command for swupd called
"bundle-list", this command will show which
bundles are installed in the local system.
This information is obtained reading
/usr/share/clear/bundles path in local filesystem.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
This function reads installed bundles from local standard
directory and store them in a list pointer passed as argument, this
function is the base for functionality as listing installed bundles to
user.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
There is a need to override the cert path at runtime, so migrate to
using a global variable that is set in init_globals().
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
We change the telemetry API to pass a class record identifier,
which will be part of the telemetry record file name. This allows
us easier to send different telemetry class records to the server,
which will make parsing simpler in the probe and on the backend.
Rev record version to #2 because of this change.
The telemetry function only works insire swupd_init() / swupd_deinit()
functions, as it accesses state_dir from globals. This restricts the
use of it a bit, so we move the timekeeping function into
main_update().
Aside from update, I've added a telemetry point in verify as well.
With this, we have several easy telemetry points in swupd that
should give us an indication how swupd is performing without divulging
lots of detailed information.
The swupd.h file contains easy defines for level, and the telemetry()
function itself is as failsafe as it can be - any error results in
a continuing swupd program, although these errors would occur in
a system that is already hopeless (disk full) anyway.
The comment from commit fc0f570 was a bit misleading. Adding more
information about which combinations are valid and tested instead.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Now that the number of pending downloads is kept below a certain limit
(see poll_fewer_than()) it is possible to open files before starting
the transfer. Using the default curl write handler and explicit
open/close of the file makes the code simpler.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The previous approach was to open/fdopen/fclose the file for each
chunk that gets passed from curl. This incurrs a huge performance hit
when close() triggers a hashing of the file content on systems where
integrity protection via IMA is enabled.
Now the file is opened only once and kept open until the download is
complete. In addition, the unnecessary usage of C file IO is avoided.
The semantic is changed as little as possible:
- file gets created only after the first chunk of data arrived
- file descriptors do not leak to child processes (O_CLOEXEC)
- data gets appended to existing files (via O_APPEND, used
to keep the code simple and avoid an additional lseek)
- data gets flushed explicitly for each chunk (via fdatasync(),
which somewhat approximates the effect that an explicit
close() may have had)
As an additional improvement, failures during close() are checked. To
keep error handling as much as before, the completion function which has
the close() takes the current curl error code and replaces it if it
encounters a write error.
[v2 of the patch with fixes by Dmitry Rozhkov, see https://github.com/pohly/swupd-client/pull/1]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Swupd used to use a sorted blob of extended file attributes as part of
the data that was used to verify the contents. This caused problems if
extra attributes were added. In particular if a clearlinux system was
being run in a container that had selinux enabled in the base OS.
Add a configure option to allow the attributes to be considered or
not. Based on advice, the default is to ignore them, as this is what
everyone expects.
Tidied up passing a --selinux flag to tar, this is also handled by a new
config option.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
We spend a lot of wasted time parsing the entire manifest in some areas, when
only the header is needed. This patch adds a header_only flag that tells the
appropriate load manifest functions to quit early after the header is read,
instead of parsing the entire (possibly very large) manifest. Another
optimization is calling fopen() with the 'm' flag, which tries to use mmap to
access the file (for reading only).
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch cuts down the amount of recursion that happens by skipping a
subsequent rescursive calls to add_subscriptions if we hit a bundle that is
already subscribed.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Instead of printing the stats for what changed overall in the MoM, it's
more interesting to report what changed on their system. That is,
reporting which bundles are new, have changed, or (in the future) have
been deleted.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To avoid calling the version-setting function multiple times for a
single subscription list for 'update', simply reference both MoM manifests
in the function. For other subcommands that only reference a single MoM
manifest, the second argument should be NULL.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
A forthcoming commit will make 'update' understand two subscription
lists, one each for the current and latest versions, so using a single
global list will no longer be sufficient.
Instead, allocate a subs list for each subcommand that needs it, and
adjust helper function signatures that need to read/write the subs list.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Updates the order of preference for content_url, version_url
and format_string, from:
1. Runtime flags
2. Configure time options
3. State dir files
to:
1. Runtime flags
2. State dir files
3. Configure time options
This patch also changes the logic of the setter functions to allow
multiple calls. Once the respective global is set, that function will
return true or 0, ie "success", (depending on the function).
Signed-off-by: Brad T. Peters <brad.t.peters@intel.com>
Remaining hits by findstatic.pl are public API functions and
dangling statistic increments (but those last are fixed by the
PR cleaning up the stats, so they can be ignored for this PR).
This is a negligable performance increase and more of a cleanup
of symbol space. In order for these to inline, we need the stats
to be exported instead, so we go from 8+1 exports to 1+1.
Since signature initialization occurs in swupd_init(), use the same
convention as other init steps by defining/using a special error code.
As a side effect, the return value type is fixed (int vs bool) for the
signature init failure case.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To make compile-time options more easily discoverable at runtime, make
the --version output more verbose by printing a few of them. Other
options can be easily added in the future by adding an appropriate
AC_DEFINE call in configure.ac, and updating the new header added in
this commit (swupd-build-opts.h).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Calculate the hash for directories is desiderable to be independent
on the dirname due to the subsequent calculation on the staged/HASH
file. Here is used const "DIRECTORY" string for input name for
all folders.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
The poorly documented libcrypto in OpenSSL apparently does not like
its initialization and cleanup functions to be called but at process
start and exit. Swupd-client already has a swupd_init() so easy on
that side. But there was no common swupd_deinit() in which to place
terminate_signature(). I add one and put the common exit cleaners in it
and fix of a number of little inconsistencies around process exit cleanup
that have come to exist because there was a common cleaner.
With this, the signature verification of both the current and latest
MoM works, where prior only a first verification succeeded.
This patch also fixes a few memory leaks, though there are still many
memory leaks in swupd bundle-add and bundle-remove, unrelated to signature
verification. And swupd search is segfaulting due to how load_mom()
is introduced in 8cf0ef91dd.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
This patch adds functionality for the swupd client to do signature
verification on the MoM, which ensures a root of trust for the rest
of the update by guaranteeing the authenticity of the MoM and content
it includes.
As we focus now on just verifying the signed MoM, a number of functions now
become static to src/signature.c.
By default MoM signature verification is disabled. Configure
--enable-signature-verification to enable it. When enabled the default
cert for verification is /usr/share/clear/update-ca/ClearLinuxRoot.pem, as
specified by concatenation of SWUPDCERT onto UPDATE_CA_CERTS_PATH. The
SWUPDCERT can be overridden via configure --with-swupdcert=some.pem.
When signature verification is enabled, and the MoM's signature does _NOT_
verify, currently only a warning is presented but the swupd operation
continues. In the future signature verification will become mandatory.
We first need to sort out a few details with mixer to insure the right
thing happens there.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In order for the chain of trust rooted at the Manifest.MoM to operate
correctly, the bundle manifest hashes (as listed in the MoM) must be
checked. For now, warnings will be emitted when hash checks fail, but
they will become errors in the near future.
This commit simplifies/splits the load_manifests() interface into
load_mom() and load_manifest(), since load_manifests() was becoming too
complex, since the handling needs for MoMs versus bundle manifests is
sufficiently different. For the new load_manifest(), the logic is
changed to first download the manifest, then check the hash, and only
then load the manifest into memory.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When running get_latest_version, the swupd_curl_get_file would allocate
a version_container struct on the stack that would then be used by the
curl callback to keep track of the buffer offset. This use was invalid
however because the callback was run after the stack had been popped
which lead to undefined behavior.
Instead change the swupd_curl_get_file function to take the struct
itself so it will refer to memory valid for the entire length of the
call.
The swupd_curl_get_file function should likely be restructured at some
point so that in memory downloads are less of a hack (only able to
download a LINE_MAX worth of data) at some point however and this will
need to be updated again.
Add new function consolidate_files that replaces
consolidate_submanifests in order to remove the requirement of modifying
manifest components in place. This change enables creating seperate
consolidated file lists for bundle-add, one for new files to be
installed on the system and one for the currently installed bundles
files. Once these have been seperated out, only files requiring
installation can be processed by do_staging and the list of files
installed on the system can be used to run verify_fix_path.
This also modifies bundle-add to stop trying to create tracking
files which was impacting testing.
In order to support verify_fix_path from bundle_add, a generic file list
from bundle list needs to be implemented to replace the one embedded in
the current consolidate_submanifests function.
This function will be used to generate a file list to be passed to a new
function for consolidating a list of files (created in the following
patch).
Note the list of files this creates is not sorted.
To support low duplication lists of files living in multiple locations
that will be required for bundle_add to be able to run
swupd_verify_fix_path add a list_clone function which will create a copy
of a list without duplicating the list data.
When pack is going to be downloaded an option for
curl called resume is activated no matter what,
however although major http server support 'Range'
command that enables this functionality, not all
complies with this, when swupd client tries to
re-download a partial file from one of the server
that does not support 'Range' it will just throw
an error and stop to work until partial download
is deleted by hand.
This patch address that issue by checking first
whether server supports 'Range' command and if not
then disable 'resume' of packs.
Signed-off-by: Jaime A. Garcia <jaime.garcia.naranjo@intel.com>
For safety, 'verify --fix' does not use swupd_rm() (like 'update') when
deleting files, etc. Instead, each deleted entry is considered
individually.
So for the deletion step to work correctly with 'verify --fix', files
contained within directories must be listed before the directories
themselves. A simple solution is to make the manifest reverse filename
sorted (opposite of the current filename sorted order).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since we are either determining the current OS version, or the latest OS
version, at any given time, rename these functions to reflect that.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since most of these checks are used by all subcommands, and are written
in slightly different ways, move all of the checks into swupd_init() to
do the checks in one place and prevent misordering of these calls.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Instead of specifying a version number argument to -m allow passing the
latest string which will query for the latest version and use that as
the install target version.
This option overrides the global setup for state_dir
variable that is now used to find out where swupd
is going to do work (state dir).
If not --statedir option is passed along, the global
STATE_DIR definition value is going to be used instead,
this last one can also be changed at compilation time.
Signed-off-by: Jaime A. Garcia <jaime.garcia.naranjo@intel.com>
Similar to how the content URL, version URL, and format string globals
are now set, do a similar reorganization to set path_prefix in its own
function and update all subcommands to use this new function.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Tudor Marcu