Now that the number of pending downloads is kept below a certain limit
(see poll_fewer_than()) it is possible to open files before starting
the transfer. Using the default curl write handler and explicit
open/close of the file makes the code simpler.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The previous approach was to open/fdopen/fclose the file for each
chunk that gets passed from curl. This incurrs a huge performance hit
when close() triggers a hashing of the file content on systems where
integrity protection via IMA is enabled.
Now the file is opened only once and kept open until the download is
complete. In addition, the unnecessary usage of C file IO is avoided.
The semantic is changed as little as possible:
- file gets created only after the first chunk of data arrived
- file descriptors do not leak to child processes (O_CLOEXEC)
- data gets appended to existing files (via O_APPEND, used
to keep the code simple and avoid an additional lseek)
- data gets flushed explicitly for each chunk (via fdatasync(),
which somewhat approximates the effect that an explicit
close() may have had)
As an additional improvement, failures during close() are checked. To
keep error handling as much as before, the completion function which has
the close() takes the current curl error code and replaces it if it
encounters a write error.
[v2 of the patch with fixes by Dmitry Rozhkov, see https://github.com/pohly/swupd-client/pull/1]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Swupd used to use a sorted blob of extended file attributes as part of
the data that was used to verify the contents. This caused problems if
extra attributes were added. In particular if a clearlinux system was
being run in a container that had selinux enabled in the base OS.
Add a configure option to allow the attributes to be considered or
not. Based on advice, the default is to ignore them, as this is what
everyone expects.
Tidied up passing a --selinux flag to tar, this is also handled by a new
config option.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
We spend a lot of wasted time parsing the entire manifest in some areas, when
only the header is needed. This patch adds a header_only flag that tells the
appropriate load manifest functions to quit early after the header is read,
instead of parsing the entire (possibly very large) manifest. Another
optimization is calling fopen() with the 'm' flag, which tries to use mmap to
access the file (for reading only).
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch cuts down the amount of recursion that happens by skipping a
subsequent rescursive calls to add_subscriptions if we hit a bundle that is
already subscribed.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Instead of printing the stats for what changed overall in the MoM, it's
more interesting to report what changed on their system. That is,
reporting which bundles are new, have changed, or (in the future) have
been deleted.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To avoid calling the version-setting function multiple times for a
single subscription list for 'update', simply reference both MoM manifests
in the function. For other subcommands that only reference a single MoM
manifest, the second argument should be NULL.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
A forthcoming commit will make 'update' understand two subscription
lists, one each for the current and latest versions, so using a single
global list will no longer be sufficient.
Instead, allocate a subs list for each subcommand that needs it, and
adjust helper function signatures that need to read/write the subs list.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Updates the order of preference for content_url, version_url
and format_string, from:
1. Runtime flags
2. Configure time options
3. State dir files
to:
1. Runtime flags
2. State dir files
3. Configure time options
This patch also changes the logic of the setter functions to allow
multiple calls. Once the respective global is set, that function will
return true or 0, ie "success", (depending on the function).
Signed-off-by: Brad T. Peters <brad.t.peters@intel.com>
Remaining hits by findstatic.pl are public API functions and
dangling statistic increments (but those last are fixed by the
PR cleaning up the stats, so they can be ignored for this PR).
This is a negligable performance increase and more of a cleanup
of symbol space. In order for these to inline, we need the stats
to be exported instead, so we go from 8+1 exports to 1+1.
Since signature initialization occurs in swupd_init(), use the same
convention as other init steps by defining/using a special error code.
As a side effect, the return value type is fixed (int vs bool) for the
signature init failure case.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To make compile-time options more easily discoverable at runtime, make
the --version output more verbose by printing a few of them. Other
options can be easily added in the future by adding an appropriate
AC_DEFINE call in configure.ac, and updating the new header added in
this commit (swupd-build-opts.h).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Calculate the hash for directories is desiderable to be independent
on the dirname due to the subsequent calculation on the staged/HASH
file. Here is used const "DIRECTORY" string for input name for
all folders.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
The poorly documented libcrypto in OpenSSL apparently does not like
its initialization and cleanup functions to be called but at process
start and exit. Swupd-client already has a swupd_init() so easy on
that side. But there was no common swupd_deinit() in which to place
terminate_signature(). I add one and put the common exit cleaners in it
and fix of a number of little inconsistencies around process exit cleanup
that have come to exist because there was a common cleaner.
With this, the signature verification of both the current and latest
MoM works, where prior only a first verification succeeded.
This patch also fixes a few memory leaks, though there are still many
memory leaks in swupd bundle-add and bundle-remove, unrelated to signature
verification. And swupd search is segfaulting due to how load_mom()
is introduced in 8cf0ef91dd.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
This patch adds functionality for the swupd client to do signature
verification on the MoM, which ensures a root of trust for the rest
of the update by guaranteeing the authenticity of the MoM and content
it includes.
As we focus now on just verifying the signed MoM, a number of functions now
become static to src/signature.c.
By default MoM signature verification is disabled. Configure
--enable-signature-verification to enable it. When enabled the default
cert for verification is /usr/share/clear/update-ca/ClearLinuxRoot.pem, as
specified by concatenation of SWUPDCERT onto UPDATE_CA_CERTS_PATH. The
SWUPDCERT can be overridden via configure --with-swupdcert=some.pem.
When signature verification is enabled, and the MoM's signature does _NOT_
verify, currently only a warning is presented but the swupd operation
continues. In the future signature verification will become mandatory.
We first need to sort out a few details with mixer to insure the right
thing happens there.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In order for the chain of trust rooted at the Manifest.MoM to operate
correctly, the bundle manifest hashes (as listed in the MoM) must be
checked. For now, warnings will be emitted when hash checks fail, but
they will become errors in the near future.
This commit simplifies/splits the load_manifests() interface into
load_mom() and load_manifest(), since load_manifests() was becoming too
complex, since the handling needs for MoMs versus bundle manifests is
sufficiently different. For the new load_manifest(), the logic is
changed to first download the manifest, then check the hash, and only
then load the manifest into memory.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When running get_latest_version, the swupd_curl_get_file would allocate
a version_container struct on the stack that would then be used by the
curl callback to keep track of the buffer offset. This use was invalid
however because the callback was run after the stack had been popped
which lead to undefined behavior.
Instead change the swupd_curl_get_file function to take the struct
itself so it will refer to memory valid for the entire length of the
call.
The swupd_curl_get_file function should likely be restructured at some
point so that in memory downloads are less of a hack (only able to
download a LINE_MAX worth of data) at some point however and this will
need to be updated again.
Add new function consolidate_files that replaces
consolidate_submanifests in order to remove the requirement of modifying
manifest components in place. This change enables creating seperate
consolidated file lists for bundle-add, one for new files to be
installed on the system and one for the currently installed bundles
files. Once these have been seperated out, only files requiring
installation can be processed by do_staging and the list of files
installed on the system can be used to run verify_fix_path.
This also modifies bundle-add to stop trying to create tracking
files which was impacting testing.
In order to support verify_fix_path from bundle_add, a generic file list
from bundle list needs to be implemented to replace the one embedded in
the current consolidate_submanifests function.
This function will be used to generate a file list to be passed to a new
function for consolidating a list of files (created in the following
patch).
Note the list of files this creates is not sorted.
To support low duplication lists of files living in multiple locations
that will be required for bundle_add to be able to run
swupd_verify_fix_path add a list_clone function which will create a copy
of a list without duplicating the list data.
When pack is going to be downloaded an option for
curl called resume is activated no matter what,
however although major http server support 'Range'
command that enables this functionality, not all
complies with this, when swupd client tries to
re-download a partial file from one of the server
that does not support 'Range' it will just throw
an error and stop to work until partial download
is deleted by hand.
This patch address that issue by checking first
whether server supports 'Range' command and if not
then disable 'resume' of packs.
Signed-off-by: Jaime A. Garcia <jaime.garcia.naranjo@intel.com>
For safety, 'verify --fix' does not use swupd_rm() (like 'update') when
deleting files, etc. Instead, each deleted entry is considered
individually.
So for the deletion step to work correctly with 'verify --fix', files
contained within directories must be listed before the directories
themselves. A simple solution is to make the manifest reverse filename
sorted (opposite of the current filename sorted order).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since we are either determining the current OS version, or the latest OS
version, at any given time, rename these functions to reflect that.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since most of these checks are used by all subcommands, and are written
in slightly different ways, move all of the checks into swupd_init() to
do the checks in one place and prevent misordering of these calls.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Instead of specifying a version number argument to -m allow passing the
latest string which will query for the latest version and use that as
the install target version.
This option overrides the global setup for state_dir
variable that is now used to find out where swupd
is going to do work (state dir).
If not --statedir option is passed along, the global
STATE_DIR definition value is going to be used instead,
this last one can also be changed at compilation time.
Signed-off-by: Jaime A. Garcia <jaime.garcia.naranjo@intel.com>
Similar to how the content URL, version URL, and format string globals
are now set, do a similar reorganization to set path_prefix in its own
function and update all subcommands to use this new function.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The intended behavior here is to check if the directory path for the
current file contains any symlinks, not the full path itself. By not
checking the directory path only, verify --fix would not delete any
symlinks that were marked deleted.
Also, because the directory path is being examined now, we can obtain
the directory file descriptor (dirfd) and use unlinkat(2) to delete
files and directories instead of unlink/rmdir; doing so avoids race
conditions in that paths may change between calling realpath() and
unlink().
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Previously include manifests that were not newer than the current
version would be silently ignored even if they had not been previously
installed causing bundles to miss included dependencies.
Fix this by adding a is_tracked field to struct file that is nonzero
when the bundle the file is contained in is marked as tracked via having
a /usr/share/clear/bundles/$name file.
The is_tracked == 0 version of the file will be dropped in the
consolidate_manifest phase if that filename from another bundle has
is_tracked != 0 to avoid downloading and installing an already existing
file.
The is_tracked == 0 version of the file will then be added to the update
list from the consolidated manifest if that file was not previously
installed on the system (in addition to the other cases it would already
have been added to the list for).
Because combinations of using the libcurl FILE protocol and HTTP/HTTPS
are not supported, make sure that the two protocols specified for -c and
-v match.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Instead of hardcoding the default version URL and content URL in the
binary, read the defaults from config files that live in
/usr/share/defaults/swupd/.
If the config files do not exist, then users can pass the -c or -v
options as necessary (or -u, which sets the same value for -c and -v).
The primary motivation for this change is to enable a better experience
for the swupd mixer; users mixing a version of Clear Linux can set the
default URLs at mix time instead of always passing -c/-v/-u at runtime.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Add support in update to properly support bundles that include other
bundles inside them. This feature is intended to work with bundle
additions by adding included bundles (from bundles already subscribed)
to the subscription list prior to downloading packs and full files.
As part of this change the add_subscriptions function from bundle.c was
modified to handle retrying manifest downloads.
Instead of taking in versions as arguments to download_subscribed_packs,
just use the version information in the subs list items (set by calling
subscription_versions_from_MoM function).
Because the subs list may contain bundles that are new in the version
being updated to, don't set an error in the subs item for things not in
a particular manifest version.
The latest_version in check_versions and read_versions was a legacy
variable used for btrfs support that no longer exists and was a
duplicate of current_version in all use cases. This patch removes the
variable completely from the above functions.
Since GNU tar fails to extract files with xattrs preserved when
Integrity Measurement Architecture (IMA) is enabled some vendors
may choose to install libarchive-based tar (bsdtar) on their embedded
devices, so the swupd server needs to be able to create archives
in its format.
This patch adds one compile-time options --enable-bsdtar that is used
to enable/disable GNU tar specific options.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
Eventually manifests will be able to include other manifests and this
patchset adds support for adding those included manifests when running
bundle-add on the including manifest.
Start handling the case where a manifest has an includes line. This line
will be used to specify other manifests that are to be installed with
the containing manifest.
Pull out code that only can be run once into a bundle installer frontend
function and move code that could be run recursively into an another
function. This is done in preparation for manifest includes.
A user was running into issues installing the kernel-container bundle on
Clear Linux. Namely, all missing files marked as boot were not
installed.
The root issue is that the bundle-add action does not set the "fix"
global variable before calling ignore(), so ignore() mistakenly returns
true when considering boot files that are missing, but not marked
deleted.
Because 'bundle-add' should be installing new boot files, and 'verify
--fix' should be repairing boot files if they have mismatching hashes,
remove the global "fix" variable entirely, since in all current call
sites, the "fix" behavior is expected.
Also, add another call to ignore() in remove_orphaned_files() to ensure
that any file that should be ignored (including deleted boot files) are
not removed during a 'verify --fix'.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To support swupd-client testing without connecting to a web server, it's
convenient to use libcurl's FILE protocol, which will query files on the
local filesystem instead.
Because the libcurl-multi download path for fullfiles has treated the
"0" response code as an error, but it has the opposite meaning when
using the FILE protocol, only set the error when connecting to a web
server.
Also, the libcurl-easy code for synchronous downloads didn't treat "0"
specially, so this commit ensures the proper handling, as well as
setting a generic download error for the FILE protocol specific
CURLE_FILE_COULDNT_READ_FILE.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The verify_fix_path function takes a path and a consolidated MOM as its arguments.
It breaks down the path into subpaths and checks if each subpath is missing.
If found missing, the path is searched for in the consolidated manifest for its hash
and downloaded synchronously. It returns success if all that subpaths are verified
and fixed.
Patrick Ohly