This release introduces some large changes,
Features:
- swupd -t/--time flag to enable verbose timing output of swupd operations
(update, verify, bundle-add)
- verifytime service added to boot to check client systems are on a sane time
- Accept multiple bundles when using bundle-remove
Fixes:
- Attempt to automatically fix statedir corruption from bad files/tars
- Fix download and hysteresis logic on error paths and improve curl usage
- Improve return codes for bundle-add to cover various scenarios of
incorrect/correct bundle additions
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This is a wrapper around remove_bundle()
to add consistency for bundle-remove
subcommand respect bundle-add that accepts
one or more bundles to be removed.
This must not be the optimal implementation
but it works fine and gives a better user
usage experience.
To give libcurl an extra opportunity to do more work, add an additional
curl_multi_perform() call interleaved between curl_multi_wait() and the
attempt to process completed transfers.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The "numfds" variable set by curl_multi_wait() does not appear to be a
reliable indicator of how many transfers to try popping of the stack. I
went with this approach because the documented BKM, namely to check if
the "running" variable changes between calls to curl_multi_perform(), is
a bit inconvenient to integrate with swupd's code flow here.
Instead, try to empty the multi stack down the hysteresis lower bound,
and only check "numfds" to detect no activity at all. This approach
seems to perform better, resulting in a more balanced sequence of
additions and removals from the stack over time.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If a system loses network connectivity in the middle of fullfile
downloads for update (or verify --fix), and there are more pending
transfers to queue than remaining available file descriptors for the
process, the hysteresis upper bound is bypassed and swupd runs out of
open file descriptors.
To avoid this issue, the upper bound (plus a small buffer for leeway)
should be enforced in poll_fewer_than(). Since there is little
visibility into the state of libcurl as to why no downloads are
progressing, if the upper bound is bypassed, simply append the most
recently added file to the failed list.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
There may be in-progress transfers in the multi queue if
poll_fewer_than() returns an error, so a full clean of the multi queue
should be deferred.
The queue is already cleaned up immediately before the next download
retry loop begins; this is the best place for cleanup, since all
transfers have either completed or failed before the cleanup is
triggered.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If one of the poll_fewer_than() calls returned an error, a jump to the
out_bad label did not occur. Fix this issue, and also check error codes
the same way for both call sites.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Because the pointer in the struct file may have been NULL'ed previously
in free_curl_list_data(), the "curl" pointer declared on the stack is
not updated.
Since the free_curl_list_data() function already checks struct file
pointer instead, call it instead. swupd_download_file_complete() is also
called by free_curl_list_data(), so that call has been removed here.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Under certain error conditions for download retries, the curl multi list
cleanup routines are called more than once, leading to double frees for
CURL object pointers.
To protect against this problem, ensure that the pointers are reset to
NULL following the frees.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For various reasons - partial downloads, power failure, user exiting -
the /var/lib/swupd state directory may become corrupt. When this happens,
manifests can be rendered unusable, causing various errors in swupd such as
signature failures, or errors reading the manifest. This patch attempts to
avoid such things causing updates or verifies to fail by cleaning up the
Manifests artifacts of what it could not use, and re-downloading them to
try again.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Arrange for "bad names" to give non-zero exit code
Arrange for "do nothing" in particular when packages are already
installed to give a zero return code.
This means that typos in names result in failures, but that trying to
add an existing package is fine.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
The verbose_time flag should be set to false if getting time fails, so timing
is not attempted again later, letting swupd continue in its regular mode.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This will ensure the system has a more correct, usable time after boot,
which not only improves swupd reliability but other services' as well.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
There may be code doing something like:
grabtime_start(×, "Outer loop");
. <do work>
grabtime_start(×, "Inner loop");
.
. <do work>
.
grabtime_stop(×);
. <do work>
grabtime_stop(×);
The "Outer loop" section would not have been printed because the stop time
and completed flag would not have been set. This fixes it by checking if we
already completed the last time grab, and traverses backwards to find the
next incomplete one, which must be the outer one.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This introduces a -t/--time option to update, verify, and bundle-add commands, displaying raw elapsed time and CPU process time for integral swupd operations within the given code paths.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes a mismatch in the help menu description and long option,
and fixes the bash backwards compatibility check.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release updates and converts the man pages to RST format, updates the
swupd bash generation script, fixes error reporting when invalid bundle names
are given to bundle-add, and introduces a new tool to verify system time and
attempt to fix it if needed so signature verification does not fail due to
system time errors.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The system clock may be terribly off, especially on new hardware that has not
yet been calibrated. Updates rely on the certificate and system time being
sane to verify validity, so if a mismatch is found the certificate will
be deemed invalid and the update stopped. This patch attempts to fix the
system time to something sane using the time from the swupd binary itself,
which should not have been touched by any user except root. If the time is
normal and verification fails, the cert cannot be trusted.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Currently if you try and add a non existent bundle, e.g. "foo" you get
two lines of output
foo bundle name is invalid, skipping it...
bundle(s) already installed, exiting now
This is caused by the add_subscriptions function calling itself
recursivly but failing to pass up results in a meaningful way. This
change makes the return value of add_subscriptions be a bitmask so it
can signal errors and packages added distinctly.
I did think about changing this function to return a struct but
decided this was a step too far.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
I noticed this printf says something about getwd(), but that's not
actually being called directly there and that's not super useful to a
non-developer human anyway. So I translated the function name to an
english description.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
bash 4.4 added the 'nosort' option for completion. Use this so the
flag options come before the bundle names.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Include feature request from IRC to not offer os-core and
os-core-update bundles as completion targets for bundle-remove.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Remove the script which creates the completion script based on the
output of swupd --help. It wasn't being used (e.g. the completion
didn't have bundle-list in it).
Restructure the completion script to use a case statement to list the
valid completion options. IMHO this makes the code easier to
understand.
Add in package name completion for bundle-add. This requires
/var/lib/swupd/XXXXX/Manifest.MoM to exist (where XXXXX is the
contents of /var/lib/swupd/version), be in the correct format
etc.
Add in package name completion for bundle-remove. This uses the
contents of /usr/share/clear/bundles to get the list of installed
bundles. It would be nice to use $(swupd bundle-list) but it aborts if
it is not being run as root, so this means you can't have completion
for "sudo swupd bundle-remove".
TODO: Fix bundle completion if --path is specified.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
This release removes unused certificates from the client, and fixes the
makefile incorrectness that occured in the previous release.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The client does not do certificate pinning anymore, and it should not provide
certificates. This patch removes the unused certificates and clarifies the
certpath option.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes swupd to use the full path provided by the certpath
option and not append a hardcoded certname to it, fixes memory corruption
on multiple swupd_init calls, and fixes the lock file descriptor leaking.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The mixer and image creator treat the certpath as the full path of the
certificate filename, and swupd should too. If someone is overriding the
certificate with the cert path option, use the supplied string and don't
append a pre-defined name to it.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When swupd_init() is called more than once in
the same supwd run, pointer corruption ocurs on
some global variables causing memory corruption
and finally a SIGABRT. This patch fixes that
condition by properly setting all globals to
NULL when swupd_deinit() is called; or more
properly free_globals().
This release adds a bundle-list subcommand to make the cli more clear,
cleans up unused files in testing dirs, and ports the post update scripts
calls to the modern clr-boot-manager directly, which is able to handle various
kinds of kernel/boot updates. Support for automated building within a docker
environment has been added to allow developers to test their changes against
latest inside of a clean Docker container.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This change adds a simple "./continous.sh" script which will allow
developers to test their changes against "clearlinux:latest" within a
clean Docker container.
If the container does not already exist, it will be created on demand.
Future builds will be done near instantly within the container.
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
Explicitly mark the return results of system as unused. This helps to
cut down on the compiler spam as we (by design) do not check the return
results of the scripts portion.
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
In the current system we depend on the legacy boot infrastructure, which
has been provided by clr-boot-manager in the way of compatibility scripts.
These scripts all do the same thing, which is to invoke clr-boot-manager
with the "update" subcommand.
Given that clr-boot-manager doesn't need to know the context of the
operation, i.e. it is able to deduce whether kernel or bootloaders need
updating, regardless, it makes little sense to use any of these scripts,
and we should begin to deprecate them.
In clr-boot-manager 2.0, we will look to remove these compat scripts
completely, however they will continue to exist until then to facilitate
necessary format bumps, etc.
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
The swupd unit tests need a group of
options for execution environment.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
This adds are required standard options
to select content server, version server,
format, and so on.
Options added:
-u, url for version string and content file downloads
-c, url for content file downloads
-v, url for version string download
-p, path to verify
-F, format suffix for version file downloads
-n, Do not attempt to enforce certificate or signature checking
-S, Specify alternate swupd state directory
-C, Specify alternate path to swupd certificates
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
"bundle-add" sub-command used to validate "list" option in a unit test,
now that option is part of "bundle-list" sub-command with a new
name: [-a, all], for this reason the test has been updated in order to
validate it using "bundle-list" sub-command.
Furthermore this test has been moved to new directory called
"bundlelist/all" this in order to keep source code integrity.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
The option [-l, --list] is no longer part of "bundle-add"
sub-command, now this option has been taken by "bundle-list"
sub-command using a new name [-a, --all], this information
is updated in swupd man page and markdown file.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
The current option name (--list) for bundle-list sub-command
could sound redundant and confuse, [-a, --all] is a more
a more appropriate name since it can show "all" available bundles
in certain clear linux release.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
This option has been used to list all available bundles
in certain clear linux release, it is present in "bundle-add"
sub-command however at this moment this option is more consistent
if "bundle-list" sub-command get it.
This is in order to keep coherence and semantic.
Signed-off-by: Mario Alfredo Carrillo Arevalo <mario.alfredo.c.arevalo@intel.com>
Tudor Marcu