This adds telemetry status output for bundle add and remove.
In the case of remove, we expect a single bundle name.
In the case of remove, we make a nice string with all the bundle
names as we iterate and use that in the telemetry output.
We change the telemetry API to pass a class record identifier,
which will be part of the telemetry record file name. This allows
us easier to send different telemetry class records to the server,
which will make parsing simpler in the probe and on the backend.
Rev record version to #2 because of this change.
This release includes several significant fixes and additions to swupd-client:
- Renable telemetry in the client with a new implementation, writing records to
a file and not blocking swupd operations at all.
- Enable travis-ci integration
- Fix xattrs enable flag in conjunction with enabling bsdtar
- Ignore xattrs when processing manifests
- Enable out of tree builds
- Enable automatic updates through timer units
- Adds man pages
- Re-enable updating of config files
- Remove block on post-update triggers, saving another 1.5-2 seconds for update
- Miscellaneous fixes for typos and updates to autotools files for completeness
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The telemetry function only works insire swupd_init() / swupd_deinit()
functions, as it accesses state_dir from globals. This restricts the
use of it a bit, so we move the timekeeping function into
main_update().
Aside from update, I've added a telemetry point in verify as well.
With this, we have several easy telemetry points in swupd that
should give us an indication how swupd is performing without divulging
lots of detailed information.
The swupd.h file contains easy defines for level, and the telemetry()
function itself is as failsafe as it can be - any error results in
a continuing swupd program, although these errors would occur in
a system that is already hopeless (disk full) anyway.
This folder will be collecting output for telemetry. The
output will only ever be written by swupd. A separate
component will be collecting these telemetrics at a later
time to send them.
Cleaning this folder should be done through tmpfiles.d.
When IMA or Smack are active on the client, the downloaded Manifest
files will be assigned certain xattrs (security.ima
resp. security.SMACK64). Those xattrs did not exist on the server side
(because it is most likely not having those kernel features enabled)
and besides, the swupd-server code wouldn't include them in the
Manifest hashes even if they existed (see write_manifest_plain() in
src/manifest.c).
Therefore the client must ignore xattrs when verifying Manifest files.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
For now, the config simply installs missing build dependencies and works
around the umask discrepancy for running the functional test suite.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Ubuntu's /bin/sh is Dash, which has different error strings than Bash
for equivalent errors.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Depending on how libcurl is configured, CURLOPT_PIPEWAIT may not be
supported, so make the unsupported case non-fatal.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
We call systemctl to update triggers once an update finishes, which enqueues a
task and then waits to finish by default thus blocking the system() call. This
adds unnecessary time to the update, so set --no-block and move on since we
already reached the end of the update.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The backslash for the second-to-last entry in EXTRA_DIST was omitted,
leading to a dist issue.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This is required by distros which ship configuration files under
/etc. Those files then are considered read-only and must be updated as
part of OS updates.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Extended the commit message and fixed the case where --enable-stateless
is explicitly given.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The comment from commit fc0f570 was a bit misleading. Adding more
information about which combinations are valid and tested instead.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Commit fc0f570d added a check that prevents using --enable-xattr
together with --enable-bsdtar, perhaps because it was assumed that this
wouldn't work because there is no special tar option as in the GNU tar
case.
But bsdtar worked fined in combination with IMA and Smack xattrs in
Ostro OS. Xattrs are set accordingly there on the server side, so the
check needs to be removed to allow the combination.
Besides that, enabling xattrs also had no effect because xattrs.c
never got to see the SWUPD_WITH_XATTRS define due to not including
config.h.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Add a large amount of manual pages to swupd-client project. These are
meant to cover a large amount of topics and should cover the needed
material for people who are either using, or mixing with swupd,
and so this goes a bit further than end user documentation.
I've put the unit stuff in section 4. Any other section seemed
less related.
We disable check-update.timer by default. Having a message in the
journal isn't helpful, and there currently is no API to properly
query for update (except for running swupd check-update).
We add auto-update timer units. These can be masked (`systemctl
mask swupd-update.service`) to disable auto-updates, or enabled
if the OS ships with them not enabled. The project ships with
them not enabled for now, may end up enabling them later in time.
When doing out-of-tree builds, the distributed
scripts are in different directory than where
the build is happening. This doesn't work:
make install-exec-hook
make[2]: Entering directory '<builddir>'
perl scripts/findstatic.pl */*.o | grep -v Checking ||:
Can't open perl script "scripts/findstatic.pl": No such file or directory
make[2]: Leaving directory '<builddir>'
We therefore need to give perl the full path
of the perl-script to run using the standard
automake variable top_srcdir:
make install-exec-hook
make[2]: Entering directory '<builddir>'
perl ../../../../../../../../tmp/swupd-client.git/scripts/findstatic.pl */*.o | grep -v Checking ||:
'bin_paths' is unique to src/search.o, should be static? (initialised variable)
'do_search' is unique to src/search.o, should be static? (function)
'download_manifests' is unique to src/search.o, should be static? (function)
'file_search' is unique to src/search.o, should be static? (function)
'lib_paths' is unique to src/search.o, should be static? (initialised variable)
'report_find' is unique to src/search.o, should be static? (function)
'scope' is unique to src/search.o, should be static? (initialised variable)
'search_type' is unique to src/search.o, should be static? (initialised variable)
make[2]: Leaving directory '<builddir>'
Signed-off-by: André Draszik <git@andred.net>
This release removes certificate pinning to enable the client to communicate with a CDN for update operations, minimizing syscalls to fdopen/fclose during file downloads and using the default curl write handler to improve performance, and returning proper return codes when calling check-update.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release removes certificate pinning to enable the client to communicate with a CDN for update operations, minimizing syscalls to fdopen/fclose during file downloads and using the default curl write handler to improve performance, and returning proper return codes when calling check-update.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
make the return code for swupd-client be 1 if there is no update
available. This follows in the tradition of grep, which returns 0 if
there are matches, 1 if there are not, and 2 for errors.
Do the same for swupd-client update --status
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Now that the number of pending downloads is kept below a certain limit
(see poll_fewer_than()) it is possible to open files before starting
the transfer. Using the default curl write handler and explicit
open/close of the file makes the code simpler.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The previous approach was to open/fdopen/fclose the file for each
chunk that gets passed from curl. This incurrs a huge performance hit
when close() triggers a hashing of the file content on systems where
integrity protection via IMA is enabled.
Now the file is opened only once and kept open until the download is
complete. In addition, the unnecessary usage of C file IO is avoided.
The semantic is changed as little as possible:
- file gets created only after the first chunk of data arrived
- file descriptors do not leak to child processes (O_CLOEXEC)
- data gets appended to existing files (via O_APPEND, used
to keep the code simple and avoid an additional lseek)
- data gets flushed explicitly for each chunk (via fdatasync(),
which somewhat approximates the effect that an explicit
close() may have had)
As an additional improvement, failures during close() are checked. To
keep error handling as much as before, the completion function which has
the close() takes the current curl error code and replaces it if it
encounters a write error.
[v2 of the patch with fixes by Dmitry Rozhkov, see https://github.com/pohly/swupd-client/pull/1]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
To enable swupd-client to communicate with a CDN for updates, the custom
cert pinning must be removed.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release changes swupd-client to ignore xattrs by default, fixing errors on systems where xattrs were enabled thus changing the hash of files on system and mismatching hashes in update manifests; printing diagnostic info on network-connectivity problems to easier debug update failures, and fixing some build warnings from autoreconf.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When running 'autoreconf', aclocal warns that the "m4" directory does
not exist, even though AC_CONFIG_MACRO_DIR in configure.ac declares that
it should exist.
To avoid the warning, track the directory in the git tree.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Certain types of network-connectivity problems hit this curl error, so
print a diagnostic message when it occurs.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Swupd used to use a sorted blob of extended file attributes as part of
the data that was used to verify the contents. This caused problems if
extra attributes were added. In particular if a clearlinux system was
being run in a container that had selinux enabled in the base OS.
Add a configure option to allow the attributes to be considered or
not. Based on advice, the default is to ignore them, as this is what
everyone expects.
Tidied up passing a --selinux flag to tar, this is also handled by a new
config option.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Swupd used to use a sorted blob of extended file attributes as part of
the data that was used to verify the contents. This caused problems if
extra attributes were added. In particular if a clearlinux system was
being run in a container that had selinux enabled in the base OS.
Add a configure option to allow the attributes to be considered or
not. Based on advice, the default is to ignore them, as this is what
everyone expects.
Tidied up passing a --selinux flag to tar, this is also handled by a new
config option.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
This release includes major updates to various components of swupd-client:
A rework of the bundle subscription code, making the consolidation steps
simpler, reporting bundle changes for updates instead of "manifest" changes,
accounting fixes for the update stats, and reworking the update list creation
logic to massively improve performance by not calling verify_file() when
not needed.
The interface for functional tests was changed to better accommodate updating
test cases when output changes (because of code changes/fixes), such that
each test case has a single expected output that is tracked as a whole,
rather than individual output lines.
Many performance enchancements to drastically reduce client update time:
- Cutting down the amount of recursion when adding subscribed bundles.
- Caching the latest version number.
- Caching manifest hash checks if they succeed.
- Checking only the manifest headers on paths where the whole manifest does not
need to be parsed.
- Enabling multiplexing in curl.
- Setting nosync on staging dir removal operations.
- Fixing statistics and time reporting for updates.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
We can't sort the update list inside a helper function
that only gets a pointer to the list; the list head can get
updated as part of the sort, and this update to the list
head doesn't get reflected in the caller, leading
to a corrupted linked list.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Adding CURLPIPE_MULTIPLEX | CURLPIPE_HTTP1 is now possible with libcurl >=
7.43.0. The value 1 tells it to first check if multiplexing is supported on
a connection before attempting any transfers.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
We spend a lot of wasted time parsing the entire manifest in some areas, when
only the header is needed. This patch adds a header_only flag that tells the
appropriate load manifest functions to quit early after the header is read,
instead of parsing the entire (possibly very large) manifest. Another
optimization is calling fopen() with the 'm' flag, which tries to use mmap to
access the file (for reading only).
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch cuts down the amount of recursion that happens by skipping a
subsequent rescursive calls to add_subscriptions if we hit a bundle that is
already subscribed.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Tudor Marcu