This release adds enhancements, bug fixes and test improvements.
Enhancements:
- List experimental bundles on bundle-* and search functions
- Moving generic parameters to a global parameters parsing so they can be
consistent all over swupd.
- Let users to define bundle aliases
Bug Fixes:
- Better overflow handling on integer cmdline arguments
- Use execve instead of system to run system binaries and avoid globing
- Fix the number of deleted files in bundle-remove
- Update when using a mirror was failing if upstream was down
- Improvements in log framework and better error and warning reporting
- Adding more debug information when downloading
- Replace strftime to asctime to avoid segfaults
- When verifying a system, don't list skiped files as errors
- Don't force download retries if the disk is full
- Fix several parallel download errors
Tests
- Several bug fixes in testlib and improvements in styles
- New tests added
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The code was assuming that dup2 was replacing the fd, but this is not the case.
It is duplicating it, so new_fd was leaking.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Shellcheck provides many useful recommendations for writing shell code.
This commit makes testlib.bash shellcheck compliant.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When download errors occurs and we need to try a download again, reduce the
number of parallel downloads by 1/3 to minimize problems on next execution.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The biggest change is handling timeout in a better way. The response
code in a timeout (and some other curl errors) is the response from
the last successful operation, so on timeouts we can get 200 (OK)
response code instead of 206(partial content) in a partial download. So
we should look at curl return and not on http return code to decide if we
are going to try to resume the download or not.
Changing slow-server test because error message has changed
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Group in a function the checks for http response code and curl return,
so we can use it for parallel downloads.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Libcurl can be blocked to the point of timeout during long
extractions in callbacks. To fix this, we instead schedule
the extraction to be done in a thread pool.
This unblocks curl, allowing downloads and extraction simultaniously.
This also results in a speedup during the download packs step.
When using the thread pool, the callback must be wrapped because
functions run in our pool must return void. Instead, we pass a
pointer to a struct and set the return value in that struct.
Libarchive's archive_write_disk API is NOT thread-safe, so for now
the threadpool only handles one thread.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Signed-off-by: Brian J Lovin <brian.j.lovin@intel.com>
Simple implementation of a thread pool to be used initially for parallel
downloads. Only major features needed were implemented. A tp_wait() function is to be
contributed later.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Signed-off-by: Brian J Lovin <brian.j.lovin@intel.com>
Two PRs that included tests for swupd update were submitted and merged
at the same time, which caused the test IDs to be inconsistent.
This commit fixes the IDs of the update tests with duplicated ID.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When in a test we create a new version and a bundle update in that new
version, we copy the latest tar for each fullfile from the previous
versions into the new version. The code currently was searching for
those tars starting with the current directory, this was working fine
when running the tests locally because tests are run serially, but this
was causing an unexpected result when running in Travis since tests are
run in parallel, so this was causing the search to sometimes find the
tar in a different test environment (from another test), which was
causing unexpected results and a race condition.
This commit fixes the issue by narrowing the search of the tars to the
web-dir of the specific test environment.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
There was a problem in the curl configuration that was causing curl to
leak the downloaded content in some scenarios, because of that we were
doing only partial checks on the search command output. Now that the
leak has been fixed we can now check the whole output by using
assert_is_output instead of assert_in_output.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When searching content, if downloading the files fail because of disk
space, do not re-attempt to download the files since it will fail again
for sure.
This commit implements that logic for the search command.
It also removes a few lines of code that were not being used and fixes
some incorrect messages.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When searching content, if downloading the files fail because of disk
space, swupd should not re-attempt to download the files since it will
fail again for sure.
This commit adds tests that verify this behavior.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When listing bundle, if downloading the files fail because of disk
space, swupd should not re-attempt to download the files since it will
fail again for sure.
This commit adds tests that verify this behavior.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When removing a bundle, if downloading the files fail because of disk
space, swupd should not re-attempt to download the files since it will
fail again for sure.
This commit adds tests that verify this behavior.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When verifying a system, if downloading the files fail because of disk
space, swupd should not re-attempt to download the files since it will
fail again for sure.
This commit adds tests that verify this behavior.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When adding a bundle, if downloading the files fail because of disk
space, do not re-attempt to download the files since it will fail again
for sure.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When a disk gets full on a file download during an update, we are
currently scheduling up to five retries for that download, which are
going to fail for sure since there is no more space on disk.
This commit changes this behavior so it only tries to download the file
once if we receive an error about the disk being full so we don't waste
time.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit adds a few tests that validate the behavior of swupd when it
tries to update a system that has a full disk in different stages of the
update.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Sometimes is useful to be able to limit the size of a test environment
to validate some scenarios that are bound to disk size.
This commit adds a -s (size) option that can be used when creating test
environments so the user can control how much space is available in that
environment. If not used, the test environment works the same as before.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When running verify --fix, if there are files in the system that are
marked as deleted in the manifest, then they are removed from the
system. However, if a component of the path of the file is a
symbolic link, the file is determined to be unsafe to be removed and
it is left in the system, the user is warned, and the not-deleted
file count is incremented.
This commit modifies this behavior so verify just skips the file, the
user is not warned with the "Not safe to delete: <FILE>" error message,
and so the not-deleted file count is not incremented since it is really
not useful for the end user.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
The test library provided capabilities to create bundles that contained
symlinks to files in it, but it was not possible to create bundles with
symlinks that pointed to directories.
This commit adds the capability to create a test bundle that contains a
symlink to a directory by using the option "-c" when creating the
bundle.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When creating a test bundle that contains a symlink like this:
create_bundle -L -n my_bundle -f /my_file -l /my_link my_test_environment
A test bundle called my_bundle is created in my_test_environment,
this bundle has one link called my_link and this link points to a
file that is also added to the bundle's manifest. The problem is
that during the creation of the bundle the symbolic link is created
to point to an existing file but later in the process the file is
renamed, causing the symlink to become dangling. This dangling link
is then added to the bundle's manifest. If we run "swupd verify
--fix" the installed bundle will always show a hash mismatch and the
good symlink will be replaced with the dangling symlink.
This commit changes the way symlinks are being created when creating a
bundle so they point to the correct file.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Glob ** will only recurse directories if globstar is set. To avoid skipping
important files run clang-format on src/ and src/lib
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
gcc-5, used by travis, is complaining about some problems that newer compilers
ignore. As we don't need pedantic, I'm taking this out. Fixes were left because
they make sense.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
All warning-related flags are on their own lines, because they look very
messy if they're all shoved together. Enabled many more warnings and
made warnings fatal.
Signed-off-by: Brian J Lovin <brian.j.lovin@intel.com>
When CURLOPT_NOBODY is set to true, curl shouldn't get the body of a file,
we should get only headers. This works fine for all tested scenarios using
http and https. It's also working in most scenarios that uses file://, but
it fails on travis environment. So, add a dummy function to ignore body data
if present.
There was a hack in search tests to ignore leaked file content. Removing that
hack.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Network status varies a lot and some times it's hard to understand how
one specific network condition is affecting swupd downloads. This patch
adds more information to make it easier to debug that.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Create a log framework so we can control the log level of each log output.
Also adds experimental flags --quiet and --debug to control log level from
cmdline. --debug is working fine, because in this mode all messages are
printed, but to have a --quiet we need to port all fprintf/printf usages
to log_*.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
We were using an http server based on the SimpleHTTPRequestHandler to
run most of our server-related tests except for one which was using an
http server based on the BaseHTTPRequestHandler.
This commit merges the functionality of both server clasesses into one
so we use the same one for all future tests.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
As part of the swupd initialization, curl is initialized, which means it
is configured and the connection is tested. When configuring curl for
this test, no timeout was being configured, which was causing curl to
hang waiting for a response from the server if the server was unresponsive.
This commit configures the curl timeouts the same way they are being
configured when downloading files, so in case we get an unresponsive
server (or mirror) it doesn't get stuck forever.
Closes#669
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Use origin/master branch to check which bash and bats scripts have changed and
run shellcheck only on them. This intends to make travis check faster.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
strftime is safer than asctime, so prefer that. It's known that there are
implementations of asctime that doesn't handle errors very well.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This function returns a bool so we need to check for true, not zero.
As this was only used on hashmap functions that aren't used on swupd,
this isn't producing any bug.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Logical OR instead of a binary AND was used. Because of that all elements were
placed on the first position of the hashmap, making it a really long list
instead of a real hashmap.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When we have SSL connections errors we try to use fallback CA paths added by the
--with-fallback-ca-paths configuration flag.
Instead of printing errors in this case, just print a warning. An error
message will be printed when we fail when trying all fallbacks.
Fixes: #746
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Otavio Pontes