Instead of initializing curl library on start, initialize it only when needed.
The advantage of this approach is that we can run some commands offline, if
there's no file to download.
Fixes#801Fixes#895Fixes#277
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
In the case that we have a bundle installed as also-add and removed later,
repair shouldn't reinstall that bundle. The same is valid for diagnose and update.
Os-install is the only exception. We should always install all bundles that are
listed as also-add in os-install.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Swupd will install optional bundles on bundle-add by default,
but will skip them if specified by the user by using the
--skip-optional / -o flag..
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
An optional bundle is not required to be installed in the system while
included bundles are. Swupd will install optional bundles on bundle-add
by default.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Currently users can set content and version urls based on http or https
protocols. This pose a security risk if users decide to use http.
This commit blocks swupd from working with http unless is specifically
allowed by using the --allow-insecure-http flag.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit adds a few more cosmetic changes to the commands that run
verify in the back for consistency.
- Different steps in the update process are separated by a blank line.
- Messages from swupd should not finish with a '.'
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When swupd is installing a bundle, it goes through the list of files to
install and removes those files that already exist in the target system
from the list, regardless of if the files were added by other bundle or
not.
This commit changes this process so the file is removed from the
list of files to install only if the file is listed as installed in the
manifest of one of the already installed bundles. This way if the file
is already present in the system, but not listed in any other manifest
it can be overwritten since we cannot assume the file is correct.
Closes#863
When staging a file using do_staging(), if the path of the file is
missing verify_fix_path() is called to try fix the missing path.
verify_fix_path() then removes bad directories (if any), downloads the
file and again and stages the new directory using do_staging() again, doing
a circular reference. On top of this bundle add is calling
verify_fix_path() after running do_staging(). All this circular
reference makes the code very difficult to understand and may cause
swupd to do extra work that is not necessary. This commit simplifies the
code by not allowing do_staging() to call verify_fix_path() so we can
manually call it later if necessary.
This commit also add some more comments in the code to make it easier to
follow.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Adding messages where swupd could take some time to finish a step so
users know better where the process is at.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
The json tests had been disabled because they were unstable. This issue
was fixed in a previous commit, so these tests should be enabled again.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit makes use of the swupd_progress_callback() function to
report progress downloading fullfiles based on how much data we have
downloaded vs the number of files downloaded. This callback will only be
used when the number of files to be downloaded are less than MAX_FILES,
calculating the total download size can be very costly if the files are
too many. If the files to be downloaded are more than MAX_FILES we will
fallback to reporting download progress based on file count as before.
When installing bundles or doing updates, swupd creates a list of files
that need to be downloaded. This list may contain files that were
already downloaded via packages and it often does. These files are then
skipped at the moment of downloading them since they are already in the
system. This causes a misleading output that shows the user that
fullfiles will be downloaded when they are actually not.
This commit filters the list of fullfiles to be downloaded to only
contain those ones that actually need to be downloaded.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit makes use of the swupd_progress_callback() function to
report progress downloading packs based on how much data we have
downloaded vs the number of files downloaded.
This commit also fixes a bug in the download_subscribed_packs function.
Swupd was not downloading the correct pack for bundles not installed in
the system that had been recently added as dependency of another
installed bundle.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Disabling json tests that checks for progress because tests are unstable.
If we have any minor changes in the code that could affect how curl is called
we could have different progress reports and because of that we would have false
negatives.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The --json-output flag can be enabled for every swupd command.
This commit converts it to be a global flag, and enables it in every
swupd command.
Closes#869
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
swupd prints some messages to stdout and others to stderr, this makes
the code somewhat confusing and also it creates a mess regarding which
messages go where. Messages are usually printed using fprintf() and
printf().
This commit replaces all calls to printf and fprintf with one of the
functions that are part of the logger. This will bring many advantages
to the code:
- readability, it is straightforward to identify where a message needs
to go based on the name of the logger functions.
- consistency, the output will go to either stderr or stdout depending
on the type of message being printed.
- the format of the messages will also be consistent, so for example,
if the message is an error the message will automatically include the
"Error: " label in the beginning of the message, this way we avoid
having things like "ERROR:", "error - ", etc...
Closes#864Closes#294Closes#489Closes#433
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Since the reporting of a process progress could be printed in different
formats in the future, not just JSON, this patch moves those functions
to their own file (progress.c) so it is decoupled from the json
formatter.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When swupd is run and the stdout is not a tty (for example when
redirecting the output to a file) the progress of a step is being
reported as a sequence of dots, which is not very useful to really
identify how far in the process we are.
This commit changes that behavior so percentages are used to report
progress when not using a tty instead of dots.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When using bundle-add to install bundles with the --json-output flag,
the output generated by swupd will be formatted to json. This is useful
for other applications that are reading the swupd output to determine
real time progress.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Instead of printing errors and warnings, just print debug information
when we are looking for the right ca path to use. If we couldn't
connect to the server, abort and print a clear error message.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When swupd needs to run a boot script it does so without checking if the
file exists in the system or not. This will cause an error to be raised
by the shell if the script does not exist.
This commit adds a check for the existance of the script before
attempting to run it.
Closes#795
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Some functions currently mask error codes by using a very general error
like SWUPD_COULDNT_INSTALL_BUNDLE (formerly EBUNDLE_INSTALL) in the case of
the bundle-add command.
This commit modifies the codes returned by the bundle-add in case of
failures so they are more meaningful for the actual issue.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Renaming codes following these rules:
- All codes should start with SWUPD_ so we never get confused what
code comes from swupd
- Always separate codes that contain multiple words using an underscore
SWUPD_MULTI_WORD_ERROR instead of using SWUPDMULTIWORDERROR
- Code 0 is reserved for SWUPD_SUCCESS (or SWUPD_OK)
- Code 1 is reserved to be used as "no" by commands that return a boolean
state (e.g. swupd autoupdate, check-update)
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This will make outputs a lot better when using --quiet because optional
curl messages wont be printed.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
If the user handles the error we won't reschedule the download, but we
should at least to consider that download a failure and return a negative
code on swupd_curl_parallel_download_end().
Tests that are suppose to get that were incorrect. Fixing that too
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The biggest change is handling timeout in a better way. The response
code in a timeout (and some other curl errors) is the response from
the last successful operation, so on timeouts we can get 200 (OK)
response code instead of 206(partial content) in a partial download. So
we should look at curl return and not on http return code to decide if we
are going to try to resume the download or not.
Changing slow-server test because error message has changed
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When adding a bundle, if downloading the files fail because of disk
space, do not re-attempt to download the files since it will fail again
for sure.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Sometimes is useful to be able to limit the size of a test environment
to validate some scenarios that are bound to disk size.
This commit adds a -s (size) option that can be used when creating test
environments so the user can control how much space is available in that
environment. If not used, the test environment works the same as before.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When we have SSL connections errors we try to use fallback CA paths added by the
--with-fallback-ca-paths configuration flag.
Instead of printing errors in this case, just print a warning. An error
message will be printed when we fail when trying all fallbacks.
Fixes: #746
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The tests that attempt to verify the client certificate were using the
swupd -u option to set the url of the "upstream server". Recently a
function was added to the test library to set the default upstream
server.
This commit modifies these tests so they used the new function to set
the upstream server instead of using the -u option. This will provide
the benefit of the test environment being more similar to a prod
environment, also makes the test setup less convoluted and the tests
easier to read.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Add a new alias filetype that swupd understands. This file's format is
the following:
A potentially multiline file with an alias name per line followed by,
ont he same line, a list of bundle names with all fields being tab
separated.
If the same alias is defined in the same file, the first definition
will be used. Malformed lines will not be parsed but it is valid to
have multiple concurrent tabs in a line.
The files are to be read from both user and system alias directories
with stateless overrides of system files with user files. This implies
you can mask a system file with a user file of the same name if the
user file is a symlink to /dev/null. You can also override a system
file with a user file of the same name but with different content (the
contents of the system file will be unused). Alias files are
considered in lexicographical order sorted the same as bundle-list. If
two files provide the same alias than the one in the file that comes
first lexicographically will be used. User files will always come
before system files, such that if a user file 'u' defines alias 'a'
and a system file 's' also defines alias 'a' the user file definition
will be used instead of the system file definition.
The intended usage of this will be for OS content to provide alias
files that will then enable users to add content using the provided
aliases as well as enabling users to define their own alias for custom
package groups.
Eventually this feature may be considered for extension to other areas
aside from bundle-add but for now only bundle-add usage is to be
supported.
This commit adds the ability of warning the user if he/she tries
to install an experimental bundle.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Tests will be cleaned up by a trap routine when interrupted early. This
change also moves test web server and certificate store clean up to the
destroy_test_environment function.
Signed-off-by: John Akre <john.w.akre@intel.com>
Some tests wrote files and directories to /tmp. This change moves the
test certificates directory into the swupd repo's top level directory
and moves other test files into their corresponding test directories.
Fixes#650
Signed-off-by: John Akre <john.w.akre@intel.com>
Every test should include a unique ID, so it is necessary to have
a way of knowing what is the next available ID for the group of
tests.
This commit helps with that issue by implementing the following
features in the test library:
- Adds the function get_test_list which can be used to print a
list of tests in a specific group directory.
- Adds the function get_next_available_id which return the next
test ID that can be used in a new test.
- When using the generate_test function to generate an empty test
the next available ID will be used for the template.
This commit also adds the ID to a bundle-add test that was missing it.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Shellcheck is a shell script static analysis tool that looks for errors and
programming pitfalls in shell scripts.
Fixing errors reported by running that tool on bats scripts
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
As part of the effort of having the functional tests better
organized and have traceability we are adding unique IDs to
every test.
This commit adds the IDs to the bundle-add tests. It also
changes the description of some of them to make them clearer,
and removes duplicates.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Previously, the only way to get a return code of 0 (success) during
a bundle-add operation was if everything that was attempted to be
added had been successful. That means that bundle-add command was
returning EBUNDLE_INSTALL whenever it would try to add a bundle that
was already installed.
This commit changes the return code of such operation since it was
determined that it is a more appropriate behavior to consider an
attempt to add a bundle that is already added as successful, even when
no bundle was added during the operation.
Closes#510
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Otavio Pontes