The current format is very useful information to get from bug reports
so add it as default output to swupd update.
Signed-off-by: William Douglas <william.douglas@intel.com>
When a bundle is removed, ensure the new orphans created are detected
by bundle-list and bundle-remove.
Signed-off-by: William Douglas <william.douglas@intel.com>
Currently sys_rm_recursive was used in any instance of deleting swupd
content from the system (update, repair and bundle-remove). This can
cause user data loss when unkown files are in directories that swupd
is deleting.
To prevent this, this patch changes how deleting content in swupd
operates. Swupd content removal is now done with sys_rm and the return
value is checked in case the removal failed due to a directory that
still had files in it. When this specific failure occurs, the
directory is added to a new list for reprocessing removals as it is
expected once the rest of the deletes on the system occur the failures
will go away as the directories will be empty (these deletes are
processed in alphabetical reverse order so leaf directories are
processed first). If the removal fails again it is presumed the
contents of the directory are not files swupd knows about and as such
should be kept somewhere else.
For handling the retention of user data, directories (with only the
content unknown to swupd) are renamed (currently using a
.deleted.$timestamp. prefix of the old name) and stored at the same
directory level they were previously found with one exception. The
exception is for nested deleted content best illustrated with an
example:
/swupd-dir1/user-file1
/swupd-dir1/swupd-dir2/user-file2
When swupd tries to remove the /swupd-dir1 content, it will store the
user files as follows:
/.deleted.$timestamp1.swupd-dir1/user-file1
/.deleted.$timestamp1.swupd-dir1/.deleted.$timestamp1.swupd-dir2/user-file2
To demarcate what was part of swupd content vs user content.
Signed-off-by: William Douglas <william.douglas@intel.com>
Instead of relying on a custom global_setup and global_teardown
functions, migrate to the use of bats' setup_file and teardown_file
functions.
Signed-off-by: William Douglas <william.douglas@intel.com>
Environment variables are used everywhere in testilb. This environment
variables are global variables that define the way testlib behaves.
However is was confusing to use the variables because they were
inconsistent between each other, for example some variables that define
paths would have absolute paths while other would have relative paths,
making it error prone while using them.
This commit makes the environment variables more consistent by following
a name convention for each type of variable, as an example, variables
that define absolute paths follow this convention ABS_<path_name>_DIR,
while variables that define relative paths are defined like this
<path_name>_DIR.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Currently there is only one location, known as "statedir", where swupd
stores data and cache. Normally it make sense to keep the data in a
path that is relative to the path prefix of the target system, since the
data is specific to that system, but cache can be independent of the
system, and that way it can be reused for other target systems if
desired.
This commit splits the cache and data into two different locations.
It also stores the cache in a path that is dependent of the mirror url
being used by swupd. The commit also makes all references to files in
the cache or data directories through getter functions thus removing the
hardcoded reference to the file's location, making it easier to move the
location in the future if needed. Lastly, this commit changes the read
permissions for the data directory and the manifest directories in the
cache to be user readable.
When swupd downloads manifests to the statedir they are downloaded in
directories that represent the version of clear that the manifests
belong to. However those version directories are stored in the statedir,
root directory. This commit moves them into their own "manifest"
directory so there is more consistency with the downloaded files.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When swupd downloads manifests to the statedir they are downloaded in
directories that represent the version of clear that the manifests
belong to. However those version directories are stored in the statedir,
root directory. This commit moves them into their own "manifest"
directory so there is more consistency with the downloaded files.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit implements the --orphan flag for bundle-remove which can be
used to remove bundles that are no longer required by tracked bundles.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Current implementation was only removing test enviroment for envs created
on test setup. And test_teardown function needed to be overriden in case
global_setup() was used.
Now always cleaning the enviroment for both use cases, test_setup() and
global_setup(). The user doesn't have the control anymore on that and
if the user wants to keep the env KEEP_ENV variable should be used.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Test environment can be created on global_setup() or test_setup() and that
change the behavior of several operations on testlib.
Because of that, let the user create the test environment as desired instead
of forcing it on test_setup()
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
2 groups were created:
- slow: Tests that are very slow because of sleeps, so we can run them all
in parallel
- system: Tests that make change to the system, so they can cause problems
when running in parallel, so we run them in series.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This patch adds 2 scripts used to balance test execution. The weight_tests.bash
runs all tests and sets a weight to them based on how long they take to run. The
other, filter_bats_list.bash, use this information to split the tests in groups to
be executed by github actions.
When a new test is added the script will consider it with an average weight, so this
shouldn't unbalance the system right away. After some time, if we notice that the
system is not balanced anymore we can just run the weight_tests.bash again to rebalance.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When user uses the --force or --recursive flag swupd prints a message
explaining what effects it will have on the system. This commit changes
this message from informational to warning to attract more attention.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
If the -R / --recursive flag is used, remove the specified bundles along
with their "deletable" dependencies.
In this context deletable means:
- the dependency is not os-core
- the dependency is not required by other installed bundle that is not
to be removed
- the dependency is not tracked (specifically installed by the user)
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Functional tests for the new "--recursive" flag in bundle-remove that
will be used to remove a bundle and its dependencies.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Changing the progress report lib to always print 0 or -1 when a step is
started and a 100 when a step finishes.
Also reorganizing all commands to have less steps (when possible) and that
are more meaningful to progress report. Hide some internal steps that doesn't
provide useful information to end users.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This commit does a major refactor of the bundle-remove command to
include these improvements:
- Adds a "--force" flag that can be used to remove a bundle that is
required by other installed bundles along with all the bundles that
depend on it
- When attempting to remove a bundle which is required by other
installed bundles, swupd used to show the list of dependencies in a tree
view, this list was very large in some cases. This commit changes the
output of such cases from a tree view to a simple deduplicated list of
bundles that require the bundle to be removed. The user can still see
the old tree view by including the --verbose when running the command
- Stops notifying telemetry of user errors, like user trying to remove
an invalid bundle, or a bundle that was not installed, etc
- Consider all bundles provided in the bundle-remove command when
checking for dependencies (required by)
- Instead of removing each bundle in the command one by one, consolidate
all bundles to be removed, validate them and remove them all at once.
This will optimize the tasks to be executed for removing bundles.
One change included in this commit for bundle-remove is propagated to
bundle-list:
- When using "bundle-list -D BUNDLE" to find what bundles depend on
BUNDLE. The list will be presented in a simplified deduplicated list of
dependencies by default instead of the tree view that used to be
default. The user can still see the old tree view by running the command
appending the --verbose flag
Closes#891Closes#732Closes#674
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
There was a bug in the logic of the setup function in testlib which was
causing the globale_teardown to be incorrectly called when tests were
being run using "bats <directory>/". This commit fixs the issue.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Instead of initializing curl library on start, initialize it only when needed.
The advantage of this approach is that we can run some commands offline, if
there's no file to download.
Fixes#801Fixes#895Fixes#277
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Swupd will install optional bundles on bundle-add by default,
but will skip them if specified by the user by using the
--skip-optional / -o flag..
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
The --json-output flag can be enabled for every swupd command.
This commit converts it to be a global flag, and enables it in every
swupd command.
Closes#869
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
swupd prints some messages to stdout and others to stderr, this makes
the code somewhat confusing and also it creates a mess regarding which
messages go where. Messages are usually printed using fprintf() and
printf().
This commit replaces all calls to printf and fprintf with one of the
functions that are part of the logger. This will bring many advantages
to the code:
- readability, it is straightforward to identify where a message needs
to go based on the name of the logger functions.
- consistency, the output will go to either stderr or stdout depending
on the type of message being printed.
- the format of the messages will also be consistent, so for example,
if the message is an error the message will automatically include the
"Error: " label in the beginning of the message, this way we avoid
having things like "ERROR:", "error - ", etc...
Closes#864Closes#294Closes#489Closes#433
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Instead of printing errors and warnings, just print debug information
when we are looking for the right ca path to use. If we couldn't
connect to the server, abort and print a clear error message.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Some functions currently mask error codes by using a very general error
like SWUPD_COULDNT_REMOVE_BUNDLE (formerly EBUNDLE_REMOVE) in the case of
the bundle-remove command.
This commit modifies the codes returned by the bundle-remove in case of
failures so they are more meaningful for the actual issue.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Renaming codes following these rules:
- All codes should start with SWUPD_ so we never get confused what
code comes from swupd
- Always separate codes that contain multiple words using an underscore
SWUPD_MULTI_WORD_ERROR instead of using SWUPDMULTIWORDERROR
- Code 0 is reserved for SWUPD_SUCCESS (or SWUPD_OK)
- Code 1 is reserved to be used as "no" by commands that return a boolean
state (e.g. swupd autoupdate, check-update)
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This will make outputs a lot better when using --quiet because optional
curl messages wont be printed.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The biggest change is handling timeout in a better way. The response
code in a timeout (and some other curl errors) is the response from
the last successful operation, so on timeouts we can get 200 (OK)
response code instead of 206(partial content) in a partial download. So
we should look at curl return and not on http return code to decide if we
are going to try to resume the download or not.
Changing slow-server test because error message has changed
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When removing a bundle, if downloading the files fail because of disk
space, swupd should not re-attempt to download the files since it will
fail again for sure.
This commit adds tests that verify this behavior.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Sometimes is useful to be able to limit the size of a test environment
to validate some scenarios that are bound to disk size.
This commit adds a -s (size) option that can be used when creating test
environments so the user can control how much space is available in that
environment. If not used, the test environment works the same as before.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When we have SSL connections errors we try to use fallback CA paths added by the
--with-fallback-ca-paths configuration flag.
Instead of printing errors in this case, just print a warning. An error
message will be printed when we fail when trying all fallbacks.
Fixes: #746
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The tests that attempt to verify the client certificate were using the
swupd -u option to set the url of the "upstream server". Recently a
function was added to the test library to set the default upstream
server.
This commit modifies these tests so they used the new function to set
the upstream server instead of using the -u option. This will provide
the benefit of the test environment being more similar to a prod
environment, also makes the test setup less convoluted and the tests
easier to read.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When running bundle-remove to remove a bundle the number in "Total
deleted files" is often wrong. This is happening because when deleting
the files from a bundle sometimes we start by deleting a directory
(with all the files within it), so all the files that got deleted are not
accounted for, only one file will be counted as deleted (the directory),
this will yield to an incorrect count of deleted files.
This commit fixes this issue by printing the total number of
files that should be deleted, minus those that failed to be deleted,
instead of counting those that succeeded.
Closes#690
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
There was a lot of copy and paste of parameters and parameters help, making it
hard to maintain global parameters sanity, like sanitizing values, keep help
updated and making sure all commands had the appropriate options.
Move all global parameters parsing to global.c and leave only specific parameters
to commands files.
Fixes: #571
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This commit adds the test ID to every bundle-remove test, removes
duplicates and make readability improvements.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Tests will be cleaned up by a trap routine when interrupted early. This
change also moves test web server and certificate store clean up to the
destroy_test_environment function.
Signed-off-by: John Akre <john.w.akre@intel.com>
Some tests wrote files and directories to /tmp. This change moves the
test certificates directory into the swupd repo's top level directory
and moves other test files into their corresponding test directories.
Fixes#650
Signed-off-by: John Akre <john.w.akre@intel.com>
The old bundleremove tests used the swupdlib.bash library that was
replaced by testlib.bash.
This commit replaces those old bundleremove tests with new versions
of the same tests that now use testlib.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Every bundle includes a tracking file which tells the OS the
bundle is installed, every bundle has this tracking file as
part of its manifest. When removing a bundle from a system,
all the files that are part of that bundle, and are not needed,
for any other bundle are removed, including the tracking file.
This patch removes a function call that is intended to remove
the tracking bundle from the system, since it is not necessary,
by the time the code calls that function, all files from the
manifest that can be removed are already removed, and this includes
the tracking file.
This patch also modifies a few tests to reflect the change in the
command output.
The operations have different command result output styles. Also,
in particular bundle-remove error is a bit redundant.
This commit changes the messages displayed by the bundle-add and
bundle-remove functions so they have consistent messages for their
corresponding operations.
Fixes#475
William Douglas