Normal test environments come with a minimal version of os-core
bundle by default (only one file and the tracking file). Users
also have the option of not having os-core at all with the -e
"empty" option.
This commit adds another option "-r" for creating a test environment
that includes a more complete version of the os-core bundle,
with a couple of useful files: os-release and format. This type
of bundle can be useful when working with some type of tests like
updates.
Use download.c parallel download api to download packs in parallel,
the same way we do for fullfiles.
Some changes were necessary in update/slow-server test. As we are using the
parallel download api to download packs the resume strategy is now the same
described in b07285. Because of that download resume is only enables when the
download is interrupted in the first try. So we force a download fail and a
download retry in server.py to check if swupd is trying to resume the download.
Because of this change the update/slow-server test is performing 2 retries
(first downloaded is interrupted to force a resume and the second is interrupted
because the slow-server don't support download resumes) and this adds a 30 seconds
delay to the test execution. This delay isn't going to slow down the test execution
considerably, but the ideal is in the future to support running tests with 0s
timeouts. For this reason update/slow-server is being skip when not running in
travis ci.
Some tests need to have not only one but two or more versions of
the server side content (web-dir).
This commit adds a function that can be used to create these new
versions.
It is very common to manipulate manifests when writing tests for
swupd, so in order to make it easier for users, and less error
prone, this commit introduces a function to manipulate data from
a manifest easily.
Having to manually add files to the pack's tar everytime we are
dealing with delta or zero packs is error prone since those packs
should have files in a specific path for them to be used correctly
by swupd client.
This commit introduces a function to add files to zero or delta
packs in a consistent way.
Tests that compare output with an expected output strip text in
ignore files first.
This commit adds a global ignore-list intended to be used as a
fallback for all tests.
When removing a dummy bundle using the test library, the library
tries to remove bundle files and directories from the file system,
however if the directory that is being removed is not empty, it
will cause the function to exit with a non zero code.
This commit fixes the bug by adding an option for ignoring this
scenario.
When creating a bundle using the testlib functions, if the user
wants the bundle to include files, those files are generated
automatically with random content.
This commit adds the capability of specifying an existing file to
be used for the bundle by using the
"-f <file in manifest>:<file to use>" notation.
When creating a test bundle, if the user didn't select any directory,
file, or link to be added to the bundle, the bundle would add
directory /usr/bin by default so the bundle would not be totally
empty, however since the bundle also always adds the tracking file
(/usr/share/clear/bundles/"$bundle_name") to the bundle, then it is
not necessary, and actually undesired to add /usr/bin by default.
This commit removes the addition of /usr/bin by default.
Test environments have all the basic elements to test swupd-client
and one of those elements is the dummy os-core bundle created and
installed in the target-dir by default. Sometimes, the person
writing a test may need to have a completely empty test environment,
meaning with no os-core bundle.
This commit adds the capability of creating a test environment with
no os-core bundle by default, by using the -e (empty) option. The
commit also modifies the os-core bundle that is being generated by
default from having one file in /ur/bin/core to being just /core,
this way the os-core bundle is smaller and faster to be created,
which is important since it is always added by default to every
test.
When creating bundles, users get to choose if they want to include
files, directories, links or dangling links in the bundle. Whatever
choice they make, all the directories that are part of the path of
the element should be created. This wasn't happening when creating
bundles with nested directories, for example /foo/bar/baz, in this
case only dir /foo/bar/baz was being added to the manifest of the
bundle but neither /foo nor /foo/bar were being added.
This commit fixes that issue. It also adds a safety feature where
it adds a '/' in front of dirs or file when user creates a bundle
in case they forget to add it themselves so the function works
properly even in that situation. For example with something like
this "-f /foo,bar/baz"
Adjusting the expected output of the test scenario. Also we no
longer need to validate it against a regex, so we can just compare
it against plain text now.
There is an issue with the get_hash_from_manifest function where
in some cases if there are two files that share the same path,
both would be picked up.
This commit fixes the issue by being more slective when choosing
the line.
This commit fixes a bug when updating the hashes of manifests in
MoM. When doing this, all bundles were being removed and re-added
to the MoM using the remove_from_manifest and add_to_manifest
functions, but this was having some undesired side effects.
This commit implements the update of hashes in MoM in a different
way.
When removing a file or directory from a manifest the filecount and
contentsize in the manifest were not being decreased. Also after
removing an item from the manifest, the hash of the manifest will
change, which means the person using this function in a test needs
to remember to update the hash of the manifest in the MoM or the
test will fail because the MoM will be discarded as corrupt.
This commit decreases the values of both fields so the manifest is
still accurate after removing elements from it.
Also after removing an element from the manifest, the MoM will have
its hashes automatically updated so to release the end user from
remembering doing this.
Finally this commit include an update for a test that use the
remove_from_manifest function since it doesn't need to update hashes
in the MoM as a separate step any more.
With the current test library a user can create a test bundle and
then add a dependency to that bundle by adding it to the bundle's
manifest. However this only worked if there was only one version
available in the web-server.
A second issue was that after modifying the manifest the hash
in the MoM would need to be updated.
This commit allow users to create a version N, create a bundle
for version N, then create a version N+M and add a dependency to
that bundle in that new version N+M.
It also recalculate all hashes in the MoM so users writing tests
don't need to remember to update them themselves.
Finally this commit includes some updates for some tests that use
the add_dependency_to_manifest function since they don't need to
update hashes in the MoM as a separate step any more.
This commit changes the way the contentsize was being read from the
manifest, making it a little more efficient and easier to read by
replacing a piped call of cat + grep + awk, to only using awk.
It also includes a couple of changes in the comments of the
function so they are more accurate.
The set_env_variables function exports different variables that
are dependent of a specific test environment. This commit adds a
few more variables that can be used to reduce the typing when writing
tests.
Using `git diff-index` is problematic for this test, because manpages
regenerated by `make` will have a modified mtime, and `git diff-index`
(being a low-level command) does not refresh the file's mtime stored in
the index. Refreshing the mtime and other stat information stored in the
index is something that happens automatically for high-level "porcelain"
commands.
Since `git diff` will suffice for checking that no uncommitted changes
to manpages exist, use it instead.
Also, use the --exit-code option instead to see the diff output in case
of failure.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Automated tests should create its own test environment including
all needed dependencies when the test starts, and should clean up
after itself once the test is over. Sometimes when developing
these tests, they don't work as expected at first and is difficult
to figure out what the problem is if the test environment is gone.
This commit allow users to temporarily skip the removal of the test
environment so he/she can debug what the problem was by only running
the test with the DEBUG_TEST env variable set to "true".
Example: DEBUG_TEST=true bats my_test.bats
This commit adds the following two assertions:
- assert_files_equal
- assert_files_not_equal
These can be used in tests for validating file equality consistently.
Some variables in the test library were missing the $ character,
whichs makes bash interpret them as strings instead of variables.
This commit fixes the variable names.
The test library should be used to create test cases based on
modular "steps", the test library helps you build these steps,
sometimes is difficult to see what parameters a function from
testlib needs.
This commit adds usage info for every function in the test lib so
it is easier for the person writing test cases to know what parameters
the function needs, by simply executing the function name with no
parameters in the command line will show this info.
Most of the functions of the test library include some type of
parameter validation, when a function is called and one of its
parameters don't comply with the expected parameters it terminates
the whole script. This is expected, but sometimes it can be
difficult to figure out what was the function that received the
incorrect parameters in the script.
This commit adds a function that prints a stack of the function
calls so when a script is terminated it is easier to figure out
the culprit.
This test is unreliable and fails seemingly randomly in our CI.
Restarting our tests always makes the test pass.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The test library includes many assertions intended to validate
a command's output. These assertions take the command output and
compare it against an expected output. Some of these assertions
take a literal string and others take a string with regular
expression values. However, regardless of what king of string is
being used for the expected output, there are some common lines
in the output that are usually not important and are safe to ignore.
An example would be empty lines or lines with only dots.
This commit adds the capability for those assertions to use an
ignore file to include patterns that could be ignored/removed from
the command's output before comparing it against the expected
output. Also with this commit empty lines and lines with only dots
will be ignored by default. If the assertion needs to validate an
exact output without ignoring anything in the ignore-list nor ignoring
dots and blank lines, then the --identical option can be used with
the assertion.
This commit include two parts.
1) It adds 6 new assertions to the test library:
- assert_is_output
- assert_is_not_output
- assert_regex_is_output
- assert_regex_is_not_output
- assert_regex_in_output
- assert_regex_not_in_output
2) It migrates the tests from the following categories to use the
new library:
- completion
- hashdump
- mirror
- search
Finally this commit changes a few test cases so they use
"assert_is_output" instead of "assert_in_output", that way
we can have a tighter control of the output we want since
assert_is_output requires the whole output to match to a
specific text instead of doing a partial match.
This commit adds the following three funtions to the test library:
- install_bundle
- remove_bundle
- clean_state_dir
These functions are useful specially when the global_setup and the
global_teardown are used instead of test_setup and test_teardown.
These functions allow the user to create all of its test resources
in the global_setup and reuse those same resources throughout all
the tests in the test script. By using the global_setup and
global_teardown functions the test execution time can be greatelly
reduced.
This commit also modifies the add-rc.bats and remove-rc.bats to
leverage the global setup and teardowns to increase the execution
speed. These times were reduced as shown:
- bundleadd_v2/add-rc.bats from 1m50.129s -> 0m20.026s
- bundleremove_v2/remove-rc.bats from 0m59.079s -> 0m15.446s
A new test library.bash was introduced that simplifies the creation
of test environments and test objects.
This commit rewrites all existing tests from
test/functional/bundleremove and test/functional/checkupdate but this
time using the new test library.
NOTE: Once all the tests have been rewriten to use the new lib, they
can be included in the Makefile so they are run with every patch,
and the old tests can be deleted.
A new test library.bash was introduced that simplifies the creation
of test environments and test objects.
This commit rewrites all existing tests from test/functional/bundlelist
but this time using the new test library. Once all the tests have been
rewriten to use the new lib, they can be included in the Makefile so
they are run with every patch, and the old tests can be deleted.
This commit adds the last rewritten tests for bundle-add, these
tests are the equivalent to the ones in the bundleadd directory
but using the new test library.
This commit adds some assertions that can be used to validate
common things in a consistent manner among tests.
The assertions added with this commit are the following:
- assert_status_is
- assert_status_is_not
- assert_dir_exists
- assert_dir_not_exists
- assert_file_exists
- assert_file_not_exists
- assert_in_output
- assert_not_in_output
More assertions may be added as needed while migrating the rest of
the tests to use the new testlib.
There is some stuff that is common for every test file, so this
commit adds a function that generates a new test file based on a
template to facilitate test creation.
The old way of creating new tests require a lot of manual steps to
create the test resources needed to run the tests. This commit is
part of a series of commits to improve the workflow for creation of
new tests.
This commit introduces a new test library that aims to replace the
old swupdlib.bash library which provides many funtions to create
new test resources easily.
The following functions were added in this commit:
- generate_random_content
- generate_random_name
- validate_path
- validate_item
- validate_param
- create_dir
- create_file
- create_link
- create_tar
- create_manifest
- add_to_manifest
- create_test_environment
- destroy_test_environment
- create_bundle
- set_env_variables
- sign_manifest
The tests in bundleadd_v2 would eventually replace those in bundleadd
Related to issue #258
It's inconsistent to list just the first file missing in a pack. As
There's already a warning saying home many files are missing in
packs, remove this print.
Curl initialization were split in 2 different functions, swupd_curl_init(),
that creates the curl handles and swupd_curl_check_network() that checks the
network and set alternative CA paths, if needed.
Merging those two functions in one to perform all curl initialization all at
once.
Also fixes:
- Fix cases where curl was used before calling swupd_curl_check_network() and
then the correct CA path wasn't set.
- Don't use #ifdef FALLBACK_CAPATHS because FALLBACK_CAPATHS is always defined.
It's set to an empty string if fallback ca paths are disabled.
- Don't use extra memory to duplicate content of FALLBACK_CAPATHS and store
it in a global.
Every bundle includes a tracking file which tells the OS the
bundle is installed, every bundle has this tracking file as
part of its manifest. When removing a bundle from a system,
all the files that are part of that bundle, and are not needed,
for any other bundle are removed, including the tracking file.
This patch removes a function call that is intended to remove
the tracking bundle from the system, since it is not necessary,
by the time the code calls that function, all files from the
manifest that can be removed are already removed, and this includes
the tracking file.
This patch also modifies a few tests to reflect the change in the
command output.
When running all tests at the same time using make -j check,
update/slow-server and check-update/slow server were trying to start
the slow server at the same port.
Castulo Martinez