This release fixes a bug with adding invalid bundle manifest hashes to
the Manifest.MoM and optimizes swupd_create_update to remove the
unnecessary rsync from the full chroot back to bundle chroots.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Just in case we encounter a bundle manifest that does not exist anymore,
or the last_change is 0, which is an invalid version to create, make
sure we add the deleted entry in the MoM and do not attempt to untar
that manifest.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The bundle manifest hashes were calculated from the STATE_DIR Manifest.bundle
files, which caused a problem when they the permissions/owner were changed. The
hashes should be calculated by untarring the Manifest.BUNDLE.tar files
referenced in the Manifest.MoM.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release includes a trivial improvement for reporting paths that
contain blacklisted characters.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To better track which files are excluded from manifests because they
contain blacklisted characters, the full path is more useful to log
rather than the file basename.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since the source files for entry points to the other two binaries are
named after the binaries, do the same for swupd_create_update.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To more easily implement tests, analyze test results, etc, store each
test in a separate directory and dump all logs and the web-dir content
there as well.
The test suite can be run with 'make check', as before, but now tests
can be run individually by directly invoking the appropriate test.bats
file (at toplevel, or within the test directory).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release fixes a sorting issue with bundle manifests: they should be
version sorted, not filename sorted.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release adds support for a bundle includes feature, which allows
bundles that "include" other bundles to depend on each other. This
change reduces duplication of files in manifests and improves build
times for larger bundle sets that contain many includes.
The two other functional changes are:
- The --format option to swupd_create_update is now mandatory.
- The value passed to --format must now be a positive integer.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since removing of all of the matching deleted files between a build with
an older manifest format and current needs to happen *after*
match_manifests() finishes accounting for new and deleted entries in the
manifests, simply move the logic into a separate function to be called
after match_manifests().
Because one or more file deletions should trigger a fresh bundle manifest
creation, account for the return value of this new function in addition
to the result of manifest pruning and whether any bundle includes were
changed. Since "os-core" is treated specially, and it's expected to
contain changed files for every build, only the non-"os-core" bundle
manifests need the trigger.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For the new full manifest and new bundle manifests, the format member
was not being initialized.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For swupd_create_update, the required -F option needed to be added for
several tests, and the option parsing order changed.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To decrease chances of specifying an incorrect format number, since the
default is a hardcoded value "3", always require the user to pass the -F
option to swupd_create_update.
In the future, it would be nice to read the default value from a config
file, with the -F option overriding the value.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To better support a workflow for incrementing the format on the command
line, it's useful to have the format be an integer instead of a string.
When reading a manifest (ie: old manifest) we can read that into struct
manifest. Then when building a new manifest we can compare against the
prior manifest's format. If the format incremented, we can prune files
previously marked as deleted in the old manifest so they no longer
appear in the new manifest.
This is beneficial because a file that has a version number in its name
will repeatedly be deleted and replaced by a new similar file. Over
time this leads to manifests getting bigger and bigger because they
retain the entire history of all file names which ever existed, even if
a huge number of them are and have been for a long time marked as
deleted.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
When deciding if a new manifest should be generated for a new version,
the included manifests were not being used to detect a manifest
change. This caused manifests that had new includes: lines in the new
version to not be generated resulting in files from the included
manifest to be missed when adding/updating that bundle.
The fix was to compare includes: lines from the current and previous
manifest versions for differences and generate new manifests when that
was the case. In order to compare manifests duplicate includes: lines
are not allowed and previously os-core could be added multiple times to
a manifest because it was always added. This change also added detection
for os-core before automatically adding it to a bundle.
Check that os-core is subtracked correctly and that $bundle-includes
files are correctly subtracked as well. Also include testing for new
bundles being subtracked.
Now that much of the includes plumbing present on the output/subtract
side, make sure all non os-core manifests explicitly include os-core,
and then use the new extra semantics of subtract_manifests() to subtract
all included bundles, not just os-core.
Using the $bundle-includes files for building submanifests from
directories and previous manifest files includes: sections, create the
manifest->includes list as the names of submanifests included in the
submanifest being instantiated.
This needs to then be processed into a list of pointers to the other
submanifests themselves once all other manifests have been
instantiated before use with functions that operate on the includes list
(subtract_manifest and write_manifest for now)..
When subtracting a manifest from another, all the included manifests
also will need to be subtracted from the original. To do this build up a
unique list of all manifests included (directly or indirectly) from the
manifest being subtracted from.
With that in place, this gives an opportunity to provide the
functionality of subtracting all included manifests from the main
manifest by calling subtract_manifests(A, A);
If a bundle includes one or more other bundles, print out one or more
includes: lines as part of the header of the manifest file.
There will be as many includes: lines as included bundles (including
being omitted if a bundle does not include other bundles).
This release enables logs for the three installed binaries and optimizes
swupd_make_pack performance by preferring to hardlink files to the pack
staging area whenever possible.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To capture all information that is logged, change init_log() to make
every log file name unique, and add a call to swupd_make_fullfiles to
enable logging.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
A big chunk of time during pack creation is spent in untar'ing of
fullfiles tar files. However, we also (most likely) have this exact
same content already in the image/ directory.
This patch makes the pack creator skip the untar if its possible to link
the file directly, and falls back to the existing behavior if the link
fails
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release fixes a bug in manifest subtraction logic that may result
in "partial" subtraction of one manifest from another. The fix now
ensures that a manifest can be fully subtracted.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
In subtract_manifests(), if the two files under consideration have the
same filename but should not be subtracted, a file is skipped in the m2
manifest because there is no "continue" statement at the end of the
first "if" block.
This results in potentially many files not getting subtracted when they
ought to be.
To clarify the logic, avoid using "continue", since it is not needed.
Convert to using one "if" block instead.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release adds support for bsdtar (compile-time option), enables
parallel building of delta packs with pack_maker.sh, adds an initial
functional test suite, and addresses a few bugs:
- Makes parallel pack generation more robust when building packs for
multiple versions simultaneously.
- Fixes basic_creator.sh to work properly on first run.
- Removes an unneeded -a option passed to tar, and fixes tar argument
ordering (files always after options).
- Makes hash calculation always account for xattrs, since swupd-client
unconditionally accounts for them.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Even though swupd_make_pack will exit immediately in the event this
error occurs, calling free() here will be needed if the error handling
ever changes (e.g. returning from the function instead of exiting).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Because pack generation is a time-consuming task, running this task in
parallel is preferable.
However, if we are generating two or more packs with the same "from"
version at the same time, the multiple jobs will use the same pack
staging directory, leading to corruption.
My solution for now is to use a staging dir with a name derived from
both "from" and "to" versions so that there are no collisions.
A long-term fix will be to implement some kind of locking mechanism to
prevent *identical* packs from being generated simultaneously, but for
now, using more unique staging directories will improve the situation.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since GNU tar fails to extract files with xattrs preserved when
Integrity Measurement Architecture (IMA) is enabled some vendors
may choose to install libarchive-based tar (bsdtar) on their embedded
devices, so the swupd server needs to be able to create archives
in its format.
This patch adds one compile-time options --enable-bsdtar that is used
to enable/disable GNU tar specific options. Also it harmonizes
the command strings to be compatible with both GNU tar and bsdtar.
Particularly it
- changes --exclude pattern from '%s'/* to more explicit '%s/?*' because
bsdtar's pattern matching is greedier than in tar: it uses tcsh's
globbing where '*' can be anything including the null string and the
original pattern would include the directory itself;
- OS file names are escaped with leading ./ to avoid collisions with
file names starting with @ which has special meaning in bsdtar.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
The latest.version file should generally be updated after the most
recent build completes, so bump it to 10 after the build completes for
version 10 in this test.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The test suite uses BATS to emit TAP and the Automake TAP driver to
consume the output.
Most of the tests are basic, and full-run.bats creates a minimal update
using autogenerated chroot content.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Similar to how swupd_create_update handles argc/argv, swupd_make_pack
and swupd_make_fullfiles should validate the argument count before
checking for root privileges, etc.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Currently swupd-client includes xattrs to hash sums of all files.
Therefore in order to avoid relaxing security and to prevent
`swupd verify` from reporting hash mismatches for updated files
it's better to include xattrs to hashes for all files.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
Patrick McCarty