Instead of giving up on directory subtraction entirely, try to
subtract directories that are in an included manifest *and* do not
have a non-directory file somewhere in their path.
For example:
bundle1:
/dir/file
bundle2: includes bundle1
/dir
In this case /dir would be subtracted from bundle2.
Also add a special case for subtracting directories if they are in
os-core as this reduces a lot of unused content from a bundle's
manifest.
Signed-off-by: William Douglas <william.douglas@intel.com>
Add package level bundles "pundles" first class support in mixer. This
involves tagging pundles with "pundle" in the bundle definition file's
"MAINTAINER" field.
This support requires changes to bundle constraints, primarily that
cycles are now allowed in bundle includes. As part of this, directory
subtraction will no longer be done for includes as the true directory
owner isn't something that can be decided on in a cycle.
Validation also needed a slight adjustment for pundles to reflect
their minimal bundle definition files.
Signed-off-by: William Douglas <william.douglas@intel.com>
Improve the usability of log messages by indicating which manifest is
invalid when printing errors.
Signed-off-by: William Douglas <william.douglas@intel.com>
Systems using certain modifiers are currently not used as frequently
so in order to reduce pack size and improve update times skip those
modifiers from being considered for packs.
Currently the SSE version is skipped when either AVX2 or AVX512
versions exist and APX is always skipped.
This change does not apply to 0 packs as it would break installer
offline support.
Signed-off-by: William Douglas <william.douglas@intel.com>
These were unintentionally left out and while unlikely to be used
should be included for correctness.
Signed-off-by: William Douglas <william.douglas@intel.com>
Add cases for optimization prefixes being changed and directories with
optimizations being skipped.
Signed-off-by: William Douglas <william.douglas@intel.com>
When considering delta peer matches, skip any directory with an
optimization prefix for consideration. The the directories are not
intended to be different between optimization versions.
Signed-off-by: William Douglas <william.douglas@intel.com>
The optimized prefix doesn't set Misc flags until the last moment
during manifest writing so ignore the flags when testing for peers
when a prefix is in place.
Also improve some comments and test cases.
Signed-off-by: William Douglas <william.douglas@intel.com>
As other mixer processes might use the manifests after they are
written, reset the path to include the prefix once the manifest write
is complete.
Signed-off-by: William Douglas <william.douglas@intel.com>
If the modifiers of a file indicate it came from a chroot path, reset
the filename to use the chroot path when reading the file information
from a manifest.
Also ensure the only case files are marked as deleted is when it is
the SSE file. Other optimized files marked as deleted will be skipped
from being output in the manifest (and if the delete is new, the other
files with the same post processed path will be marked as new in the
version being built to ensure the deleted file is replaced).
Signed-off-by: William Douglas <william.douglas@intel.com>
The codebase uses sortFilesVersionName prior to writing out manifests
and that should be used in setupModifiers as well.
Signed-off-by: William Douglas <william.douglas@intel.com>
This change allows for a chroot with specific versioned content with
matching optimization levels to be added and accounted for. The paths
will be stripped of a version prefix (currently /V3 or /V4) and the
file's Modifier will be setup accordingly. The modifier will then be
updated to reflect the optimization level of the file and the total
optimization of any file with the same path.
With this change the previous Modifier setting has been revamped and
the old 'C', 'b' and 's' fields will be ignored. The 'b' posed a
slight problem that needed to be accounted for with how ghosted files
are detected and this has been done through looking only at the files
path.
Signed-off-by: William Douglas <william.douglas@intel.com>
Sometimes (like hex coded license files) there's value in not only ignoring digits
for detecting renames, but also ignoring a-f to deal with hex mismatches
in https://github.com/clearlinux/mixer-tools/issues/782 a funky issue is found where
we do not use the right versions for delta files in some cases.
A full fix is complicated, but the client IGNORES the versions! so we can just hardcode
the versions to anything (10 and 20 here) and it'll suddenly resolve itself
we need to not return "nil" if a delta file is missing, because that causes the caller
to not use the fallback option (which would include the full file)...
and we need the fallback option to have a complete pack
Signed-off-by: Arjan van de Ven <arjan@linux.intel.com>
This reverts commit 4c115e5a3e.
With content changes so that the /usr/bin and /usr/lib64 files should
still exist even using optimized libs this is no longer needed.
Signed-off-by: William Douglas <william.douglas@intel.com>
While creating delta packs, only possible delta files are found.
If delta file is not found, log the error in debug mode instead
of generating warning, as delta might not have been
created for valid reason.
fixes#628
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Fixes#666
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Improve the performance of processing bundles by parallelizing readIncludes()
and detectManifestChanges() operations.
Fixes#725
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Currently, mixer build update performs multiple hash calculations for each file in the chroot.
Every manifest file entry and subtracted file will have their hashes calculated.
It would be a significant performance improvement to calculate the hash one time for each file and re-use it.
fixes#585
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
By default, files with the prefix "/bin/", "/usr/bin/" and "/usr/local/bin/"
will have a new export flag "x" in the 4th byte of the flag section of the
bundle Manifest.
The user can override this by using a new keyword "un-export()" in
the bundle definition file.
E.g. un-export(/usr/bin/file2)
Note: The export flag is at a bundle-file level and hence does not get
written to the Full manifest.
Fixes#708
Includes bat tests for the export flag feature.
Also, adds an entry for the content-chroot tests to Travis.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Change RenameFlag to MiscFlag since "rename" has been deprecated and
this flag will be used as a placeholder for various other flags.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Currently while catching errors using channels and select
statements, when the error is caught in select statements, it
is not breaking from the outside loop.
The correct approach is to let the existing go routines finish
executing and return one of the errors caught by the error channel.
It should also close the channel in order to stop executing new go
routines.
fixes#673
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
When the previous manifest's list of files was longer than the new
manifest's list of files there was a chance for previously deleted file
entries to be unexpectedly dropped. This change prevents previously
deleted files from dropping when the previous manifest's file list is
longer than the new manifest's list.
Fixes#684
Signed-off-by: John Akre <john.w.akre@intel.com>
Generate an empty bundle os-core-update-index to replace the old index,
deleting all files included in previous manifests.
- if the previous index exists and it's not empty, we generate an empty index
- if the previous index doesn't exist we don't generate any index
- if the previous index is an empty index, we just point to it in the new MoM
Fix#541
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Tracking file and directory was being created with incorrect permissions.
This is important because tracking files have excatly the same hash, except
this one.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Add support to zstd for fullfile creation. This compression method can
be enable via config using option "external-zstd".
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch allows users to configure the compression methods that will
be used for fullfiles creation. The compression method can be set in the
Swupd.COMPRESSION property in builder.conf. Instead of enabling all
methods as previosly, xz is the only method that comes enabled by
default.
Note: internal gzip is still used for Link and Directory fullfile
creation.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When reading a MoM always ignore IManifests. This is important to prevent
IManifests to be carried out to next mom in an update
Also adding some deprecation warnings
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
William Douglas