This change has proven to cause unnecessary churn for the bundle
tracking files when their contents ended up not being consumed by any
tools.
This reverts commit 6f840d712e.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Add the packages found in a bundle to the bundle tracking file. This
will be handy when trying to figure out client side what packages are
installed on the system.
Signed-off-by: William Douglas <william.douglas@intel.com>
While creating delta packs, only possible delta files are found.
If delta file is not found, log the error in debug mode instead
of generating warning, as delta might not have been
created for valid reason.
fixes#628
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Fixes#666
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Mixer will now always update the LAST_VER file after `build update`.
This file is used by `build validate` (MCA) to determine whether
the version in comparison is a +10.
Fixes#735
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
In addition to printing a summary of unresolvable package names whenever
DNF fails, also print error messages for any "unknown" DNF error, and
finally print the full stdout/stderr from failed DNF commands for later
inspection.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When bundle is deleted in a mix, bundle info file is not generated for that
bundle. When parsing the manifest, if bundle info is empty, consider that
bundle to be deleted in the mix and do not resolve packages for that
bundle during the MCA process.
Fixes#743
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
By creating a separate installroot for each package resolving goroutine,
dnf can run in parallel without contention on the lock file when
accessing the dnf cache. Now, package and file resolution occurs within
the same installroot so that the dnf cache that was fetched when resolving
packages can be re-used when resolving files.
Fixes#730
Signed-off-by: John Akre <john.w.akre@intel.com>
When a downstream mix crossed multiple upstream format bumps, the
upstreamversion and mix state variables were updated in the workspace,
but the changes were not updated in memory. This change reloads the
upstreamversion when it is staged/unstaged and reloads the mix state
after crossing each format boundary. Additionally, this change
increments the format number for each upstream format that the
downstream mix crosses.
Fixes#688
Signed-off-by: John Akre <john.w.akre@intel.com>
When adding special case files to the full chroot, they will create
missing directories in their path with permissions that may conflict
with content that will be added at a later point. By adding special case
files last, they will not create directories with potentially
conflicting permissions.
Fixes#738
Signed-off-by: John Akre <john.w.akre@intel.com>
When comparing files between content and full chroots, do not follow
symlinks so that the links can be compared.
Fixes#739
Signed-off-by: John Akre <john.w.akre@intel.com>
When calling the copyFileWithFlags function with the preserveSrc flag
set, file permissions must be set after creating the destination file
and setting the file owner. This is necessary because umask can prevent
newly created files from acquiring the source file's permissions and
file ownership changes can impact the setuid/setgid permissions.
Similarly directory permissions are set after creating the directory to
avoid unexpected permission behavior caused by umask.
Signed-off-by: John Akre <john.w.akre@intel.com>
Newer versions of DNF require that yum repos are initialized, so Mixer
must initialize the local repo, even when it's empty.
Fixes#731
Signed-off-by: John Akre <john.w.akre@intel.com>
When using content chroots, file uid/gid values should be preserved when
copying them to the full chroot.
Fixes#727
Signed-off-by: John Akre <john.w.akre@intel.com>
When resolving the full mix bundle set, the mixbundles file is now
updated once with the finalized set of bundles. Previously, the
mixbundles file was written many times with intermediate results which
scaled poorly when building with many bundles.
Fixes#724
Signed-off-by: John Akre <john.w.akre@intel.com>
When building a mix with a version of a package repo that contains
existing cache contents, dnf may resolve incorrect results.
For instance, Mixer does not version the local package repo, so when
incrementally building a mix with local content, dnf will continue to
re-use the same cache for the local repo. This was problematic when the
version of an rpm in the local repo was updated. In this case, the stale
cache metadata was used to incorrectly resolve files for the package.
Cleaning the dnf cache before using it prevents old cache metadata from
impacting dnf results.
Fixes#719
Signed-off-by: John Akre <john.w.akre@intel.com>
By default, files with the prefix "/bin/", "/usr/bin/" and "/usr/local/bin/"
will have a new export flag "x" in the 4th byte of the flag section of the
bundle Manifest.
The user can override this by using a new keyword "un-export()" in
the bundle definition file.
E.g. un-export(/usr/bin/file2)
Note: The export flag is at a bundle-file level and hence does not get
written to the Full manifest.
Fixes#708
Includes bat tests for the export flag feature.
Also, adds an entry for the content-chroot tests to Travis.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Currently, build bundles generates an error if one of the mix bundles
has only an 'also-add()' entry in its definition.
Instead, build bundles should not error out and must process such
bundles.
Fixes#718
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Delta packs are generated concurrently, so their results output must be
printed once to prevent interleaved results. Additionally, this change
redirects warnings to stderr so that can be separated from the results.
Fixes#551
Signed-off-by: John Akre <john.w.akre@intel.com>
Passing '--local' with mixer bundle create will skip
the upstream check and create a new empty local bundle
if it does not already exist.
fixes#713
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Files in the content chroots specified by the "contents" keyword in the
bundle definition file are compared against the full chroot to resolve
conflicts which are fatal. When there are no conflicts, files from the
content chroots are copied into the full chroot and added to the
corresponding bundle-info file.
Fixes#705
Signed-off-by: John Akre <john.w.akre@intel.com>
The content keyword in the bundle definition file specifies the path to
a chroot whose file contents will be added to the bundle's manifest.
Ex: content(/path)
Signed-off-by: John Akre <john.w.akre@intel.com>
Existing image and output directories for the current mix version will
be removed when building a mix. This prevents mixer from building on top
of directories contaminated by a prior mix.
Fixes#669
Signed-off-by: John Akre <john.w.akre@intel.com>
If mix is minimal mix with empty os-core, directory path for
os-release file does not exist.
Create a directory path before updating the os-release file
BUILD_ID field in os-release file specifies the upstream Clear
Linux version from which the mix is being created.
If upstream Clear Linux is not found, BUILD_ID is empty.
fixes#706
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
In order to enable a minimal mix for 3rd party, "filesystem" package is made optional.
In case the "filesystem" package is found in any of the mix bundles,
install the "filesystem" package first. If it’s not found in any of the mix bundles,
then do not install it.
swupd-client expects a mix to have a bundle named "os-core" and the "os-release" file.
Hence they are a hard requirement for creating a mix in Mixer.
If the "os-release" file is not found, then create and inject a default one.
fixes#704
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Currently while catching errors using channels and select
statements, when the error is caught in select statements, it
is not breaking from the outside loop.
The correct approach is to let the existing go routines finish
executing and return one of the errors caught by the error channel.
It should also close the channel in order to stop executing new go
routines.
fixes#673
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
If `build bundles` fail to resolve packages,
print the missing packages in the error message.
Fixes#697
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
When searching for deprecated bundles to detect bundle deletion, both
the upstream and downstream bundles must be checked. Previously,
upstream bundles were not checked during bundle deletion which prevented
downstream mixes from detecting upstream bundle deletions.
Fixes#694
Signed-off-by: John Akre <john.w.akre@intel.com>
Now the mixer repo list command displays repo priorities and the mixer
repo add command can set the repo's priority with the --priority flag.
Signed-off-by: John Akre <john.w.akre@intel.com>
The set-priority repo command updates the priority value for the
specified repo in the DNF config file. This command uses the following
format:
mixer repo set-priority <repo> <priority>
Fixes#691
Signed-off-by: John Akre <john.w.akre@intel.com>
The functions to set values in the DNF config were nearly identical.
This change consolidates the repo set functionality into a single
function to increase code re-use.
Signed-off-by: John Akre <john.w.akre@intel.com>
The bundle tracking file is added to each manifest by mixer and does not
exist in an rpm. This change adds a check for the bundle tracking file
when adding a bundle.
Signed-off-by: John Akre <john.w.akre@intel.com>
The diff results analysis section was refactored to make it easier to
handle special case manifest files.
Signed-off-by: John Akre <john.w.akre@intel.com>
If an rpm is not found in the assumed cache dir of its repo, update the
repo cache dir list based on the location of the rpm.
This is an attempt to improve performance in case the rpms are not
stored directly within the repo baseurl.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
The MCA statistics table replaced the tabwriter implementation with the
tablewriter package. Now the statistics table is a complete box with
line wrapping support for long lines.
This change also adds the --table-width parameter which is used to set the
maximum width of the MCA statistics table. The default value is the
width of the calling terminal and the table can be disabled by providing
a negative number for the table-width.
Signed-off-by: John Akre <john.w.akre@intel.com>
Currently, in order to improve performance, dnf install is not used
to extract the packages. Hence the database used to create os-packages
file is not being populated and thus the os-packages file is empty.
Moreover the os-packages file is not required and used by anybody.
Remove functions related to creation of the os-packages file.
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Currently, for repos with "file" baseurl scheme, mixer expects the
rpms to be directly within the baseurl path.
But it is possible that rpms can be in sub directories within the repo baseurl.
In order to handle this, if an rpm is not found directly within the repo baseurl,
run repoquery to determine the actual location of the rpm.
This is expensive and it is recommended that rpms are stored directly
within the repo baseurl.
Includes a bat test.
Fixes#674
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
The rpm command can fail to download packages over the network, but can
recover with download retries.
Signed-off-by: John Akre <john.w.akre@intel.com>
To improve performance instead of querying packages downloaded by dnf,
the package URIs are resolved with dnf repoquery which are used to query
the packages with rpm. This implementation is much faster for dnf
repositories on the host system because no package downloads will occur.
Additionally, the new implementation does not require rpms to be named
with the NVRA format (name-version-release.arch.rpm), so alternatively
named rpms will be resolved successfully.
Fixes: #665
Signed-off-by: John Akre <john.w.akre@intel.com>
Patrick McCarty