This release includes minor bug fixes along with implementation of
logging.
Enhancements:
- Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`.
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Other Changes:
- Remove the --no-publish flag.
Mixer will now always update the LAST_VER file after `build update`.
- Update github actions to v2 to handle CI re-runs.
- Update Makefile to build man pages as a prerequisite for the
`build` target.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Reduce cluttering the logs by logging command tags only for the external
applications used within Mixer.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Fixes#666
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Update man page to include `format` flag.
Also, update Makefile to build man pages as a prerequisite for the
`build` target.
Fixes#557
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Mixer will now always update the LAST_VER file after `build update`.
This file is used by `build validate` (MCA) to determine whether
the version in comparison is a +10.
Fixes#735
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes performance improvements, minor bug fixes and
removal of the Mixin feature.
Changes:
- Remove the Mixin feature as Mixer now supports creation of 3rd party content.
- Calculate the hash for each file just once.
- Parallelize bundle processing.
- Parallelize bundle file resolving.
- Update mix variables when crossing upstream formats.
- Fix MCA bundle deletion during format bump.
- Add more debug messages for DNF errors.
- Fix permission issue in bat test.
- Replace Travis CI with GitHub Actions.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
In addition to printing a summary of unresolvable package names whenever
DNF fails, also print error messages for any "unknown" DNF error, and
finally print the full stdout/stderr from failed DNF commands for later
inspection.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When bundle is deleted in a mix, bundle info file is not generated for that
bundle. When parsing the manifest, if bundle info is empty, consider that
bundle to be deleted in the mix and do not resolve packages for that
bundle during the MCA process.
Fixes#743
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
By creating a separate installroot for each package resolving goroutine,
dnf can run in parallel without contention on the lock file when
accessing the dnf cache. Now, package and file resolution occurs within
the same installroot so that the dnf cache that was fetched when resolving
packages can be re-used when resolving files.
Fixes#730
Signed-off-by: John Akre <john.w.akre@intel.com>
When a downstream mix crossed multiple upstream format bumps, the
upstreamversion and mix state variables were updated in the workspace,
but the changes were not updated in memory. This change reloads the
upstreamversion when it is staged/unstaged and reloads the mix state
after crossing each format boundary. Additionally, this change
increments the format number for each upstream format that the
downstream mix crosses.
Fixes#688
Signed-off-by: John Akre <john.w.akre@intel.com>
Improve the performance of processing bundles by parallelizing readIncludes()
and detectManifestChanges() operations.
Fixes#725
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Currently, mixer build update performs multiple hash calculations for each file in the chroot.
Every manifest file entry and subtracted file will have their hashes calculated.
It would be a significant performance improvement to calculate the hash one time for each file and re-use it.
fixes#585
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
This release includes minor enhancements and bug fixes.
Enhancements:
- Initialize local yum repo even when its empty as newer versions of
DNF requires that yum repos be initialized.
- Install special case files in the end so that they will not create
directories with potentially conflicting permissions.
The order of installation is now packages, contents and then special
case files.
Bug Fixes:
- Set file permissions in the chroot after creating it and setting the
file owner. This is necessary because umask can prevent newly created
files from acquiring the source file's permissions.
- Don't follow symlinks when comparing files between content and full
chroot.
- Skip format bump check in MCA.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
When adding special case files to the full chroot, they will create
missing directories in their path with permissions that may conflict
with content that will be added at a later point. By adding special case
files last, they will not create directories with potentially
conflicting permissions.
Fixes#738
Signed-off-by: John Akre <john.w.akre@intel.com>
When comparing files between content and full chroots, do not follow
symlinks so that the links can be compared.
Fixes#739
Signed-off-by: John Akre <john.w.akre@intel.com>
When calling the copyFileWithFlags function with the preserveSrc flag
set, file permissions must be set after creating the destination file
and setting the file owner. This is necessary because umask can prevent
newly created files from acquiring the source file's permissions and
file ownership changes can impact the setuid/setgid permissions.
Similarly directory permissions are set after creating the directory to
avoid unexpected permission behavior caused by umask.
Signed-off-by: John Akre <john.w.akre@intel.com>
Newer versions of DNF require that yum repos are initialized, so Mixer
must initialize the local repo, even when it's empty.
Fixes#731
Signed-off-by: John Akre <john.w.akre@intel.com>
This release includes support for building 3rd party content along with
other enhancements and bug fixes.
New features (beta) for 3rd party support:
- Create a minimal mix using just an os-core bundle that can be empty.
- Add BUILD_ID field in os-release file that specifies the upstream
version from which the mix is created.
- Build bundles from contents by providing the content path using a new
keyword "content()" in the bundle definition.
- Introduce a new export flag "x" in bundle manifests to indicate exportable
files in "/bin/", "/usr/bin/" and "/usr/local/bin/".
The user can override this flag by specifying the files using a new
keyword "un-export()" in the bundle definition.
Other changes:
- Fix error to handle bundles that contain only optional bundles.
- Fix build errors due to stale dnf cache by cleaning the dnf cache before using it.
- Clean mix workspace before building bundles.
- Add --local flag for bundle creation that skips the upstream check and
creates empty local bundles.
- Improve performance by writing mixbundles file just once for the resolved bundle set.
- Make bundle messages consistent.
- Print delta pack results together to prevent interleaved results.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
When using content chroots, file uid/gid values should be preserved when
copying them to the full chroot.
Fixes#727
Signed-off-by: John Akre <john.w.akre@intel.com>
When resolving the full mix bundle set, the mixbundles file is now
updated once with the finalized set of bundles. Previously, the
mixbundles file was written many times with intermediate results which
scaled poorly when building with many bundles.
Fixes#724
Signed-off-by: John Akre <john.w.akre@intel.com>
When building a mix with a version of a package repo that contains
existing cache contents, dnf may resolve incorrect results.
For instance, Mixer does not version the local package repo, so when
incrementally building a mix with local content, dnf will continue to
re-use the same cache for the local repo. This was problematic when the
version of an rpm in the local repo was updated. In this case, the stale
cache metadata was used to incorrectly resolve files for the package.
Cleaning the dnf cache before using it prevents old cache metadata from
impacting dnf results.
Fixes#719
Signed-off-by: John Akre <john.w.akre@intel.com>
By default, files with the prefix "/bin/", "/usr/bin/" and "/usr/local/bin/"
will have a new export flag "x" in the 4th byte of the flag section of the
bundle Manifest.
The user can override this by using a new keyword "un-export()" in
the bundle definition file.
E.g. un-export(/usr/bin/file2)
Note: The export flag is at a bundle-file level and hence does not get
written to the Full manifest.
Fixes#708
Includes bat tests for the export flag feature.
Also, adds an entry for the content-chroot tests to Travis.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Change RenameFlag to MiscFlag since "rename" has been deprecated and
this flag will be used as a placeholder for various other flags.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Currently, build bundles generates an error if one of the mix bundles
has only an 'also-add()' entry in its definition.
Instead, build bundles should not error out and must process such
bundles.
Fixes#718
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Delta packs are generated concurrently, so their results output must be
printed once to prevent interleaved results. Additionally, this change
redirects warnings to stderr so that can be separated from the results.
Fixes#551
Signed-off-by: John Akre <john.w.akre@intel.com>
Passing '--local' with mixer bundle create will skip
the upstream check and create a new empty local bundle
if it does not already exist.
fixes#713
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Bundle chroots do not resolve conflicts against the full chroot and the
workflow is not streamlined. Now, the bundle definition file should be
used to integrate custom content into the mix.
The bundle definition file supports content chroots which are integrated
into the bundle's info file and resolved against the full chroot.
Ex: content(/path)
Signed-off-by: John Akre <john.w.akre@intel.com>
Files in the content chroots specified by the "contents" keyword in the
bundle definition file are compared against the full chroot to resolve
conflicts which are fatal. When there are no conflicts, files from the
content chroots are copied into the full chroot and added to the
corresponding bundle-info file.
Fixes#705
Signed-off-by: John Akre <john.w.akre@intel.com>
The CopyFileWithOptions function has the ability to disable symlink
resolution, preserve the source file permissions, and disable disk syncs.
Signed-off-by: John Akre <john.w.akre@intel.com>
The content keyword in the bundle definition file specifies the path to
a chroot whose file contents will be added to the bundle's manifest.
Ex: content(/path)
Signed-off-by: John Akre <john.w.akre@intel.com>
Existing image and output directories for the current mix version will
be removed when building a mix. This prevents mixer from building on top
of directories contaminated by a prior mix.
Fixes#669
Signed-off-by: John Akre <john.w.akre@intel.com>
If mix is minimal mix with empty os-core, directory path for
os-release file does not exist.
Create a directory path before updating the os-release file
BUILD_ID field in os-release file specifies the upstream Clear
Linux version from which the mix is being created.
If upstream Clear Linux is not found, BUILD_ID is empty.
fixes#706
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
In order to enable a minimal mix for 3rd party, "filesystem" package is made optional.
In case the "filesystem" package is found in any of the mix bundles,
install the "filesystem" package first. If it’s not found in any of the mix bundles,
then do not install it.
swupd-client expects a mix to have a bundle named "os-core" and the "os-release" file.
Hence they are a hard requirement for creating a mix in Mixer.
If the "os-release" file is not found, then create and inject a default one.
fixes#704
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
This release includes minor enhancements and bug fixes.
Changes:
- Add root permission check to inform the user when certain commands
require root permission.
- Refactor error handling in go channels to prevent potential hanging of
the application in certain scenarios.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
The build commands require root in order to run dnf, clr-installer, or
access files in the update folder that are owned by root.
Fixes#690
Signed-off-by: John Akre <john.w.akre@intel.com>
Currently while catching errors using channels and select
statements, when the error is caught in select statements, it
is not breaking from the outside loop.
The correct approach is to let the existing go routines finish
executing and return one of the errors caught by the error channel.
It should also close the channel in order to stop executing new go
routines.
fixes#673
Signed-off-by: Ashlesha Atrey <ashlesha.atrey@intel.com>
Reagan Lopez