Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Fixes#666
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Mixer will now always update the LAST_VER file after `build update`.
This file is used by `build validate` (MCA) to determine whether
the version in comparison is a +10.
Fixes#735
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Existing image and output directories for the current mix version will
be removed when building a mix. This prevents mixer from building on top
of directories contaminated by a prior mix.
Fixes#669
Signed-off-by: John Akre <john.w.akre@intel.com>
Currently mixer allows users to set multiple repos using 'mixer repo set' commands and name them anyhow.
But the code within the mixer is tightly coupled to having only two repos - "clear" and "local".
This inconsistency is fixed.
fixes#654, #659
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
Switch from ister.py to clr-installer for generating OS images.
Will migrate existing ister JSON file if it exist.
Both ISO generate and Raw Image retention enabled by default.
Signed-off-by: Mark D Horn <mark.d.horn@intel.com>
Built-in Docker support was originally added to enable auto-format bump.
At that time, mixer could only handle one format version at a time, so
we needed 2 different binaries to cross a format boundary: 1 for the +10
and one for the +20. The docker support would allow us to download an
image with the right version for that given format and simplify the
process.
Mixer now has support for multiple formats and a single binary can
handle the whole format bump process. Built in docker image is no longer
needed.
Note: For Docker usage due to compatibility issues with the host system,
mixer-ci Docker image should be used instead.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
In getUpstreamBundles(),
- Remove the prune param as it is always true. Fixes#624
- Remove the version param as it is available in the method.
In getUpstreamBundlesPath(),
- Remove the version param as it is available in the method.
Also, remove the need for downloadUpstreamBundles() method by including
its code in getUpstreamBundles().
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Currently, the bundle defintions are downloaded from the default
clr-bundles repo https://github.com/clearlinux/clr-bundles/archive/.
The user can now specify a different upstream bundles url using a mixer init flag or mixer
config parameter.
E.g. mixer init --upstream-bundles-url=https://github.com/reaganlo/clr-bundles/archive/
E.g. mixer config set Swupd.UPSTREAM_BUNDLES_URL https://github.com/reaganlo/clr-bundles/archive/
Since there is a tight coupling between the --upstream-version and bundles
repo, the user should ensure that the bundles repo has the same structure as the
clr-bundles repo and the tar.gz files should be named after an upstream
version. i.e. If the mix is based on an upstream-version 31300, the bundles repo
should have a file 31300.tar.gz
Fixes#614
Note: Setting this value using `mixer config set` will only update it in builder.conf.
It does not fetch the upstream bundles. A dedicated command for that purpose
will have to be implemented instead.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
InitMix() calls AddBundles() and getUpstreamBundles().
AddBundles() already calls getUpstreamBundles().
So remove the redundant function call in InitMix().
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
golangci-lint identified a couple errors that were not found by
gometalinter. This patch fixes the detected errors.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The latest file in version/formatXX and version/latest_version isn't signed by mixer.
Mixer signs the these files so clients could check the signature of that file too.
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
MixState needs to know the server state dir in order to find the value
for previous version since its path will not always be relative to the
execution folder.
With this patch, MixState is always loaded after MixConfig. Since the
conversion of older formats of MixConfig account for format transfer,
changing the order does not prevent the format value to be properly
moved to the state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If publishing an update, also update the www/version/latest_version.
This is useful for various consumers of swupd data: swupd-client can
easily discover the latest version (and not only the latest version in
a given format); swupd-inspector and swupd-extract will be able to
automatically figure the latest version of a mix given the update URL.
If --no-publish is used, the file will not be updated like the other
related files. Clear Linux itself uses that so it can decide later on
whether to publish or not an update to the end users.
Related to #467. With latest_version is possible to peek at
www/<latest_version>/format and figure its format.
The bsdiff logger was shared globally between several goroutines and in
some cases, a goroutine would override the existing logger with a newly
created one.
This change centralizes the logger creation so that it is only created
once which prevents logger creation race conditions.
Signed-off-by: John Akre <john.w.akre@intel.com>
swupd-server used to support building bsdiff based deltas of manifest
files. This change adds that ability back for bundle manifests with
the 'mixer build delta-manifests' command.
Download failures can occur when downloading large sets of RPMs from a
server with a poor connection. This change adds retry functionality to
attempt RPM downloads multiple times. Also, the retries flag was added to
the following Mixer build commands.
retries:
Fixes: #531
Signed-off-by: John Akre <john.w.akre@intel.com>
Fixes#482
Since clients can't update over format bump boundaries anyways do not
attempt to create delta-packs covering that jump.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This commit fulfills enhancement #58. When building an image without providing a template,
if the default image template is not present in the mix directory, mixer will do the following:
- Copy image template from ister template "/usr/share/defaults/ister/release-image-config.json".
- Update image template with the bundle list from "mixbundles" file.
In case the "mixbundles" file is empty or not present, use ister template as is.
- Inform the user about the above steps.
- Continue the existing build process.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
One may wish to add or edit repos before building any mixes, but the DNF
(.yum-mix.conf) is not available until build-bundles is run.
Fixes#452
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The existence of local repo and local rpms folder has no impact if they
are not used in the mix. This patch removes the `--local-rpms` flag for
`mixer init` and make it always true. This way both folders are always
created and configured in builder.conf during init.
This patch also update the test cases for config convert to address this
change since.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Fixes#391
This patch greatly improves code readability by breaking up the goliath
builder.go into several new files containing similar functionality. Now
builder.go only contains the top-level build functions, constructors,
and types.
This patch contains no code functionality changes, the code is simply
moved around.
The builder.go file has been split out as follows:
* builder.go: contains all the top-level commands accessible via the
commandline, such as BuildBundles, BuildUpdate, BuildImage etc. along
with the type and const definitions.
* bundle_control.go: contains the mixer bundle * top-level commands and
all the helpers that are used only by this functionality.
* bundle_validate.go: contains the bundle validation functions.
* bundles.go: added getClosestAncestorOwner to this file as it was the
only file using this function (plus one reference in builder.go by the
top-level BuildBundles).
* deltapacks.go: contains the internal function createDeltaPacks used by
exported function in builder.go.
* repo_control.go: contains the dnf configuration controlling functions
such as AddRepo, SetURLRepo, etc. Ideally this will eventually be
moved to the config package.
* helpers.go: contains generic helper functions used by several files in
the builder package.
* init.go: contains the functions to initialize the mix via mixer
init.
* metadata.go: contains various metadata getters and setters for reading
and updating versions, formats, and metadata files.
* update.go: contains the internal functions used by mixer build update.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Fixes#323
Provide a better error message when listing repositories. The error now
gives a hint to the user to initialize their workspace before running
the command.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch changes how Warnings and Error messages are printed
throughout the code to use the `log` package instead of `fmt`. This
allows for finer control over what is printed and where it is printed
to.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When writing config files or buffers with configuration, mixer should
fail right away if an error occurs. This provides the user with better
errors since it prevents mixer from failing on config usage which
yields unrelated errors.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When building the upstream URL, if the b.UpstreamURL was set do a url
with a subpath included, the subpath was being discarded by
URL.ResolveReference(). In order to preserve the subpath in the base,
the provided subpath must be manually added to the base URL object.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When generating the manifest for bundles, use the user defined
--bundle-workers to determine the number of goroutines to use instead of
spawning one goroutine per bundle.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The error when mixer failed to download the upstream version was too
confusing since there were many layers of wrapped errors.
This patch removes the excessive wrapping and suggests a missing proxy
configuration to the user in case this error occurs.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running 'mixer init' with the --offline flag, there is no upstream
to fetch the default bundles from, so the mix should be initialized
without the default bundles.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This change moves the setting of mixer version string from source code
to a variable defined during compilation. This approach is preferable
because:
1- builder.go does not need to change just to change version though
functionally is the same.
2- simplifies logic in Makefile, no need to retrieve value from source.
This release adds more customizability of the DNF conf through the mixer
command line, adds a --clean option and does not wipe the outputdir by default
when re-running a 'build' command for the same build number, switches to zstd
compression for delta packs, and adds misc fixes to internal code structure.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This features may help users guarantee correctness of bundles built by mixer in
lieu of some shortcomings with dnf where dnf produces an unexpected dependency
solution. This mainly happens when a user is intending to fully substitute
a package from upstream's repo with one from their own. If anything is wrong
with the substitution, a feature like this would expose the issue early on.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
By default, build bundles removes image/ver and www/ver, so that builds
of the same number start with a clean slate. There are specific use cases
where this is not desirable, and the statedir already has a version folder
populated/mounted which shouldn't be wiped. This allows mixer to write
the full chroot into an actual mounted file system for example that is
attached to update/image/VER/full.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes several bugs dealing with running in a docker
container, and provides better warnings/error messages when the
environment and conf are not setup correctly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes issues with setting up the new mixer.state file. The
file now correctly transfers over the FORMAT version from the builder.conf,
falling back to the system /usr/share/defaults/swupd/format, and only then
defaulting to "1".
Deletes now persist in manifests over minversions. The only time they should
be removed is when performing a format bump.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release introduces major changes to the way Mixer is called and
run. Previously, mixer gained support for automatically building format
bumps, but was disabled by default (with --native=true always turned
on). The flag forces mixer to run natively on the system, and not
perform format bump builds automatically. With this release, the default
becomes --native=false, meaning all mixer 'build' commands will run in a
docker container. Running in a container ensures the proper version of
the tooling is run for specific build versions, because one version of
tooling cannot build for 2 different formats.
Mixer will automatically pull the appoproriate container that can create
builds for a given format, so the format bump process can truly be
automated and correct. It will still use a container for regular,
non-format bump builds for consistency.
Note that aside from the initial docker pull,
creating a mix will not be any slower than running natively, and allows
users to build versions that require tooling outside of their
host system version.
Another significant change involves the configuration for Mixer. The
builder.conf is now updated to be a stable, relatively unchanging
config to describe your workspace. The stateful items (versions, format,
etc) are moved into a mixer.state file which holds ephemeral values that
are changed build-to-build. This keeps all stateless and stateful items
centralized and follows a single format (TOML). The new format is used
by default, but can be overridden by setting --new-config=false during
the init phase. Both configs are versioned so mixer can easily detect
format changes and config errors.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The mixer.state file holds the temporary values used or produced between
mixes or mixes steps. The format variable was the only transient
variable in builder.conf so it was moved to this new state file.
This file should hold all transient values. All the temporary files
should be centralized on this new state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release updates mixer to build slim packs across minversions, reducing the
pack download size for clients and saving space on our infrastructure. Mixer
also properly avoids making a network connection except for commands that
absolutely need it.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release includes fixes to place the <bundle>-extra-files into a persistent
location, which will automatically inject listed files into a mix, reducing
manual intervention by mixer users when building a mix with unconventional
or files that are not part of any bundle by default. Bsdiff error output
is more standard and consistent, displaying the proper delta information to
make debugging failing cases easier.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Mixer will not work with source RPMs added and crash looking for files
that will not actually be provided. This patch removes them as it parses
the RPM list so only valid, usable RPMs get added to the repo.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch adds the --skip-fullfiles options for the 'build' subcommand
of mixer. This command will prevent fullfiles from being generated
during 'build update'. Likewise, the --skip-packs option prevents zero
packs form being generated.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
BuildUpdate API has become too convoluted, to the point where some of
its parameters are not even used anymore. This patch reorganize the
function API and simplifies parameter passing to underlying functions.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
William Douglas