The default runtime in upstream Clear will be set to the kata runtime,
which does not support host networking. This is needed for mixer, so we
must force --runtime=runc during all docker runs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes a data race which cause concurrent map read/writes
when resolving packages and bundles. Versioning is also moved into the
Makefile so it does not have to be updated in the source code anymore,
and added at compile time.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Two issues, first is resolved by renaming the error variable so a
goroutine isn't sharing err from outer scope with all other goroutines.
The second issue is resolved by using a sync.Map to safely read and
write to a map via the Range function.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This change moves the setting of mixer version string from source code
to a variable defined during compilation. This approach is preferable
because:
1- builder.go does not need to change just to change version though
functionally is the same.
2- simplifies logic in Makefile, no need to retrieve value from source.
This release adds more customizability of the DNF conf through the mixer
command line, adds a --clean option and does not wipe the outputdir by default
when re-running a 'build' command for the same build number, switches to zstd
compression for delta packs, and adds misc fixes to internal code structure.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This features may help users guarantee correctness of bundles built by mixer in
lieu of some shortcomings with dnf where dnf produces an unexpected dependency
solution. This mainly happens when a user is intending to fully substitute
a package from upstream's repo with one from their own. If anything is wrong
with the substitution, a feature like this would expose the issue early on.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
By default, build bundles removes image/ver and www/ver, so that builds
of the same number start with a clean slate. There are specific use cases
where this is not desirable, and the statedir already has a version folder
populated/mounted which shouldn't be wiped. This allows mixer to write
the full chroot into an actual mounted file system for example that is
attached to update/image/VER/full.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Export the GetBundleInfo file so other packages can use it. Simplify
function signature so it doesn't need a swupd config file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Other packages want to use the IndexBundle const to skip processing this
manifest for some validation cases.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes several bugs dealing with running in a docker
container, and provides better warnings/error messages when the
environment and conf are not setup correctly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
If the docker image for a given format is updated, users with cached
version of the image may keep running in the older version.
This patch makes sure mixer tries to pull an updated version before
running. If the pull fails, mixer will print a warning instead of
failing to allow users with cached images to keep building offline.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If DOCKER_IMAGE_PATH is not set, the docker command will be malformed
and the user will receive a cryptic message.
This patch makes sure mixer fails early and provides the user an
informative error.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The default value for DOCKER_IMAGE_PATH was only set for the old INI
config. As a result, TOML configs would have an empty path that would
generate an invalid docker command to be issued.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release fixes issues with setting up the new mixer.state file. The
file now correctly transfers over the FORMAT version from the builder.conf,
falling back to the system /usr/share/defaults/swupd/format, and only then
defaulting to "1".
Deletes now persist in manifests over minversions. The only time they should
be removed is when performing a format bump.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The new dnf release in Clear now does this by default, so we do not have to
load the nosync library anymore.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When config is created during mixer init, default values are set for
all variables. To mimic this behavior, LoadDefaults must be called on
conversion as well.
There is also no need to perform a conversion if the parsed version
number is the same as the current version number.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The old INI config does not have versioning, so when performing a
conversion, it needs to be set.
Also, since now the config format is identified on parsing, there is no
need to force Old Config when starting the conversion. This allows the
convert command to also convert between different TOML versions
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The default value for FORMAT is not always 1 anymore, so instead of
checking if the bumped format is 2, we need to check if it is 1 more
than the initial format.
Temp commit - DO NOT MERGE
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Instead of always using '1' as the default FORMAT value, check first if
the value was available in a legacy config. If not, try using the system
format instead. If both fail, fallback to the hardcoded value as it was
previously done.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Format bumps ensure clients will get the proper delete record before
continuing to the new format. Trim the records so our manifests stay a
reasonable size.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
It is important for deletes to persist over minversions because
minversions are not necessarily tied to format bumps (which force
clients to get the deletes before cleaning them up).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release introduces major changes to the way Mixer is called and
run. Previously, mixer gained support for automatically building format
bumps, but was disabled by default (with --native=true always turned
on). The flag forces mixer to run natively on the system, and not
perform format bump builds automatically. With this release, the default
becomes --native=false, meaning all mixer 'build' commands will run in a
docker container. Running in a container ensures the proper version of
the tooling is run for specific build versions, because one version of
tooling cannot build for 2 different formats.
Mixer will automatically pull the appoproriate container that can create
builds for a given format, so the format bump process can truly be
automated and correct. It will still use a container for regular,
non-format bump builds for consistency.
Note that aside from the initial docker pull,
creating a mix will not be any slower than running natively, and allows
users to build versions that require tooling outside of their
host system version.
Another significant change involves the configuration for Mixer. The
builder.conf is now updated to be a stable, relatively unchanging
config to describe your workspace. The stateful items (versions, format,
etc) are moved into a mixer.state file which holds ephemeral values that
are changed build-to-build. This keeps all stateless and stateful items
centralized and follows a single format (TOML). The new format is used
by default, but can be overridden by setting --new-config=false during
the init phase. Both configs are versioned so mixer can easily detect
format changes and config errors.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
With native false, mixer will run in a container by default, so update
the bat tests to run native since they already will run in a container.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Use TOML format as the default mixer format. The INI format can still be
used by setting the --new-config flag to false during init.
This change also makes TOML the default format for the bat tests.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Instead of relying on a user provided parameter, try to parse the
builder conf in all formats that mixer understands and assume the format
from there. Currently it can be "TOML with version", "TOML without
version" or "INI".
With this change, once the mix is initialized with 'mixer init', there
is no need to pass the --new-config flag around anymore.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch adds a new comment line on top of new config to specify the
current file format for that config.
With this change, new config files are expected to start with
\#VERSION X.X
where X are integer values.
This patch also change the encoding and decoding of toml to use the
io.Reader and io.Writer interfaces instead of a filename. This allows
the version to be read/write before the toml is processed.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The mixer.state file holds the temporary values used or produced between
mixes or mixes steps. The format variable was the only transient
variable in builder.conf so it was moved to this new state file.
This file should hold all transient values. All the temporary files
should be centralized on this new state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The test was only passing because err != nil for a reason other than the
expected reason. The test is no longer even relevant because format
changes are now required for automated format bump.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Solve a race issue by locking tmpManifests before appending to it.
Process the full manifest last in case the rsync fallback (bundle
chroots provided instead of bundle-info files) is used. Obviously all
this rsyncing must be done before handling the full chroot.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Parallelize the manifest initiation step which reads in all file
information from the full chroot.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The output for 'build update' cuts off after initial manifest processing
and leaves no status messages for a very long time. Put in a few
messages so users don't think it is hanging after it says all manifests
are done being processed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
MarkDeprecated constructs a deprecation message as follows:
<flag> has been deprecated, <message>
So message should not have leading capitalization.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release updates mixer to build slim packs across minversions, reducing the
pack download size for clients and saving space on our infrastructure. Mixer
also properly avoids making a network connection except for commands that
absolutely need it.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Tudor Marcu