This release fixes several bugs dealing with running in a docker
container, and provides better warnings/error messages when the
environment and conf are not setup correctly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes issues with setting up the new mixer.state file. The
file now correctly transfers over the FORMAT version from the builder.conf,
falling back to the system /usr/share/defaults/swupd/format, and only then
defaulting to "1".
Deletes now persist in manifests over minversions. The only time they should
be removed is when performing a format bump.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The new dnf release in Clear now does this by default, so we do not have to
load the nosync library anymore.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When config is created during mixer init, default values are set for
all variables. To mimic this behavior, LoadDefaults must be called on
conversion as well.
There is also no need to perform a conversion if the parsed version
number is the same as the current version number.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The old INI config does not have versioning, so when performing a
conversion, it needs to be set.
Also, since now the config format is identified on parsing, there is no
need to force Old Config when starting the conversion. This allows the
convert command to also convert between different TOML versions
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The default value for FORMAT is not always 1 anymore, so instead of
checking if the bumped format is 2, we need to check if it is 1 more
than the initial format.
Temp commit - DO NOT MERGE
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Instead of always using '1' as the default FORMAT value, check first if
the value was available in a legacy config. If not, try using the system
format instead. If both fail, fallback to the hardcoded value as it was
previously done.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Format bumps ensure clients will get the proper delete record before
continuing to the new format. Trim the records so our manifests stay a
reasonable size.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
It is important for deletes to persist over minversions because
minversions are not necessarily tied to format bumps (which force
clients to get the deletes before cleaning them up).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release introduces major changes to the way Mixer is called and
run. Previously, mixer gained support for automatically building format
bumps, but was disabled by default (with --native=true always turned
on). The flag forces mixer to run natively on the system, and not
perform format bump builds automatically. With this release, the default
becomes --native=false, meaning all mixer 'build' commands will run in a
docker container. Running in a container ensures the proper version of
the tooling is run for specific build versions, because one version of
tooling cannot build for 2 different formats.
Mixer will automatically pull the appoproriate container that can create
builds for a given format, so the format bump process can truly be
automated and correct. It will still use a container for regular,
non-format bump builds for consistency.
Note that aside from the initial docker pull,
creating a mix will not be any slower than running natively, and allows
users to build versions that require tooling outside of their
host system version.
Another significant change involves the configuration for Mixer. The
builder.conf is now updated to be a stable, relatively unchanging
config to describe your workspace. The stateful items (versions, format,
etc) are moved into a mixer.state file which holds ephemeral values that
are changed build-to-build. This keeps all stateless and stateful items
centralized and follows a single format (TOML). The new format is used
by default, but can be overridden by setting --new-config=false during
the init phase. Both configs are versioned so mixer can easily detect
format changes and config errors.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
With native false, mixer will run in a container by default, so update
the bat tests to run native since they already will run in a container.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Use TOML format as the default mixer format. The INI format can still be
used by setting the --new-config flag to false during init.
This change also makes TOML the default format for the bat tests.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Instead of relying on a user provided parameter, try to parse the
builder conf in all formats that mixer understands and assume the format
from there. Currently it can be "TOML with version", "TOML without
version" or "INI".
With this change, once the mix is initialized with 'mixer init', there
is no need to pass the --new-config flag around anymore.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch adds a new comment line on top of new config to specify the
current file format for that config.
With this change, new config files are expected to start with
\#VERSION X.X
where X are integer values.
This patch also change the encoding and decoding of toml to use the
io.Reader and io.Writer interfaces instead of a filename. This allows
the version to be read/write before the toml is processed.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The mixer.state file holds the temporary values used or produced between
mixes or mixes steps. The format variable was the only transient
variable in builder.conf so it was moved to this new state file.
This file should hold all transient values. All the temporary files
should be centralized on this new state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The test was only passing because err != nil for a reason other than the
expected reason. The test is no longer even relevant because format
changes are now required for automated format bump.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Solve a race issue by locking tmpManifests before appending to it.
Process the full manifest last in case the rsync fallback (bundle
chroots provided instead of bundle-info files) is used. Obviously all
this rsyncing must be done before handling the full chroot.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Parallelize the manifest initiation step which reads in all file
information from the full chroot.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The output for 'build update' cuts off after initial manifest processing
and leaves no status messages for a very long time. Put in a few
messages so users don't think it is hanging after it says all manifests
are done being processed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
MarkDeprecated constructs a deprecation message as follows:
<flag> has been deprecated, <message>
So message should not have leading capitalization.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release updates mixer to build slim packs across minversions, reducing the
pack download size for clients and saving space on our infrastructure. Mixer
also properly avoids making a network connection except for commands that
absolutely need it.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Add an additional fileContentInManifest check to pack creation to
exclude full files that are represented in the from manifest even if the
version changed in the to manifest. This allows mixer to exclude
unchanged files from delta packs over minversion bumps.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release includes fixes to place the <bundle>-extra-files into a persistent
location, which will automatically inject listed files into a mix, reducing
manual intervention by mixer users when building a mix with unconventional
or files that are not part of any bundle by default. Bsdiff error output
is more standard and consistent, displaying the proper delta information to
make debugging failing cases easier.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Mixer will not work with source RPMs added and crash looking for files
that will not actually be provided. This patch removes them as it parses
the RPM list so only valid, usable RPMs get added to the repo.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch adds temporary tests for the format bump feature. A permanent
test will require the generation of custom docker images with an updated
build of mixer. Until this tooling is available, this test provides a
minimum integrity check on the feature.
The added test checks the operation of a format bump on a real format
boundary. In addition, it checks the integrity of 4 key values:
- image/LasT_VER should point to the +20 version
- Previous version in +20 MoM should point to +0
- Previous version in +10 MoM should point to +0
- FORMAT property in builder.conf should have the +20 format
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Instead of reading the bundle-extra-files configuration file from the
bundle directory under /image/<ver> (which does not exist until mixer
build bundles is run) read from the persistent mixer workspace. This
means users no longer have to copy these files into place after building
bundles.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch adds the --skip-fullfiles options for the 'build' subcommand
of mixer. This command will prevent fullfiles from being generated
during 'build update'. Likewise, the --skip-packs option prevents zero
packs form being generated.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
BuildUpdate API has become too convoluted, to the point where some of
its parameters are not even used anymore. This patch reorganize the
function API and simplifies parameter passing to underlying functions.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Since we build +10 after we build +20 when doing a format bump, the last
built format will be the old format, so the new format must be manually
restored to builder.conf after +10 is built.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Since we build +20 before +10, if LAST_VER file is not returned to the
original state, the manifest for +10 will erroneously point to +20 as
its previous version.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Since we build +20 before +10, when finishing the +10 build the upstream
points to the old upstream version. This means that fetching the format
will get +10 format instead of the +20 format which will make
checkBumpNeeded() erroneously return true.
This patch fix a crash where format bump would loop forever until it hit
a format beyond the last existing one and crash.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Apparently there was an error during rebase of the format bump patch
that brought back BuildConf variable which is not supposed to exist
anymore. This reverts it back to the proper state.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When using reflection to iterate the config fields, trying to iterate on
unexported fields will cause mixer to crash. Also, when checking for
docker mounts, there is no need to check any of the unexported fields.
This patch fix this crash by skipping the search on unexported fields.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The only use case where a missing LAST_VER is a valid use case is when
checking if a bump is needed since the user might be running the build
for the first time. In all other cases, the caller expects a valid
version to be returned is error is nil. This caused mixer to print some
cryptic errors if those other callers were used on a new workspace.
This patch changes the default behavior to always return an error if
LAST_VER is missing and let the caller decide using os.IsNotExist() api
if the error should be propagated or not.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The SetProperty API is used to search for the property withing config,
set it and save the config. In this code, the property is set directly
just a few lines above, so saving is already enough to propagate it to
the config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release sets a timeout for delta creation time to 1 minute, in an effort
to skip deltas which typically end up being too large to use for packs anyway.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Tudor Marcu