For a clear linux mix, the os-release file will be updated as follows:
- NAME="Clear Linux Mix"
Eventually this value should be read from user input when #424 is implemented.
- ID=<based on above NAME field>
E.g. ID=clear-linux-mix
- VERSION="<mix-ver> (<upstream-ver>)"
E.g. VERSION="10 (24750)"
- PRETTY_NAME="<NAME> <mix-ver> (<upstream-ver)"
E.g. PRETTY_NAME="Clear Linux Mix 10 (24750)"
- VERSION_ID=<mix-ver>
E.g. VERSION_ID=10
- BUILD_ID=<upstream-ver>
E.g. BUILD_ID=24750
- MIX_UPSTREAM_URL="<upstream-url>"
This is a new field in os-release.
E.g. MIX_UPSTREAM_URL="https://download.clearlinux.org"
These automated tests will check if conversion between multiple older
versions to the current version is working properly.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
With the automatic conversion on load time, mixer ill always use the new
config format. Even if the mix is initialized with the legacy config, it
will be converted right away. As such, there is no need to allow the
creation of legacy configs anymore nor to check if it is being used.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch introduces automatic conversion of config files to the latest
format. This will seamlessly transition all users to the latest format
without the need for manually converting the config with the `mixer
config convert` command.
This patch also introduces the config_conver.go file, which centralizes
all conversion operations and also the parsing for the legacy config in
order to keep config.go clean and focused on current config features.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
All the values inside buildBundlesConfig are already parsed and
available in the code through builder.Config, so there is no need to
parse them again from the config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release fixes many bugs in the tooling:
- Fix double parsing of the config file
- Do not add default bundles when 'init' called with offline
- Reduce download timeout to 10 seconds instead of 30 on file download
- Improve download upstream error message
- Enforce UTF-8 locale when running DNF commands in mixer
- Ignore possible %ghost files in RPMs
- Fix global --offline usage to completely block all outbound network
checks on all commands, even when running in container mode
- Fix error in concurrent manifest creation code to properly exit on
first error hit
- Use --bundle-workers for manifest generation like other commands
- Fix to use passed in minversion for +10 format bump build
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When generating the manifest for bundles, use the user defined
--bundle-workers to determine the number of goroutines to use instead of
spawning one goroutine per bundle.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When there is an error in one of the worker goroutines, the error is
read but before it is returned, the variable is reused inside the
following for loop to check the error of addFilesFromChroot(). Since we
always have a "full" bundle, it is always overwritten.
Another issue is that the err variable was shared among the goroutines
and was set directly instead of through the error channel. This would
allow one routine to overwrite the error from another.
Lastly, there is a problem with the loop that feeds bundleChan with
data. The second select statement calls 'break' when an error is found,
but 'break' will exit the 'select' block, not the 'for' loop. In order to
stop the loop, we move the break to an if statement that checks for a
condition set on error. This change ensures that the worker goroutines
stop once an error is found.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When --offline is passed, any command executed should not reach over the
network. This introduces a tricky scenario when it is passed to a build
command with --native=false (default), because the command in the
container will be run with --offline, but the native mixer binary will
reach over the network to pull/update docker containers before executing
in them. This patch also makes that process offline only, attempting to
use a cached docker image so that it can run offline fully, but exits if
none exist.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Due to the way that we use repoquery, we are returned files that
may be %ghost files in RPMs. These files may be missing and this
is legal and compliant.
While Clear Linux OS does not use %ghost files in any of the RPMs
created, there is nothing wrong with permitting other vendor RPM
files that may list %ghost files, and mixer should just ignore
these files.
This change does just that: If repoquery returns files that are
not existing in the fullimage, prints a warning and continues with
out dropping the manifest for that bundle.
In some situations dnf may try to combine files from different
encodings. If the command is not running is a UTF-8 locale, this
combination will be missing a new line, causing and error.
An example of this occurrence is the command `dnf repoquery --quiet -l
systemd ca-certs-static ...` which combines a file list from systemd and
ca-certs-static
This patch enforces UTF-8 locale on every dnf execution.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The error when mixer failed to download the upstream version was too
confusing since there were many layers of wrapped errors.
This patch removes the excessive wrapping and suggests a missing proxy
configuration to the user in case this error occurs.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running 'mixer init' with the --offline flag, there is no upstream
to fetch the default bundles from, so the mix should be initialized
without the default bundles.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running any 'mixer config' sub-command, the config will be parsed
in the command itself, so there is no need to pre-parse them in the pre
execution. Those commands also don't rely on network nor cause any
format conflict, so it is safe to skip all the remaining checks like it
is done for the 'init' command.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
We were assigning a map to a map to assign it to a string slice. Instead
just assign the map keys to the string slice directly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When ranging over the syncmap to add packages to a bundle we were
mistakenly ranging over the entire set of bundles and adding all bundle
packages to the same bundle (nondeterministically, since we were doing
concurrent writes to a syncmap). In reality we just need to range over
the repoPkgMap itself and not load it from the syncmap at all. Do this
operation before even storing it.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The default runtime in upstream Clear will be set to the kata runtime,
which does not support host networking. This is needed for mixer, so we
must force --runtime=runc during all docker runs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes a data race which cause concurrent map read/writes
when resolving packages and bundles. Versioning is also moved into the
Makefile so it does not have to be updated in the source code anymore,
and added at compile time.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Two issues, first is resolved by renaming the error variable so a
goroutine isn't sharing err from outer scope with all other goroutines.
The second issue is resolved by using a sync.Map to safely read and
write to a map via the Range function.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This change moves the setting of mixer version string from source code
to a variable defined during compilation. This approach is preferable
because:
1- builder.go does not need to change just to change version though
functionally is the same.
2- simplifies logic in Makefile, no need to retrieve value from source.
This release adds more customizability of the DNF conf through the mixer
command line, adds a --clean option and does not wipe the outputdir by default
when re-running a 'build' command for the same build number, switches to zstd
compression for delta packs, and adds misc fixes to internal code structure.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This features may help users guarantee correctness of bundles built by mixer in
lieu of some shortcomings with dnf where dnf produces an unexpected dependency
solution. This mainly happens when a user is intending to fully substitute
a package from upstream's repo with one from their own. If anything is wrong
with the substitution, a feature like this would expose the issue early on.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
By default, build bundles removes image/ver and www/ver, so that builds
of the same number start with a clean slate. There are specific use cases
where this is not desirable, and the statedir already has a version folder
populated/mounted which shouldn't be wiped. This allows mixer to write
the full chroot into an actual mounted file system for example that is
attached to update/image/VER/full.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Export the GetBundleInfo file so other packages can use it. Simplify
function signature so it doesn't need a swupd config file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Other packages want to use the IndexBundle const to skip processing this
manifest for some validation cases.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes several bugs dealing with running in a docker
container, and provides better warnings/error messages when the
environment and conf are not setup correctly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
If the docker image for a given format is updated, users with cached
version of the image may keep running in the older version.
This patch makes sure mixer tries to pull an updated version before
running. If the pull fails, mixer will print a warning instead of
failing to allow users with cached images to keep building offline.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If DOCKER_IMAGE_PATH is not set, the docker command will be malformed
and the user will receive a cryptic message.
This patch makes sure mixer fails early and provides the user an
informative error.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The default value for DOCKER_IMAGE_PATH was only set for the old INI
config. As a result, TOML configs would have an empty path that would
generate an invalid docker command to be issued.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release fixes issues with setting up the new mixer.state file. The
file now correctly transfers over the FORMAT version from the builder.conf,
falling back to the system /usr/share/defaults/swupd/format, and only then
defaulting to "1".
Deletes now persist in manifests over minversions. The only time they should
be removed is when performing a format bump.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The new dnf release in Clear now does this by default, so we do not have to
load the nosync library anymore.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When config is created during mixer init, default values are set for
all variables. To mimic this behavior, LoadDefaults must be called on
conversion as well.
There is also no need to perform a conversion if the parsed version
number is the same as the current version number.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The old INI config does not have versioning, so when performing a
conversion, it needs to be set.
Also, since now the config format is identified on parsing, there is no
need to force Old Config when starting the conversion. This allows the
convert command to also convert between different TOML versions
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Rodrigo Chiossi