From format 27 onwards, Mixer will recognize bundles with "[Status] : Experimental"
as experimental bundles and mark their status flag as "e" in the Manifest.MoM.
i.e. These bundles will include an "e" in the 2nd position of the manifest flags,
as shown here:
Me.. 4b91c3122e7e32f1e3edb597c6f89ff32cfffd977afb5e5 10 some-bundle
Fixes#497
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
When creating manifests for a new build, also create to-Manifests
for each bundle that has changes in this version.
The goal on publishing to-Manifests is to reduce the download size
on updates, according to what was discussed on
clearlinux/swupd-client#498
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
`bundle add` will now add valid bundles and skip only the invalid ones,
thereby fixing issue #489.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Some testcases showed these checks were incomplete and not handling
corner cases. Improve the logic of these checks to comply with the new
tests.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of requiring upstream bundles be read from a child directory
under where mixer is currently run read them from underneath the mixer
workspace (builder.Config.Mixer.VersionPath) the same way local-bundles
is handled.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Start to add some unit tests for the bundle_control file. There is much
work to still be done as far as testing goes for these methods.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
With the removal of the editor feature in `bundle edit`,
the term `edit` is not relevant anymore. Hence, changing it to `create`.
In order to prevent breakage of existing use-cases,
`edit` is aliased to `create`and `--suppress-editor` is made hidden and deprecated.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
`bundle edit` will no longer open the editor. It will just create new bundles or copy existing bundles.
This command will locate the bundle by first looking in local-bundles, and then in upstream-bundles.
If the bundle is only found upstream, the bundle file will be copied to the local-bundles directory.
If the bundle is not found anywhere, a blank template will be created with the correct name.
This commit also fixes issue #448 and increases test coverage for `bundle validate`.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Updating mixver should not require a network connection, nor reach out
to the server to find out any more information than is needed.
Fixes#282
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fixes#482
Since clients can't update over format bump boundaries anyways do not
attempt to create delta-packs covering that jump.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This commit fulfills enhancement #58. When building an image without providing a template,
if the default image template is not present in the mix directory, mixer will do the following:
- Copy image template from ister template "/usr/share/defaults/ister/release-image-config.json".
- Update image template with the bundle list from "mixbundles" file.
In case the "mixbundles" file is empty or not present, use ister template as is.
- Inform the user about the above steps.
- Continue the existing build process.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
DNF creates two additional state directories in the full chroot if they
missing: /etc/dnf/ and /etc/dnf/modules.d.
Update the DNF state path slice so that the /etc/dnf directory tree is
removed from the full chroot after full chroot creation. This avoids
adding those two directories to Manifest.full.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
One may wish to add or edit repos before building any mixes, but the DNF
(.yum-mix.conf) is not available until build-bundles is run.
Fixes#452
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Currently, as part of `bundle list`, all the bundle names are read
and validated. As a result, `bundle list` fails if an invalid
bundle exists even if it is not added to the mix. This commit removes
bundle name validation as part of `bundle list` thereby fixing #461.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
The existence of local repo and local rpms folder has no impact if they
are not used in the mix. This patch removes the `--local-rpms` flag for
`mixer init` and make it always true. This way both folders are always
created and configured in builder.conf during init.
This patch also update the test cases for config convert to address this
change since.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Fixes#391
This patch greatly improves code readability by breaking up the goliath
builder.go into several new files containing similar functionality. Now
builder.go only contains the top-level build functions, constructors,
and types.
This patch contains no code functionality changes, the code is simply
moved around.
The builder.go file has been split out as follows:
* builder.go: contains all the top-level commands accessible via the
commandline, such as BuildBundles, BuildUpdate, BuildImage etc. along
with the type and const definitions.
* bundle_control.go: contains the mixer bundle * top-level commands and
all the helpers that are used only by this functionality.
* bundle_validate.go: contains the bundle validation functions.
* bundles.go: added getClosestAncestorOwner to this file as it was the
only file using this function (plus one reference in builder.go by the
top-level BuildBundles).
* deltapacks.go: contains the internal function createDeltaPacks used by
exported function in builder.go.
* repo_control.go: contains the dnf configuration controlling functions
such as AddRepo, SetURLRepo, etc. Ideally this will eventually be
moved to the config package.
* helpers.go: contains generic helper functions used by several files in
the builder package.
* init.go: contains the functions to initialize the mix via mixer
init.
* metadata.go: contains various metadata getters and setters for reading
and updating versions, formats, and metadata files.
* update.go: contains the internal functions used by mixer build update.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Fixes#323
Provide a better error message when listing repositories. The error now
gives a hint to the user to initialize their workspace before running
the command.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch changes how Warnings and Error messages are printed
throughout the code to use the `log` package instead of `fmt`. This
allows for finer control over what is printed and where it is printed
to.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Use the logger interface to write stopwatch output. With this change,
any error that occurs while trying to write the output is handled
internally by the log module.
This patch also fix a bug where the line break was printed to STDOUT
instead of the stopwatch buffer.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When writing config files or buffers with configuration, mixer should
fail right away if an error occurs. This provides the user with better
errors since it prevents mixer from failing on config usage which
yields unrelated errors.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Make package file parsing ignore '#' and anything after that until the
end of the line. Also trim spaces. This makes the comment syntax the
same as bundle files.
At this point, one could consider trying to merge both parsing
functions, but given how simple the rules for this one are, kept them
separate. We can revisit this later.
When building the upstream URL, if the b.UpstreamURL was set do a url
with a subpath included, the subpath was being discarded by
URL.ResolveReference(). In order to preserve the subpath in the base,
the provided subpath must be manually added to the base URL object.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
For a clear linux mix, the os-release file will be updated as follows:
- NAME="Clear Linux Mix"
Eventually this value should be read from user input when #424 is implemented.
- ID=<based on above NAME field>
E.g. ID=clear-linux-mix
- VERSION="<mix-ver> (<upstream-ver>)"
E.g. VERSION="10 (24750)"
- PRETTY_NAME="<NAME> <mix-ver> (<upstream-ver)"
E.g. PRETTY_NAME="Clear Linux Mix 10 (24750)"
- VERSION_ID=<mix-ver>
E.g. VERSION_ID=10
- BUILD_ID=<upstream-ver>
E.g. BUILD_ID=24750
- MIX_UPSTREAM_URL="<upstream-url>"
This is a new field in os-release.
E.g. MIX_UPSTREAM_URL="https://download.clearlinux.org"
All the values inside buildBundlesConfig are already parsed and
available in the code through builder.Config, so there is no need to
parse them again from the config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When generating the manifest for bundles, use the user defined
--bundle-workers to determine the number of goroutines to use instead of
spawning one goroutine per bundle.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When --offline is passed, any command executed should not reach over the
network. This introduces a tricky scenario when it is passed to a build
command with --native=false (default), because the command in the
container will be run with --offline, but the native mixer binary will
reach over the network to pull/update docker containers before executing
in them. This patch also makes that process offline only, attempting to
use a cached docker image so that it can run offline fully, but exits if
none exist.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In some situations dnf may try to combine files from different
encodings. If the command is not running is a UTF-8 locale, this
combination will be missing a new line, causing and error.
An example of this occurrence is the command `dnf repoquery --quiet -l
systemd ca-certs-static ...` which combines a file list from systemd and
ca-certs-static
This patch enforces UTF-8 locale on every dnf execution.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The error when mixer failed to download the upstream version was too
confusing since there were many layers of wrapped errors.
This patch removes the excessive wrapping and suggests a missing proxy
configuration to the user in case this error occurs.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running 'mixer init' with the --offline flag, there is no upstream
to fetch the default bundles from, so the mix should be initialized
without the default bundles.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
We were assigning a map to a map to assign it to a string slice. Instead
just assign the map keys to the string slice directly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When ranging over the syncmap to add packages to a bundle we were
mistakenly ranging over the entire set of bundles and adding all bundle
packages to the same bundle (nondeterministically, since we were doing
concurrent writes to a syncmap). In reality we just need to range over
the repoPkgMap itself and not load it from the syncmap at all. Do this
operation before even storing it.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The default runtime in upstream Clear will be set to the kata runtime,
which does not support host networking. This is needed for mixer, so we
must force --runtime=runc during all docker runs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Two issues, first is resolved by renaming the error variable so a
goroutine isn't sharing err from outer scope with all other goroutines.
The second issue is resolved by using a sync.Map to safely read and
write to a map via the Range function.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This change moves the setting of mixer version string from source code
to a variable defined during compilation. This approach is preferable
because:
1- builder.go does not need to change just to change version though
functionally is the same.
2- simplifies logic in Makefile, no need to retrieve value from source.
This release adds more customizability of the DNF conf through the mixer
command line, adds a --clean option and does not wipe the outputdir by default
when re-running a 'build' command for the same build number, switches to zstd
compression for delta packs, and adds misc fixes to internal code structure.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This features may help users guarantee correctness of bundles built by mixer in
lieu of some shortcomings with dnf where dnf produces an unexpected dependency
solution. This mainly happens when a user is intending to fully substitute
a package from upstream's repo with one from their own. If anything is wrong
with the substitution, a feature like this would expose the issue early on.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
By default, build bundles removes image/ver and www/ver, so that builds
of the same number start with a clean slate. There are specific use cases
where this is not desirable, and the statedir already has a version folder
populated/mounted which shouldn't be wiped. This allows mixer to write
the full chroot into an actual mounted file system for example that is
attached to update/image/VER/full.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes several bugs dealing with running in a docker
container, and provides better warnings/error messages when the
environment and conf are not setup correctly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
If the docker image for a given format is updated, users with cached
version of the image may keep running in the older version.
This patch makes sure mixer tries to pull an updated version before
running. If the pull fails, mixer will print a warning instead of
failing to allow users with cached images to keep building offline.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If DOCKER_IMAGE_PATH is not set, the docker command will be malformed
and the user will receive a cryptic message.
This patch makes sure mixer fails early and provides the user an
informative error.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release fixes issues with setting up the new mixer.state file. The
file now correctly transfers over the FORMAT version from the builder.conf,
falling back to the system /usr/share/defaults/swupd/format, and only then
defaulting to "1".
Deletes now persist in manifests over minversions. The only time they should
be removed is when performing a format bump.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release introduces major changes to the way Mixer is called and
run. Previously, mixer gained support for automatically building format
bumps, but was disabled by default (with --native=true always turned
on). The flag forces mixer to run natively on the system, and not
perform format bump builds automatically. With this release, the default
becomes --native=false, meaning all mixer 'build' commands will run in a
docker container. Running in a container ensures the proper version of
the tooling is run for specific build versions, because one version of
tooling cannot build for 2 different formats.
Mixer will automatically pull the appoproriate container that can create
builds for a given format, so the format bump process can truly be
automated and correct. It will still use a container for regular,
non-format bump builds for consistency.
Note that aside from the initial docker pull,
creating a mix will not be any slower than running natively, and allows
users to build versions that require tooling outside of their
host system version.
Another significant change involves the configuration for Mixer. The
builder.conf is now updated to be a stable, relatively unchanging
config to describe your workspace. The stateful items (versions, format,
etc) are moved into a mixer.state file which holds ephemeral values that
are changed build-to-build. This keeps all stateless and stateful items
centralized and follows a single format (TOML). The new format is used
by default, but can be overridden by setting --new-config=false during
the init phase. Both configs are versioned so mixer can easily detect
format changes and config errors.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The mixer.state file holds the temporary values used or produced between
mixes or mixes steps. The format variable was the only transient
variable in builder.conf so it was moved to this new state file.
This file should hold all transient values. All the temporary files
should be centralized on this new state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Reagan Lopez