When building the upstream URL, if the b.UpstreamURL was set do a url
with a subpath included, the subpath was being discarded by
URL.ResolveReference(). In order to preserve the subpath in the base,
the provided subpath must be manually added to the base URL object.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
For a clear linux mix, the os-release file will be updated as follows:
- NAME="Clear Linux Mix"
Eventually this value should be read from user input when #424 is implemented.
- ID=<based on above NAME field>
E.g. ID=clear-linux-mix
- VERSION="<mix-ver> (<upstream-ver>)"
E.g. VERSION="10 (24750)"
- PRETTY_NAME="<NAME> <mix-ver> (<upstream-ver)"
E.g. PRETTY_NAME="Clear Linux Mix 10 (24750)"
- VERSION_ID=<mix-ver>
E.g. VERSION_ID=10
- BUILD_ID=<upstream-ver>
E.g. BUILD_ID=24750
- MIX_UPSTREAM_URL="<upstream-url>"
This is a new field in os-release.
E.g. MIX_UPSTREAM_URL="https://download.clearlinux.org"
All the values inside buildBundlesConfig are already parsed and
available in the code through builder.Config, so there is no need to
parse them again from the config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When generating the manifest for bundles, use the user defined
--bundle-workers to determine the number of goroutines to use instead of
spawning one goroutine per bundle.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When --offline is passed, any command executed should not reach over the
network. This introduces a tricky scenario when it is passed to a build
command with --native=false (default), because the command in the
container will be run with --offline, but the native mixer binary will
reach over the network to pull/update docker containers before executing
in them. This patch also makes that process offline only, attempting to
use a cached docker image so that it can run offline fully, but exits if
none exist.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In some situations dnf may try to combine files from different
encodings. If the command is not running is a UTF-8 locale, this
combination will be missing a new line, causing and error.
An example of this occurrence is the command `dnf repoquery --quiet -l
systemd ca-certs-static ...` which combines a file list from systemd and
ca-certs-static
This patch enforces UTF-8 locale on every dnf execution.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The error when mixer failed to download the upstream version was too
confusing since there were many layers of wrapped errors.
This patch removes the excessive wrapping and suggests a missing proxy
configuration to the user in case this error occurs.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running 'mixer init' with the --offline flag, there is no upstream
to fetch the default bundles from, so the mix should be initialized
without the default bundles.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
We were assigning a map to a map to assign it to a string slice. Instead
just assign the map keys to the string slice directly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When ranging over the syncmap to add packages to a bundle we were
mistakenly ranging over the entire set of bundles and adding all bundle
packages to the same bundle (nondeterministically, since we were doing
concurrent writes to a syncmap). In reality we just need to range over
the repoPkgMap itself and not load it from the syncmap at all. Do this
operation before even storing it.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The default runtime in upstream Clear will be set to the kata runtime,
which does not support host networking. This is needed for mixer, so we
must force --runtime=runc during all docker runs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Two issues, first is resolved by renaming the error variable so a
goroutine isn't sharing err from outer scope with all other goroutines.
The second issue is resolved by using a sync.Map to safely read and
write to a map via the Range function.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This change moves the setting of mixer version string from source code
to a variable defined during compilation. This approach is preferable
because:
1- builder.go does not need to change just to change version though
functionally is the same.
2- simplifies logic in Makefile, no need to retrieve value from source.
This release adds more customizability of the DNF conf through the mixer
command line, adds a --clean option and does not wipe the outputdir by default
when re-running a 'build' command for the same build number, switches to zstd
compression for delta packs, and adds misc fixes to internal code structure.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This features may help users guarantee correctness of bundles built by mixer in
lieu of some shortcomings with dnf where dnf produces an unexpected dependency
solution. This mainly happens when a user is intending to fully substitute
a package from upstream's repo with one from their own. If anything is wrong
with the substitution, a feature like this would expose the issue early on.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
By default, build bundles removes image/ver and www/ver, so that builds
of the same number start with a clean slate. There are specific use cases
where this is not desirable, and the statedir already has a version folder
populated/mounted which shouldn't be wiped. This allows mixer to write
the full chroot into an actual mounted file system for example that is
attached to update/image/VER/full.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes several bugs dealing with running in a docker
container, and provides better warnings/error messages when the
environment and conf are not setup correctly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
If the docker image for a given format is updated, users with cached
version of the image may keep running in the older version.
This patch makes sure mixer tries to pull an updated version before
running. If the pull fails, mixer will print a warning instead of
failing to allow users with cached images to keep building offline.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If DOCKER_IMAGE_PATH is not set, the docker command will be malformed
and the user will receive a cryptic message.
This patch makes sure mixer fails early and provides the user an
informative error.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release fixes issues with setting up the new mixer.state file. The
file now correctly transfers over the FORMAT version from the builder.conf,
falling back to the system /usr/share/defaults/swupd/format, and only then
defaulting to "1".
Deletes now persist in manifests over minversions. The only time they should
be removed is when performing a format bump.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release introduces major changes to the way Mixer is called and
run. Previously, mixer gained support for automatically building format
bumps, but was disabled by default (with --native=true always turned
on). The flag forces mixer to run natively on the system, and not
perform format bump builds automatically. With this release, the default
becomes --native=false, meaning all mixer 'build' commands will run in a
docker container. Running in a container ensures the proper version of
the tooling is run for specific build versions, because one version of
tooling cannot build for 2 different formats.
Mixer will automatically pull the appoproriate container that can create
builds for a given format, so the format bump process can truly be
automated and correct. It will still use a container for regular,
non-format bump builds for consistency.
Note that aside from the initial docker pull,
creating a mix will not be any slower than running natively, and allows
users to build versions that require tooling outside of their
host system version.
Another significant change involves the configuration for Mixer. The
builder.conf is now updated to be a stable, relatively unchanging
config to describe your workspace. The stateful items (versions, format,
etc) are moved into a mixer.state file which holds ephemeral values that
are changed build-to-build. This keeps all stateless and stateful items
centralized and follows a single format (TOML). The new format is used
by default, but can be overridden by setting --new-config=false during
the init phase. Both configs are versioned so mixer can easily detect
format changes and config errors.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The mixer.state file holds the temporary values used or produced between
mixes or mixes steps. The format variable was the only transient
variable in builder.conf so it was moved to this new state file.
This file should hold all transient values. All the temporary files
should be centralized on this new state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release updates mixer to build slim packs across minversions, reducing the
pack download size for clients and saving space on our infrastructure. Mixer
also properly avoids making a network connection except for commands that
absolutely need it.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release includes fixes to place the <bundle>-extra-files into a persistent
location, which will automatically inject listed files into a mix, reducing
manual intervention by mixer users when building a mix with unconventional
or files that are not part of any bundle by default. Bsdiff error output
is more standard and consistent, displaying the proper delta information to
make debugging failing cases easier.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Mixer will not work with source RPMs added and crash looking for files
that will not actually be provided. This patch removes them as it parses
the RPM list so only valid, usable RPMs get added to the repo.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch adds the --skip-fullfiles options for the 'build' subcommand
of mixer. This command will prevent fullfiles from being generated
during 'build update'. Likewise, the --skip-packs option prevents zero
packs form being generated.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
BuildUpdate API has become too convoluted, to the point where some of
its parameters are not even used anymore. This patch reorganize the
function API and simplifies parameter passing to underlying functions.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Apparently there was an error during rebase of the format bump patch
that brought back BuildConf variable which is not supposed to exist
anymore. This reverts it back to the proper state.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When using reflection to iterate the config fields, trying to iterate on
unexported fields will cause mixer to crash. Also, when checking for
docker mounts, there is no need to check any of the unexported fields.
This patch fix this crash by skipping the search on unexported fields.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The only use case where a missing LAST_VER is a valid use case is when
checking if a bump is needed since the user might be running the build
for the first time. In all other cases, the caller expects a valid
version to be returned is error is nil. This caused mixer to print some
cryptic errors if those other callers were used on a new workspace.
This patch changes the default behavior to always return an error if
LAST_VER is missing and let the caller decide using os.IsNotExist() api
if the error should be propagated or not.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The SetProperty API is used to search for the property withing config,
set it and save the config. In this code, the property is set directly
just a few lines above, so saving is already enough to propagate it to
the config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release sets a timeout for delta creation time to 1 minute, in an effort
to skip deltas which typically end up being too large to use for packs anyway.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release sets a timeout for delta creation time to 1 minute, in an effort
to skip deltas which typically end up being too large to use for packs anyway.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Set timeout to 1 minute for bsdiff. The majority of all delta creations
take significantly less than 1 minute to run, which signifies that bsdiff
is working on a delta that may be very large, or very difficult to diff. In
all the cases where bsdiff took multiple minutes to finish, the delta ended
up not being used because it was larger than the compressed fullfile. This
attempts to skip those cases and improve delta creation time.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes mixer to not exit when upstreamurl is not found, and
to attempt to use the given format in the builder.conf. For some cases,
the mixer user may in fact be "upstream" so it does not make sense to check
upstream format compatibility.
A new tool is available called swupd-inspector, which contains various
commands that help read data of a swupd repository: download the fullfile for
a specific file, see the Manifest for a specific version, see the differences
between two versions.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This is the case for when an OSV *is upstream* and does not have a set
upstreamurl. There is nothing to download and figure out the format range
for since they set the bounds manually, so don't crash on this error unless
it is running in a container non-natively.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release further parallelizes the delta creation code to do some
more complex breakup of tasks. Typically users create multiple versions
back of deltas. While each version ran massively parallel delta jobs,
each version itself ran one after another synchronously. This began to
cause problems when certain files took several minutes to complete,
because the next version could not start until the previous finished,
compounding the runtime by the number of versions back * long file.
Now, the code check if it can run a thread for each version, and then
splits up threads within those version pools so when the large file is
hit, all versions run at the same time, capping it to 1 long file run.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Run each version back for delta packs in its own thread, and divide the total
number of CPUs available and use that in each version thread. There are
cases where only a few files remain to delta, and they take multiple
minutes to finish. Each version runs synchronously with many threads, so
at the end of a version we have to wait for the long delta to finish before
running through another. version This changes it so by the end of the run -
all the large/slow files are building at the same time, capping it to just
one long file run, vs long run * number of versions back.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fix a number of AFB bugs:
- Previous build's upstream format was being incorrectly looked up,
resulting in upstream format boundary crossing detection failing.
- PreRun check was incorrectly skipping format build commands, resulting
in "stage for mix" being run twice.
- Only the UInt32 upstream version was being modified, resulting in the
second half of 'mixer build upstream-format' being run in the wrong
container.
- The wrong format was being printed out for 'mixer versions' if
crossing an upstream format boundary.
Refactor Cobra command PreRun
Simplified the PreRun logic that happens before every command. This also
fixes an issue where format bump build commands could be run with
--native=false, when they need to be run natively.
Fix AFB mixversion and LAST_VER setting
1) Changes how the format bump builds set mixversion so that it no
longer skips a version number in some situations.
2) Updates the update/image/LAST_VER file after the +10 build back
to the +20 build value, so mixer is ready to do the next mix.
Refactor file download code
This patch refactors and consolidates the different file download code
scattered throughout helper.go and builder.go.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch changes the way SERVER_STATE_DIR is created if it does not
already exist, in order to facilitate mounting the directory inside the
Docker container.
By default, SERVER_STATE_DIR in 'builder.conf' is set to PWD +
"/update", a path that typically does not already exist and is created
automaticallly the first time mixer builds bundles. Historically, 'mixer
build bundles' was run as root; now it is run as root within a Docker
container. In either case, this "update" directory ends up owned by
root.
For security reasons, mixer will not mount to the container any directory
to which the caller does not have read/write permissions. This means that
on subsequent calls, mixer refuses to mount the "update" directory it just
created, as it is owned by root.
This patch resolves this for most users by creating the SERVER_STATE_DIR
with the uid/gid of its closest existant ancester. In the default case,
this is the uid/gid of the PWD. Users with existing mixes will need to
manually chown the SERVER_STATE_DIR to a uid/gid they have read/write
permission to.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch includes the following changes:
Reorder which version builds first to simplify
The +20 build can be built fully at the beginning all the way though,
removing the need to swap tooling to build the +10 (just so things are
built in chronological order). The +10 build is created with the -
possibly - old tooling only after, and reduces the amount of
back-and-forth swapping of tooling.
Fix misc errors with formats and version blocks
The right format number needs to be passed into both functions, and we
*must* ignore the fact that the version and format are lower than the
previous build in the format bump case. It reduces complexity and tool
changes by building the +20 version all the way through first.
Two top level commands now exist for running format bumps:
- build upstream-format This command builds the necessary builds to
cross a mix over a format bump.
- build format-bump This command builds a bump for downstream users.
Fix docker mount path lookup
This patch fixes a bug with the way mixer determined which fields in
'builder.conf' were mountable paths.
Previously, mixer treated every field in [Builder] and [Mixer] as paths
on the filesystem (or paths to files whose parent directories needed to
be mounted). Introducing the "DOCKER_IMAGE_PATH" field broke this
approach.
This patch introduces a new "mount" tag on the config struct fields that
indicates whether a config field represents a mountable path. This has
several benefits:
1) It allows us to know definitively which fields we should look at.
2) It allows us to look at the entire 'builder.conf', not just the
[Builder] and [Mixer] sections.
3) It removes the restriction that paths in 'builder.conf' be absolute.
This absolute check was mostly a sloppy way of checking if a field was
in fact a path.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch adds an access check on each directory in the config file
before mounting it to the container. This prevents the user from
accidentally (or surreptitiously) granting the container access to
directories they shouldn't.
The config parsing for directory extraction code is also refactored to
provide better results and more accurate error messages.
This patch also adds unit tests for directory access checks, directory extraction,
and mount path list reduction.
For these tests to not fail inside the Docker container on Travis, they
need to not be run as root. This patch thus also creates a non-root user
(but with wheelnopw membership), and runs as this user instead.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This step cannot occur for every format bump build as it previously did,
as it is only valid for format bump builds for crossing an upstream
format boundary. If you are simply bumping your own mix format,
independent of upstream, this should not occur.
Moved the call to CheckFormatBumpNeeded, as this is the only place that
definitively knows this step needs to occur.
Adds an "UnstageMixFromBump" command that resets the "upstreamversion"
file back to the contents found in "upstreamversion.bump", if it exists.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This commit temporarily sets the default value for --native to true,
making running inside a container an opt-in behavior. This will make the
testing and transition to containerized mixer easier. Eventually this
switch will be flipped back to false, making containerized the default
running mode.
To run in container, append '--native=false' to your command.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Rodrigo Chiossi