This release further parallelizes the delta creation code to do some
more complex breakup of tasks. Typically users create multiple versions
back of deltas. While each version ran massively parallel delta jobs,
each version itself ran one after another synchronously. This began to
cause problems when certain files took several minutes to complete,
because the next version could not start until the previous finished,
compounding the runtime by the number of versions back * long file.
Now, the code check if it can run a thread for each version, and then
splits up threads within those version pools so when the large file is
hit, all versions run at the same time, capping it to 1 long file run.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Run each version back for delta packs in its own thread, and divide the total
number of CPUs available and use that in each version thread. There are
cases where only a few files remain to delta, and they take multiple
minutes to finish. Each version runs synchronously with many threads, so
at the end of a version we have to wait for the long delta to finish before
running through another. version This changes it so by the end of the run -
all the large/slow files are building at the same time, capping it to just
one long file run, vs long run * number of versions back.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fix a number of AFB bugs:
- Previous build's upstream format was being incorrectly looked up,
resulting in upstream format boundary crossing detection failing.
- PreRun check was incorrectly skipping format build commands, resulting
in "stage for mix" being run twice.
- Only the UInt32 upstream version was being modified, resulting in the
second half of 'mixer build upstream-format' being run in the wrong
container.
- The wrong format was being printed out for 'mixer versions' if
crossing an upstream format boundary.
Refactor Cobra command PreRun
Simplified the PreRun logic that happens before every command. This also
fixes an issue where format bump build commands could be run with
--native=false, when they need to be run natively.
Fix AFB mixversion and LAST_VER setting
1) Changes how the format bump builds set mixversion so that it no
longer skips a version number in some situations.
2) Updates the update/image/LAST_VER file after the +10 build back
to the +20 build value, so mixer is ready to do the next mix.
Refactor file download code
This patch refactors and consolidates the different file download code
scattered throughout helper.go and builder.go.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch changes the way SERVER_STATE_DIR is created if it does not
already exist, in order to facilitate mounting the directory inside the
Docker container.
By default, SERVER_STATE_DIR in 'builder.conf' is set to PWD +
"/update", a path that typically does not already exist and is created
automaticallly the first time mixer builds bundles. Historically, 'mixer
build bundles' was run as root; now it is run as root within a Docker
container. In either case, this "update" directory ends up owned by
root.
For security reasons, mixer will not mount to the container any directory
to which the caller does not have read/write permissions. This means that
on subsequent calls, mixer refuses to mount the "update" directory it just
created, as it is owned by root.
This patch resolves this for most users by creating the SERVER_STATE_DIR
with the uid/gid of its closest existant ancester. In the default case,
this is the uid/gid of the PWD. Users with existing mixes will need to
manually chown the SERVER_STATE_DIR to a uid/gid they have read/write
permission to.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch includes the following changes:
Reorder which version builds first to simplify
The +20 build can be built fully at the beginning all the way though,
removing the need to swap tooling to build the +10 (just so things are
built in chronological order). The +10 build is created with the -
possibly - old tooling only after, and reduces the amount of
back-and-forth swapping of tooling.
Fix misc errors with formats and version blocks
The right format number needs to be passed into both functions, and we
*must* ignore the fact that the version and format are lower than the
previous build in the format bump case. It reduces complexity and tool
changes by building the +20 version all the way through first.
Two top level commands now exist for running format bumps:
- build upstream-format This command builds the necessary builds to
cross a mix over a format bump.
- build format-bump This command builds a bump for downstream users.
Fix docker mount path lookup
This patch fixes a bug with the way mixer determined which fields in
'builder.conf' were mountable paths.
Previously, mixer treated every field in [Builder] and [Mixer] as paths
on the filesystem (or paths to files whose parent directories needed to
be mounted). Introducing the "DOCKER_IMAGE_PATH" field broke this
approach.
This patch introduces a new "mount" tag on the config struct fields that
indicates whether a config field represents a mountable path. This has
several benefits:
1) It allows us to know definitively which fields we should look at.
2) It allows us to look at the entire 'builder.conf', not just the
[Builder] and [Mixer] sections.
3) It removes the restriction that paths in 'builder.conf' be absolute.
This absolute check was mostly a sloppy way of checking if a field was
in fact a path.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch adds an access check on each directory in the config file
before mounting it to the container. This prevents the user from
accidentally (or surreptitiously) granting the container access to
directories they shouldn't.
The config parsing for directory extraction code is also refactored to
provide better results and more accurate error messages.
This patch also adds unit tests for directory access checks, directory extraction,
and mount path list reduction.
For these tests to not fail inside the Docker container on Travis, they
need to not be run as root. This patch thus also creates a non-root user
(but with wheelnopw membership), and runs as this user instead.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This step cannot occur for every format bump build as it previously did,
as it is only valid for format bump builds for crossing an upstream
format boundary. If you are simply bumping your own mix format,
independent of upstream, this should not occur.
Moved the call to CheckFormatBumpNeeded, as this is the only place that
definitively knows this step needs to occur.
Adds an "UnstageMixFromBump" command that resets the "upstreamversion"
file back to the contents found in "upstreamversion.bump", if it exists.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This commit temporarily sets the default value for --native to true,
making running inside a container an opt-in behavior. This will make the
testing and transition to containerized mixer easier. Eventually this
switch will be flipped back to false, making containerized the default
running mode.
To run in container, append '--native=false' to your command.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch Dockerizes mixer build commands. A new '--native' flag is added
that, if passed, will run mixer on the host system the way it always has.
This defaults to false.
For build commands, if '--native' is false, mixer now pulls an image file
published by upstream that contains the core Clear Linux OS and mixer
toolchain for the version of Clear Linux defined in the upstreamversion file,
which mixer than uses to build a Docker image. Mixer then re-runs the original
mixer build command within a container of that Docker image.
This ensures that you are always using a version of the mixer toolchain
compatible with the upstream version off of which you are building. This is
required for building across format boundaries. Additionally, mixer now
checks if a build crosses format boundaries, and instructs you how to
first perform a format-bump build.
Mixer attempts to re-use already downloaded or built Docker images to
minimize the performance impact of running inside a container. This eliminates
the time spent on re-building the container image across subsequent
runs. Mixer uses the latest-released version for a given format, however,
so image content will become stale regularly.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This adds support for detecting if a format bump is needed, and the
supplementary functions to perform one. All of the build steps are done
for the user, and just need to be called in the correct order to
generate the two builds.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release includes fixes for mixin, notably enabling new config for all
executions instead of per cmd basis, and enables users to name the bundle
they want the package added to instead of defaulting to the repo name.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release adds bundle information files under /usr/share/clear/allbundles
to allow clients to perform future bundle-info operations. Write with 'pretty'
indented JSON so it is grep-able and human-readable in the mean-time. Users
will have to install the os-core-update-index bundle to get this functionality.
The delta pack creation code has been re-written and parallelized to
greatly improve performance. All deltas are now created in parallel
using the full manifests, and then the packs are created in parallel by
just using the pre-created deltas from the delta/ folder, instead of
doing all the processing per bundle.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Build all of the deltas first using the full manifest, and then add them
to the packs instead of finding and creating deltas synchronously on a
per-bundle-basis.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Add bundle information files under /usr/share/clear/allbundles to allow
clients to perform future bundle-info operations. Write with 'pretty'
indented JSON so it is grep-able and human-readable in the mean-time.
Users will have to install the os-core-update-index bundle to get this
functionality.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
List the repo title (which is not upper cased) instead of the key 'name'
in the section body. This is important because repo set-url and repo add
operates on the section title not the key name.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Download a file is a generic operation. This patch moves this
functionality from builder to helpers to allow it to be reused
by other packages.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Instead of relying on external sources to keep track of the active
config file, store it as a private variable inside MixConfig.
This change prevents inconsistencies regarding filename and also
simplifies the code since the caller doesn't need to initialize the path
or know the filename if it was not the one that initialized that config
object.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Throughout the code `b` is used as Builder object variable. This patch
renames its use in createDeltaPacks() to avoid confusion.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch adds a couple new features for LoadDefaults().
First, it introduces LoadDefaultsForPath(), which allows the default
values to be set base on a given path instead of always assuming $PWD.
It also removes the restriction that only allowed defaults to be set
when --new-config set was declared. Removing this restriction ensures
that there are always sane parameters available, even if none exists
in the config file.
Lastly, it moves the check for local RPMs to LoadDefaults from
CreateDefaultConfig(). This prevents GetWd() to be called twice and also
allows LOCAL_REPO_DIR and LOCAL_RPM_DIR to be set in the same call.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch moves config out of the builder package into its own package.
This change is required to reuse MixConfig inside swupd package. Builder
package includes Swupd package, so keeping config there would cause a
cyclic dependency.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Previously 'mixer init' would add four default bundles to your
mixbundles list, unless it already existed. This resulted in confusing
behavior. Some users resorted to creating an empty mixbundles list
before running 'mixer init' in order to avoid these bundles. Others had
a mixbundles list already, but expected the bundles to be added anyway.
This patch changes the behavior to *always* add the four default
bundles, even if the mixbundles list already exists, unless a new
'--no-default-bundles' flag is passed. This does not affect the behavior
of '--all-local' or '--all-upstream'.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release fixes the check for changed includes, and fixes 'mixin' to
support local content properly.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release introduces a new mixer “config set” command to set properties
in the builder.conf, fixes for delta creation, mixin to read correct
upstream versions, and documentation for adding custom content from a build
root without building RPMs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Previously, when a "required" config value was missing, the config
object's variable name was reported, rather than the name actually used
in the builder.conf file itself.
This resulted in the following output, for example:
ERROR: Missing required field in config file: ServerStateDir
rather than
ERROR: Missing required field in config file: SERVER_STATE_DIR
This patch falls back to the variable name if a config field does not
have a "toml" name defined.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch introduces `mixer config set` command which allows any
arbitrary property withing the configuration file to be set via command
line.
The command receives the properties in the format `Section.Property`.
Only existing sections and corresponding properties can be set and this
input is validated by the command. On the other hand, the value provided
can be arbitrary and will not be validated by this command.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release introduces a new binary 'mixin' which allows users
running official upstream images to create custom content that swupd
can consume, without generating a full mix. This is a port of the
swupd-add-pkg script that lived in the swupd-client project.
Manual pages for both mixer and mixin have been added.
Bug fixes include:
- Updating RPM url in BAT tests that became stale
- Compresing the Manifest.Full
- Cleaning up manifest creation code
- Removing unused pack scripts (pack creation done by mixer now)
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The client needs to download the Manifest.full for mixer-integration to
work and expects a tar to exist.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release adds 'mixer repo' commands to allow users to add, remove,
modify, and init RPM repositories for mixer use during the build bundles
stage. This is an important prerequisite for enabling additive repos on
the client side.
Rename flags are removed from manifests and rename detection is now done
between any arbitrary version pairs during the delta-pack creation
stage. This allows us to maximize delta creations for delta-packs to
make updates faster for clients.
When reading the local bundle directory files with leading '.'s are
ignored, allowing users to track this as a git repository (with .git
directory).
A bug causing files to be skipped during fullfile creation due to a
shared error variable was fixed by declaring local-scope error variables
for all fullfile workers.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
'mixer repo set-url' allows the user to set the url of a RPM repo to the
url they provide. This is helpful when users want to build mixes off a
mirror of upstream Clear Linux instead of the CDN when the CDN is slow.
mixer repo set-url clear <mirror-url>
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Previously the code was using ioutils.ReadDir which does a stat call
on every file, but none of the code required the results of the stat
calls, just the names.
In addition programs such as git would like to hold information in the
directories. The usual unix way to do this is to have filenames
beginning with '.', and these are ignored by default by things such as
shell globbing and the ls program.
fixes#269
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
The 'mixer repo init' command writes the default DNF configuration file
used by mixer if it does not already exist. The configuration file is
written with the default "Clear" repository and potentially the "local"
repository depending on user setup.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The 'mixer repo list' command lists all available repos in the DNF
configuration file used by mixer.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Do direct string matching instead of insensitive string matching. The
insensitive load will break when someone tries to add something with
capital letters twice. For example, the following will be allowed when
it should not be:
mixer repo add QtPy url
mixer repo add QtPy url
Do a regular load so the string comparison with upper-case names will
work.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The add-repo subcommand adds a remote repo configuration to the DNF conf
defined in the builder.conf. This allows users to configure mixer to use
remote RPM repositories when building mixes instead of being limited to
just local RPMs and upstream Clear Linux RPMs.
This feature is an important pre-requisite for enabling PPA-style repos
on the client side.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Sometimes dnf output will not contain the "Installing:\n" pattern,
e.g. if there is a package that doesn't exist in a bundle, the output
looks like
Last metadata expiration check: 0:00:06 ago on Thu 19 Apr 2018 05:09:16 PM PDT.
No match for argument: libc
and mixer would crash because the Split resulting slice wouldn't have
a second element.
The actual error will be caught later when the install of the "full
chroot" happens, so it is fine for now just ignore it.
Ideally the goroutine handling should be similar to fullfiles, so the
workers have a way to indicate that an error occurred and the program
can finish early. Added that as a TODO.
The recent change in 'mixer build bundles' contained a parsing bug that
caused some packages to be dropped. The way 'dnf install' output was
split caused the regex to miss the final line on some bundles. This
patch fixes this bug.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release contains a bug fix for name collisions between upstream
and local packages. Now the repository priority in the DNF conf is
correctly respected during the 'mixer build bunldles' command.
This release also includes improvements to Mixer's Travis CI.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Previously, mixer failed to build update content if local and upstream
repositories had a package name collision with different content.
'mixer build bundles' relies on the output of 'dnf repoquery -l' to get
the contents of a given package and 'dnf install' to put the contents
of that package into the full chroot. 'dnf repoquery' does not utilize
the repository prioritization resolver that 'dnf install' does, and
instead lists the contents of every matching package in every repository.
This results in the <bundle>-info file containing the file lists of both
the upstream and the local version of the rpm, but the full chroot only
containing the actual files from the (higher-priority) local rpm. 'mixer
build bundles' then fails as it attempts to resolve missing files.
This patch fixes this by separating the packages in each bundle based on
the repository they come from according to the 'dnf install' resolution.
For each bundle, a separate 'dnf repoquery -l --repo' command is made
for each repository, guaranteeing that only the correct repopository is
queried for a given package.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release moves the new swupd and chroot functionality out from
behind --new-swupd and --new-chroots, making it the standard behavior
for all users. The flags are left as deprecated to not break scripts,
but they should be removed by users going forward.
Additional improvements in this release include clearing out DNF state
files from the internal chroot representation, and removing an error
response from the delta pack generation in the case that --to == --from.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
The new manifest creation implementation has been tested in a dev-ops
workflow and is robust enough to remove the old implementation. This
patch deprecates and hides the --new-swupd flag and only runs the new
implementation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The new bundle building implementation is now well-tested and used in
heavy dev-ops workflows. Deprecate and hide the --new-chroots flag and
only run the new implementation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The use case for this is when you want to always allow scripted calls to
build delta-packs to create a delta-pack from the min-version, which
might match the current version for a full min-version. This is
conceptually equivalent to finding 0 previous versions. Print the
information and return nil.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Tudor Marcu