This release adds 'mixer repo' commands to allow users to add, remove,
modify, and init RPM repositories for mixer use during the build bundles
stage. This is an important prerequisite for enabling additive repos on
the client side.
Rename flags are removed from manifests and rename detection is now done
between any arbitrary version pairs during the delta-pack creation
stage. This allows us to maximize delta creations for delta-packs to
make updates faster for clients.
When reading the local bundle directory files with leading '.'s are
ignored, allowing users to track this as a git repository (with .git
directory).
A bug causing files to be skipped during fullfile creation due to a
shared error variable was fixed by declaring local-scope error variables
for all fullfile workers.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of using the same error variable across all workers declare a
new error variable for each goroutine. This prevents race conditions
where multiple routines are using the same error variable to make
decisions and failures are missed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
'mixer repo set-url' allows the user to set the url of a RPM repo to the
url they provide. This is helpful when users want to build mixes off a
mirror of upstream Clear Linux instead of the CDN when the CDN is slow.
mixer repo set-url clear <mirror-url>
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Previously the code was using ioutils.ReadDir which does a stat call
on every file, but none of the code required the results of the stat
calls, just the names.
In addition programs such as git would like to hold information in the
directories. The usual unix way to do this is to have filenames
beginning with '.', and these are ignored by default by things such as
shell globbing and the ls program.
fixes#269
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
The 'mixer repo init' command writes the default DNF configuration file
used by mixer if it does not already exist. The configuration file is
written with the default "Clear" repository and potentially the "local"
repository depending on user setup.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The 'mixer repo list' command lists all available repos in the DNF
configuration file used by mixer.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Do direct string matching instead of insensitive string matching. The
insensitive load will break when someone tries to add something with
capital letters twice. For example, the following will be allowed when
it should not be:
mixer repo add QtPy url
mixer repo add QtPy url
Do a regular load so the string comparison with upper-case names will
work.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This test adds a test for the following case:
* 10 contains files A B C
* 20 contains files A1 B C1
* 30 contains files A B1 C2
* 10->20 pack has deltas for A->A1, C->C1
* 20->30 pack has deltas for A1->A, B->B1, C1->C2
* 10->30 pack has deltas for B->B1, C->C2
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of performing rename detection for manifest creation between the
current and previous version, perform rename detection for every delta
pack that is created, thus maximizing the number of deltas created for
each pack.
This removes the now-unused rename flag from the manifest. Now that
swupd-client optimistically applies all deltas in a pack it no longer
relies on a rename flag to tell it if a delta is available.
All deleted files now have a zeroed hash, some tests had to be updated
to reflect and check this.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The add-repo subcommand adds a remote repo configuration to the DNF conf
defined in the builder.conf. This allows users to configure mixer to use
remote RPM repositories when building mixes instead of being limited to
just local RPMs and upstream Clear Linux RPMs.
This feature is an important pre-requisite for enabling PPA-style repos
on the client side.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Sometimes dnf output will not contain the "Installing:\n" pattern,
e.g. if there is a package that doesn't exist in a bundle, the output
looks like
Last metadata expiration check: 0:00:06 ago on Thu 19 Apr 2018 05:09:16 PM PDT.
No match for argument: libc
and mixer would crash because the Split resulting slice wouldn't have
a second element.
The actual error will be caught later when the install of the "full
chroot" happens, so it is fine for now just ignore it.
Ideally the goroutine handling should be similar to fullfiles, so the
workers have a way to indicate that an error occurred and the program
can finish early. Added that as a TODO.
The recent change in 'mixer build bundles' contained a parsing bug that
caused some packages to be dropped. The way 'dnf install' output was
split caused the regex to miss the final line on some bundles. This
patch fixes this bug.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release contains a bug fix for name collisions between upstream
and local packages. Now the repository priority in the DNF conf is
correctly respected during the 'mixer build bunldles' command.
This release also includes improvements to Mixer's Travis CI.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Previously, mixer failed to build update content if local and upstream
repositories had a package name collision with different content.
'mixer build bundles' relies on the output of 'dnf repoquery -l' to get
the contents of a given package and 'dnf install' to put the contents
of that package into the full chroot. 'dnf repoquery' does not utilize
the repository prioritization resolver that 'dnf install' does, and
instead lists the contents of every matching package in every repository.
This results in the <bundle>-info file containing the file lists of both
the upstream and the local version of the rpm, but the full chroot only
containing the actual files from the (higher-priority) local rpm. 'mixer
build bundles' then fails as it attempts to resolve missing files.
This patch fixes this by separating the packages in each bundle based on
the repository they come from according to the 'dnf install' resolution.
For each bundle, a separate 'dnf repoquery -l --repo' command is made
for each repository, guaranteeing that only the correct repopository is
queried for a given package.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
In the recent BATS test rewrite, two of the 'run.bats' file deletions
got lost along the way. The tests were not being run (because their
corresponding description.txt files *were* removed); they were just
taking up space. This patch removes them.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch updates the Travis Dockerfile to build off the new
clearlinux/mixer-ci image.
Previously, the Dockerfile built off the base clearlinux:latest image,
which meant it spent quite some time (~7 min) just building the Docker
image (swup bundl-add'ing the packages it needed) before it could build
and test mixer. This patch eliminates nearly all of that time, as the
Docker build only needs to download an image and copy the mixer files
over before the mixer build and tests can begin.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release moves the new swupd and chroot functionality out from
behind --new-swupd and --new-chroots, making it the standard behavior
for all users. The flags are left as deprecated to not break scripts,
but they should be removed by users going forward.
Additional improvements in this release include clearing out DNF state
files from the internal chroot representation, and removing an error
response from the delta pack generation in the case that --to == --from.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
The new manifest creation implementation has been tested in a dev-ops
workflow and is robust enough to remove the old implementation. This
patch deprecates and hides the --new-swupd flag and only runs the new
implementation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The new bundle building implementation is now well-tested and used in
heavy dev-ops workflows. Deprecate and hide the --new-chroots flag and
only run the new implementation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The use case for this is when you want to always allow scripted calls to
build delta-packs to create a delta-pack from the min-version, which
might match the current version for a full min-version. This is
conceptually equivalent to finding 0 previous versions. Print the
information and return nil.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The fullDir path does not change for each bundle. Move it outside the
bundle install loop so it is not reassigned every iteration.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When building the full chroot DNF lays down some state directories/files
which end up in the Manifest.full later on just by virtue of them being
there. These files should be removed from the chroot when all package
operations are completed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release refactors the BAT tests and adds new tests for the bundle
command, fixes a bug where the version URL was set to the content URL,
reports packages as part of bundle list, and cleans up temp dirs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
A recent bug in swupd-client prevents users from updating due to a
maximum contensize check that was a couple orders of magnitude off.
These clients are broken and cannot update if they have installed
bundles that violate this maximum.
To allow these broken clients to update temporarily set the maximum
contentsize to the bogus contentsize expected by the client.
Remove this check once a format bump has occurred.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Previously, --new-chroots was accidentally using the 'CONTENTURL' value
from builder.conf for both the 'contenturl' and 'versionurl' files.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
'mixer build bundles' creates a temp dir to use for one of the dnf
commands. This patch cleans that dir up when mixer is done building the
bundles.
This is really minor, as by default it writes to /tmp, which gets
cleaned up eventually anyway. However, ioutil.TempDir() will write to
wherever $TMPDIR points, which could be anywhere.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch changes the output for 'mixer bundle list' and 'mixer bundle
list --tree' to distinguish between bundles and packages. This should
help clarify exactly where items in the mix are coming from.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch significantly overhauls the BATS tests for building mixes,
with the goal of reducing redundancy and increasing accuracy.
Each of the following concepts is only tested once:
- Building a mix with a full, real upstream bundle
- Updating the version of a mix (technically tested twice)
- Adding a bundle to a mix
- Removing a bundle from a mix
- Building a mix with a custom local bundle
- Adding a custom package to an upstream bundle
- Adding an upstream package to a custom bundle
Because all but one of the tests use stripped down versions of upstream
bundles for efficiency, they all test:
- Building a mix with a local, edited upstream bundle
This dramatically reduces the number of actual builds that occur.
The test descriptions accurately describe what the tests are actually
doing, and they report if they are using stripped down versions of upstream
bundles.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Add a new set of tests that run through a sequence of 'mixer bundle'
commands and test that they are working as expected.
Note that this is re-using the "01" number, as the rest are being
reworked as a separate commit.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This step was completely unnecessary and created empty chroot
directories in the image dir. This was a product of a too-direct port of
the old swupd-server for this specific part of the rewrite. It probably
wasn't even needed in the old swupd-server.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch updates the 'mixer bundle list' command to read the local and
upstream package files. Now 'mixer bundle list local', for example, will
list package-bundles defined in local-packages alongside bundles defined
in local-bundles/.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release contains four major additions:
1) Support for package-level bundles. A new local and upstream package
list has been introduced, which mixer parses to automatically generate
single-package bundles.
2) The bundle building algorithm has been significantly optimized. This
step now runs approximately three times faster than the previous version.
3) New TOML configuration support. Mixer now has an optional
'--new-config' flag that adds support for TOML-based configuration. A
new 'mixer config' command has been added to validate existing
builder.conf files, as well as convert legacy (INI) files into the new
TOML format.
4) Update to how mixer generates the DNF conf file used during bundle
building. This removes the dependency on m4.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Add 'convert' as a 'mixer config' subcommand. This command allows the
conversion from old config files to the new TOML config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Add 'mixer config' utility command along with 'validate' subcommand,
which parses a config file and print the expanded values
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch adds a new type of config file for mixer, based on the TOML
format. The new format is not compatible with the old parser and
vice-versa. The new flag --new-config enables the use of the new format.
This patch also adds to the parser the properties that exist in the
config file but were not parsed by the older parser.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The process of checking if a path was provided and getting pwd if not is
required in multiple parts of the code, so this path allows for code
reuse.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Separate the parsing step into parsing, environment variables expansion
and validation. This allows the last two steps to be reused even if the
parse method changes.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Separete config loading and parsing from Builder object into its own
MixConfig class.
This patch also removes the parsing of CLEARVER and MIXVER since they
are parsed from .mixversion and .clearversion files instead of
builder.conf
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The mixerlib file and the CLEANDIRS variable in all of the individual
bat Makefiles still pointed to the old "local" and "rpms" directories in
some locations.
This, made 'make batclean' not clean up directories correctly. This likely
made some of the tests not actually do what they were supposed to do, as it
was looking for rpms in 'local-rpms', but putting them in 'rpms'.
This patch also updates the url for the "json-c" rpm, as the previous url no
longer is active and was to a source rpm.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Previously, mixer used m4 to generate the dnf conf (.yum.conf), and only
did so if the file did not already exist, so as to not overwrite people's
custom conf files. However, this meant that the check if a "[local]"
section was needed only ran if the file didn't exist. If a mix had already
been built, this check was skipped, and the local section was never inserted.
This patch removes the use of the yum.conf.in file and m4, instead using an
embedded template. Additionally, if the file already exists but the local
section is needed, mixer will now check the file to see if it has the necessary
local section. If not, the section will be appended to the file.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
When building bundles perform all DNF transactions in bulk. There are
two reasons to do this.
1. DNF is too smart for its own good. It will resolve dependencies
differently depending on the packages listed in the transaction.
Therefore we must perform these transactions with all relevant
packages listed for the bundle being processed.
2. DNF transactions take time. When performing much fewer transactions
in bulk the bundle builder can execute in a shorter time.
Certain manifest validation tools also require a complete dep-solved
list of all packages and included packages for a bundle to validate the
bundle manifest. For this reason use the bundle.AllPackages list and
leave the bundle.DirectPackages list as an informational list of just
the direct packages specified in the bundle definition file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Matthew Johnson