This release introduces a new mixer “config set” command to set properties
in the builder.conf, fixes for delta creation, mixin to read correct
upstream versions, and documentation for adding custom content from a build
root without building RPMs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Some users are asking for instructions on how to build custom bundles
without having to go through the pain of packaging their content in an
RPM. Provide a tutorial explaining how this can be achieved.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
If the upstreamversion file exists read the current version from there.
Only read from /usr/lib/os-release if the upstreamversion file does not
exist. The os-release file is modified every time a local mix is
migrated to (version * 1000) so reading from that file every time causes
the mix version to multiply by 1000 every mix build.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The name used in battests for config set were using the name used in the
code for the properties instead of the name found in the config file
causing the test to fail.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Previously, when a "required" config value was missing, the config
object's variable name was reported, rather than the name actually used
in the builder.conf file itself.
This resulted in the following output, for example:
ERROR: Missing required field in config file: ServerStateDir
rather than
ERROR: Missing required field in config file: SERVER_STATE_DIR
This patch falls back to the variable name if a config field does not
have a "toml" name defined.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
LOCAL_BUNDLE_DIR and LOCAL_REPO_DIR must be added using TOML format.
This patch changes bat tests to try to use new-config for setting the
properties. If it fails, it will fall back to appending INI properties
to the end of the file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Allow global arguments to be passed to mixer during `make batcheck`
through the $MIXARGS environment variable.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch introduces `mixer config set` command which allows any
arbitrary property withing the configuration file to be set via command
line.
The command receives the properties in the format `Section.Property`.
Only existing sections and corresponding properties can be set and this
input is validated by the command. On the other hand, the value provided
can be arbitrary and will not be validated by this command.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Call the ExitStatus method to get the actual exit code of the bsdiff
command to check for the FULLDL return code.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Add a unit test to make sure we do not create delta files larger than
the compressed full file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Do not keep a delta if its size is larger than or equal to the size of
the compressed full file since download size is not being optimized in
this case.
This is not a critical error so do not remove the delta if the
supporting information can not be determined. For example, this check
fails if full files were not created before creating deltas because
there is no compressed full file to check. This is the case with many of
our unit tests.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
One of the checks started failing, not sure why it was passing before
but not any more. Update all checks against $line to use integer
comparison instead of string comparison.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release introduces a new binary 'mixin' which allows users
running official upstream images to create custom content that swupd
can consume, without generating a full mix. This is a port of the
swupd-add-pkg script that lived in the swupd-client project.
Manual pages for both mixer and mixin have been added.
Bug fixes include:
- Updating RPM url in BAT tests that became stale
- Compresing the Manifest.Full
- Cleaning up manifest creation code
- Removing unused pack scripts (pack creation done by mixer now)
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This patch moves the complicated processing loops out of top-level
functions to make them more readable and thus reduce gocyclo complaints
for this operation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Write a flag file to MIXWS/.valid-mix when a mixer build is successful
so swupd-client will know when the user's mix is in a good state to run
an update from.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The client needs to download the Manifest.full for mixer-integration to
work and expects a tar to exist.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The pack-maker.sh and superpack-maker.sh scripts are no longer used or
supported by mixer. Remove these scripts from the mixer repository.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of requiring the user to add a bundle name when invoking mixin
package add implicitly add the package to a bundle named after the
repository it is found in. This also effectively adds a check that the
package the user is trying to add actually exists.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Allow the user to manipulate their DNF configuration by initializing the
builder.conf if it does not exist.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The build command builds the mixer content under /usr/share/mix for use
by swupd update --migrate.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This commit rewrites swupd-add-pkg into a Go binary provided by mixer.
It uses mixer libraries to make manifest merges and manipulation easier
and is a first step in enabling external repos for mixer-integration
client-side.
Some minor changes to the swupd library were necessary to make the 'm'
flag a supported flag for integration manifests.
The usage of the new binary is slightly different from the old script.
mixin <bundle-name> <path/to/rpm>
Instead of requiring the user to copy the rpm to /usr/share/mix/rpms it
does the work for the user.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release adds 'mixer repo' commands to allow users to add, remove,
modify, and init RPM repositories for mixer use during the build bundles
stage. This is an important prerequisite for enabling additive repos on
the client side.
Rename flags are removed from manifests and rename detection is now done
between any arbitrary version pairs during the delta-pack creation
stage. This allows us to maximize delta creations for delta-packs to
make updates faster for clients.
When reading the local bundle directory files with leading '.'s are
ignored, allowing users to track this as a git repository (with .git
directory).
A bug causing files to be skipped during fullfile creation due to a
shared error variable was fixed by declaring local-scope error variables
for all fullfile workers.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of using the same error variable across all workers declare a
new error variable for each goroutine. This prevents race conditions
where multiple routines are using the same error variable to make
decisions and failures are missed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
'mixer repo set-url' allows the user to set the url of a RPM repo to the
url they provide. This is helpful when users want to build mixes off a
mirror of upstream Clear Linux instead of the CDN when the CDN is slow.
mixer repo set-url clear <mirror-url>
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Previously the code was using ioutils.ReadDir which does a stat call
on every file, but none of the code required the results of the stat
calls, just the names.
In addition programs such as git would like to hold information in the
directories. The usual unix way to do this is to have filenames
beginning with '.', and these are ignored by default by things such as
shell globbing and the ls program.
fixes#269
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
The 'mixer repo init' command writes the default DNF configuration file
used by mixer if it does not already exist. The configuration file is
written with the default "Clear" repository and potentially the "local"
repository depending on user setup.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The 'mixer repo list' command lists all available repos in the DNF
configuration file used by mixer.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Do direct string matching instead of insensitive string matching. The
insensitive load will break when someone tries to add something with
capital letters twice. For example, the following will be allowed when
it should not be:
mixer repo add QtPy url
mixer repo add QtPy url
Do a regular load so the string comparison with upper-case names will
work.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This test adds a test for the following case:
* 10 contains files A B C
* 20 contains files A1 B C1
* 30 contains files A B1 C2
* 10->20 pack has deltas for A->A1, C->C1
* 20->30 pack has deltas for A1->A, B->B1, C1->C2
* 10->30 pack has deltas for B->B1, C->C2
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of performing rename detection for manifest creation between the
current and previous version, perform rename detection for every delta
pack that is created, thus maximizing the number of deltas created for
each pack.
This removes the now-unused rename flag from the manifest. Now that
swupd-client optimistically applies all deltas in a pack it no longer
relies on a rename flag to tell it if a delta is available.
All deleted files now have a zeroed hash, some tests had to be updated
to reflect and check this.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The add-repo subcommand adds a remote repo configuration to the DNF conf
defined in the builder.conf. This allows users to configure mixer to use
remote RPM repositories when building mixes instead of being limited to
just local RPMs and upstream Clear Linux RPMs.
This feature is an important pre-requisite for enabling PPA-style repos
on the client side.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Sometimes dnf output will not contain the "Installing:\n" pattern,
e.g. if there is a package that doesn't exist in a bundle, the output
looks like
Last metadata expiration check: 0:00:06 ago on Thu 19 Apr 2018 05:09:16 PM PDT.
No match for argument: libc
and mixer would crash because the Split resulting slice wouldn't have
a second element.
The actual error will be caught later when the install of the "full
chroot" happens, so it is fine for now just ignore it.
Ideally the goroutine handling should be similar to fullfiles, so the
workers have a way to indicate that an error occurred and the program
can finish early. Added that as a TODO.
The recent change in 'mixer build bundles' contained a parsing bug that
caused some packages to be dropped. The way 'dnf install' output was
split caused the regex to miss the final line on some bundles. This
patch fixes this bug.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release contains a bug fix for name collisions between upstream
and local packages. Now the repository priority in the DNF conf is
correctly respected during the 'mixer build bunldles' command.
This release also includes improvements to Mixer's Travis CI.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Previously, mixer failed to build update content if local and upstream
repositories had a package name collision with different content.
'mixer build bundles' relies on the output of 'dnf repoquery -l' to get
the contents of a given package and 'dnf install' to put the contents
of that package into the full chroot. 'dnf repoquery' does not utilize
the repository prioritization resolver that 'dnf install' does, and
instead lists the contents of every matching package in every repository.
This results in the <bundle>-info file containing the file lists of both
the upstream and the local version of the rpm, but the full chroot only
containing the actual files from the (higher-priority) local rpm. 'mixer
build bundles' then fails as it attempts to resolve missing files.
This patch fixes this by separating the packages in each bundle based on
the repository they come from according to the 'dnf install' resolution.
For each bundle, a separate 'dnf repoquery -l --repo' command is made
for each repository, guaranteeing that only the correct repopository is
queried for a given package.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Tudor Marcu