The recent change in 'mixer build bundles' contained a parsing bug that
caused some packages to be dropped. The way 'dnf install' output was
split caused the regex to miss the final line on some bundles. This
patch fixes this bug.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release contains a bug fix for name collisions between upstream
and local packages. Now the repository priority in the DNF conf is
correctly respected during the 'mixer build bunldles' command.
This release also includes improvements to Mixer's Travis CI.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Previously, mixer failed to build update content if local and upstream
repositories had a package name collision with different content.
'mixer build bundles' relies on the output of 'dnf repoquery -l' to get
the contents of a given package and 'dnf install' to put the contents
of that package into the full chroot. 'dnf repoquery' does not utilize
the repository prioritization resolver that 'dnf install' does, and
instead lists the contents of every matching package in every repository.
This results in the <bundle>-info file containing the file lists of both
the upstream and the local version of the rpm, but the full chroot only
containing the actual files from the (higher-priority) local rpm. 'mixer
build bundles' then fails as it attempts to resolve missing files.
This patch fixes this by separating the packages in each bundle based on
the repository they come from according to the 'dnf install' resolution.
For each bundle, a separate 'dnf repoquery -l --repo' command is made
for each repository, guaranteeing that only the correct repopository is
queried for a given package.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release moves the new swupd and chroot functionality out from
behind --new-swupd and --new-chroots, making it the standard behavior
for all users. The flags are left as deprecated to not break scripts,
but they should be removed by users going forward.
Additional improvements in this release include clearing out DNF state
files from the internal chroot representation, and removing an error
response from the delta pack generation in the case that --to == --from.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
The new manifest creation implementation has been tested in a dev-ops
workflow and is robust enough to remove the old implementation. This
patch deprecates and hides the --new-swupd flag and only runs the new
implementation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The new bundle building implementation is now well-tested and used in
heavy dev-ops workflows. Deprecate and hide the --new-chroots flag and
only run the new implementation.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The use case for this is when you want to always allow scripted calls to
build delta-packs to create a delta-pack from the min-version, which
might match the current version for a full min-version. This is
conceptually equivalent to finding 0 previous versions. Print the
information and return nil.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The fullDir path does not change for each bundle. Move it outside the
bundle install loop so it is not reassigned every iteration.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When building the full chroot DNF lays down some state directories/files
which end up in the Manifest.full later on just by virtue of them being
there. These files should be removed from the chroot when all package
operations are completed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release refactors the BAT tests and adds new tests for the bundle
command, fixes a bug where the version URL was set to the content URL,
reports packages as part of bundle list, and cleans up temp dirs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Previously, --new-chroots was accidentally using the 'CONTENTURL' value
from builder.conf for both the 'contenturl' and 'versionurl' files.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
'mixer build bundles' creates a temp dir to use for one of the dnf
commands. This patch cleans that dir up when mixer is done building the
bundles.
This is really minor, as by default it writes to /tmp, which gets
cleaned up eventually anyway. However, ioutil.TempDir() will write to
wherever $TMPDIR points, which could be anywhere.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch changes the output for 'mixer bundle list' and 'mixer bundle
list --tree' to distinguish between bundles and packages. This should
help clarify exactly where items in the mix are coming from.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch updates the 'mixer bundle list' command to read the local and
upstream package files. Now 'mixer bundle list local', for example, will
list package-bundles defined in local-packages alongside bundles defined
in local-bundles/.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release contains four major additions:
1) Support for package-level bundles. A new local and upstream package
list has been introduced, which mixer parses to automatically generate
single-package bundles.
2) The bundle building algorithm has been significantly optimized. This
step now runs approximately three times faster than the previous version.
3) New TOML configuration support. Mixer now has an optional
'--new-config' flag that adds support for TOML-based configuration. A
new 'mixer config' command has been added to validate existing
builder.conf files, as well as convert legacy (INI) files into the new
TOML format.
4) Update to how mixer generates the DNF conf file used during bundle
building. This removes the dependency on m4.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Add 'convert' as a 'mixer config' subcommand. This command allows the
conversion from old config files to the new TOML config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Add 'mixer config' utility command along with 'validate' subcommand,
which parses a config file and print the expanded values
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch adds a new type of config file for mixer, based on the TOML
format. The new format is not compatible with the old parser and
vice-versa. The new flag --new-config enables the use of the new format.
This patch also adds to the parser the properties that exist in the
config file but were not parsed by the older parser.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The process of checking if a path was provided and getting pwd if not is
required in multiple parts of the code, so this path allows for code
reuse.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Separate the parsing step into parsing, environment variables expansion
and validation. This allows the last two steps to be reused even if the
parse method changes.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Separete config loading and parsing from Builder object into its own
MixConfig class.
This patch also removes the parsing of CLEARVER and MIXVER since they
are parsed from .mixversion and .clearversion files instead of
builder.conf
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Previously, mixer used m4 to generate the dnf conf (.yum.conf), and only
did so if the file did not already exist, so as to not overwrite people's
custom conf files. However, this meant that the check if a "[local]"
section was needed only ran if the file didn't exist. If a mix had already
been built, this check was skipped, and the local section was never inserted.
This patch removes the use of the yum.conf.in file and m4, instead using an
embedded template. Additionally, if the file already exists but the local
section is needed, mixer will now check the file to see if it has the necessary
local section. If not, the section will be appended to the file.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
When building bundles perform all DNF transactions in bulk. There are
two reasons to do this.
1. DNF is too smart for its own good. It will resolve dependencies
differently depending on the packages listed in the transaction.
Therefore we must perform these transactions with all relevant
packages listed for the bundle being processed.
2. DNF transactions take time. When performing much fewer transactions
in bulk the bundle builder can execute in a shorter time.
Certain manifest validation tools also require a complete dep-solved
list of all packages and included packages for a bundle to validate the
bundle manifest. For this reason use the bundle.AllPackages list and
leave the bundle.DirectPackages list as an informational list of just
the direct packages specified in the bundle definition file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch adds the capability to add package-scope bundles to your mix
either from upstream "packages" file or from a "local-packages" file in
your mix directory. These files list one package per line like so:
helloworld
vim
emacs
The bundles created from these packages will be named after the package
itself and will only have that one package defined (plus filesystem as a
hard-coded depencency). The bundle will have no includes. A bundle
definition for the helloworld package will look like this:
Name: helloworld
Filename: /path/to/{local-packages,packages}
DirectPackages: [helloworld, filesystem]
These packages can be added as bundles simply by running
mixer bundle add helloworld
and mixer will look for the definition in the following order:
1. local-bundles directory
2. local-packages file
3. upstream-bundles directory
4. packages file (upstream)
and will take the first definition it finds. The rest of mixer will see
this package as just another bundle and update content will be created
in the same manner as all other bundles.
The validate-swupd.sh script was added to define a helloworld package,
bundle add it, and test it with the client.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release adds an os-packages file to list all package/srpms in
the build, which can be useful for determining more information on
packages and their corresponding SRPM name.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This file is intended to be used by validation tooling to identify
orphaned packages and verify there are no file collisions in the build.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes the following bugs:
- Resolve /lib to /usr/lib instead of /usr/lib64
- Install DirectPackages to full chroot instead of AllPackages
- DirectPackage is the fully resolved list for the bundle(s)
- Skip continued lines in dnf install output causing missing lines in parsing
- Clear cache before DNF install to full chroot causing dnf tracebacks
Smaller fixes such as adding trailing newline to mixbundles, parsing
versions numbers on init, normalizing bundle name output, and updating
all (currently possible) yum references to dnf were also added.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Previously mixer wrote the mix bundle list as a newline-separated list,
rather than as a newline-delimited list. This made things like "wc -l"
report one fewer than expected, as the final line contained no newline.
This trailing newline does not affect the existing parsing code.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Keeping the cache around has resulted in interesting errors with invalid
bytes being read from the cache and resulting in DNF tracebacks. As a
fix for this tell DNF not to keepcache and manually clear the cache
before doing the full install. This seems to either be a bug with DNF
itself or maybe with the way Clear Linux is doing its packaging (though
this is just a guess).
The error seen is a UnicodeDecodeError from the DNF source code. It is
trying to decode a non-UTF-8 byte as UTF-8:
Traceback (most recent call last):
File "/usr/bin/dnf", line 58, in <module>
main.user_main(sys.argv[1:], exit_code=True)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 179, in user_main
errcode = main(args)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 64, in main
return _main(base, args, cli_class, option_parser_class)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 99, in _main
return cli_run(cli, base)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 123, in cli_run
ret = resolving(cli, base)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 154, in resolving
base.do_transaction(display=displays)
File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 238, in do_transaction
super(BaseCli, self).do_transaction(display)
File "/usr/lib/python3.6/site-packages/dnf/base.py", line 781, in do_transaction
self._run_transaction(cb=cb)
File "/usr/lib/python3.6/site-packages/dnf/base.py", line 925, in _run_transaction
self._verify_transaction(cb.verify_tsi_package)
File "/usr/lib/python3.6/site-packages/dnf/base.py", line 1018, in _verify_transaction
self.history.sync_alldb(po)
File "/usr/lib/python3.6/site-packages/dnf/yum/history.py", line 1399, in sync_alldb
self._save_rpmdb(ipkg) and
File "/usr/lib/python3.6/site-packages/dnf/yum/history.py", line 1360, in _save_rpmdb
val = getattr(ipkg, attr, None)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
DNF likes to set line-width to a default 80 characters when not running
in a TTY (this was confirmed in the source code) and perform a
complicated 'smart wrap' on the fields after any fields that overflow
their column. This results in wrapped lines that do not have the
information we are looking for. Fortunately these lines are prefixed by
more than one space (' ') character. Since package names are always at
the beginning of the line and only prefixed by one space character, if
there is more than one space we can skip the line when parsing.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Do not just install the AllPackages list to the full chroot, since this
is just the list of all included packages. The DirectPackages need to be
installed instead since this is the fully resolved list for each bundle.
AllPackages do not need to be installed because all bundles are being
installed to the same place.
Do a check on the length of the DirectPackages list before actually
trying to install them so we don't try to 'dnf install ' (no package
specified) for bundles that don't specify their own packages and only
include other bundles.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Recently, code was added to ReadVersions to parse the mixversion
and upstreamversion string values as uint32 values. This patch
adds the same logic to init (the only command that does not
result in a call to ReadVersions), so that the parsed versions
will be consistently available.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Recent changes deprecated the use of individual bundle chroots when
building bundle contents and moved away from using yum to using dnf.
Because of this much of our variable and file names and comments were
inconsistent with the implementation.
This includes a change to make 'mixer build chroots' an alias to 'mixer
build bundles' and updates to the bat tests to call the new 'mixer build
bundles' command. This is not a breaking change since 'mixer build
chroots' behaves the way it always has (except internally with the
recent changes mentioned).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Previously, the names of bundles were output differently from one
command to another. Some would insert the bundle name as-is, some would
surround it with single quotes, and some would use double quotes.
This patch normalizes all output (that I could find in builder.conf) to
use the same output format: all now use the %q formatter, which renders
them with double quotes.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release includes a significant change to the way chroots are built.
Rather than building individual chroots for each bundle, a single full
chroot is generated, and JSON metadata files are created for each bundle.
This allows us to scale much more easily on machines with limited disk
space. Because chroots are now decoupled from manifest creation, this
also makes it possible in the future to move to bundle builds on
distributed machines.
Additionally, a change was made to allow developers to add extra files
to a bundle in addition to those included by the bundle packages.
A change was included that fails when the manifest cannot be parsed.
Minor bug fixes are also included.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This overhaul leaves several tests using the old method of using chroots
to generate the update to verify we aren't totally deprecating the old
method.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Deprecate use of chroot-building for each individual bundle. This allows
us to scale much easier on machines with limited disk space. Because
chroots are decoupled from manifest creation this also makes it possible
in the future to move to doing bundle builds on distributed machines and
allows our bundle numbers to scale. The bundle file list is stored in a
JSON file that is read later by the manifest creation step.
This leaves a fallback in swupd/fullchroot.go to allow users to add
content to a chroot in order to extend a typical build.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes:
- Travis: add workaround to failing clrtrust
- build-deltas: Do not fail on missing previous versions
- Update 'mixer bundle add' to skip existing bundles
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Previously, repeated calls to 'mixer bundle add <bundle>' would yield
the same "Adding bundle <bundle> from <source>" output. Because the
Mix Bundle List is treated as a set, and duplicate bundles are removed,
these redundant additions do not change the resultant list, and are
effectively no-ops.
This patch:
a) Explictly skips the bundles if they already exist.
b) Reports this case to the user
As a result, functionality is equivalent, but the no-op behavior is
more explicit and output is more clear.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
A version can be missing if the minversion is newer than the previous
version being checked. This is not an error, but should be printed as a
warning to the user in case this is not expected.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release changes basic bundle validation to support previously
non-passing bundles in upstream so mixes with previous versions are not
so strictly filtered. Hardlinking only happens if --keep-chroots is
passed to avoid race conditions and wasting time hardlinking when it
does not need to happen. Issues with bundle list were fixed, offline
mode was added for mixer to skip caching upstream bundle definitions, a
default value for local-bundle directory was added, and other updates to
variables to match the new naming schemes 4.0.* introduced.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Previously, "basic" bundle validation included checking that the
bundle filename (currently used as the "bundle name") matched the
"Title" field in the bundle header (currently not used), and that
the "Title" field itself was valid. This was a step toward future
usage, where mixer will use the header "Title" itself.
This caused mixer to be incompatible with previous versions of CLR,
for which these filename-to-title missmatches were allowed and thus
present.
This patch moves this checking exclusively to "--strict" validation,
and removes bundle content validation from "basic" altogether. Basic
validation now only checks that the bundle filename is valid and that
the bundle contents syntax can be parsed.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Due to using nosync, hardlinking could start before all of the individual
bundle chroots are actually removed. However, we do not need to hardlink if
--keep-chroots is not passed, because the full chroot implicitly will not
have any duplicate files. In this case, just delete the individual bundle
chroots and don't hardlink.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fixes a bug in the location for where the Mix Bundle List
(stored in the 'mixbundles' file) is read/write during 'mixer
bundle add' and 'mixer bundle remove' commands.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch adds a persistent "--offline" flag to the top-level mixer
command, which means every command can pass this flag to make mixer
work offline.
When mixer works offline, it skips caching upstream bundle definition
files. This has the implication that every bundle in the user's mix
must be available in local-bundles.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
LOCAL_BUNDLE_DIR is a new builder.conf field introduced in mixer 4.0.
It is included in the auto-generated conf from mixer init, and it is
referenced in the updated documentation.
Previously, however, it was marked as a "required" flag in the
ReadBuilderConf function. This meant that existing users, who did not
already have this value, were given an error message saying it was
missing, but no useful information about how to fix this.
This patch provides default fall-back value of the user's PWD +
"/local-bundles" if the field is missing. If this happens, a warning
message is output, informing the user to update their conf.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Kevin C. Wells