This release changes basic bundle validation to support previously
non-passing bundles in upstream so mixes with previous versions are not
so strictly filtered. Hardlinking only happens if --keep-chroots is
passed to avoid race conditions and wasting time hardlinking when it
does not need to happen. Issues with bundle list were fixed, offline
mode was added for mixer to skip caching upstream bundle definitions, a
default value for local-bundle directory was added, and other updates to
variables to match the new naming schemes 4.0.* introduced.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Previously, "basic" bundle validation included checking that the
bundle filename (currently used as the "bundle name") matched the
"Title" field in the bundle header (currently not used), and that
the "Title" field itself was valid. This was a step toward future
usage, where mixer will use the header "Title" itself.
This caused mixer to be incompatible with previous versions of CLR,
for which these filename-to-title missmatches were allowed and thus
present.
This patch moves this checking exclusively to "--strict" validation,
and removes bundle content validation from "basic" altogether. Basic
validation now only checks that the bundle filename is valid and that
the bundle contents syntax can be parsed.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Due to using nosync, hardlinking could start before all of the individual
bundle chroots are actually removed. However, we do not need to hardlink if
--keep-chroots is not passed, because the full chroot implicitly will not
have any duplicate files. In this case, just delete the individual bundle
chroots and don't hardlink.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fixes a bug in the location for where the Mix Bundle List
(stored in the 'mixbundles' file) is read/write during 'mixer
bundle add' and 'mixer bundle remove' commands.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This patch adds a persistent "--offline" flag to the top-level mixer
command, which means every command can pass this flag to make mixer
work offline.
When mixer works offline, it skips caching upstream bundle definition
files. This has the implication that every bundle in the user's mix
must be available in local-bundles.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Apparently linting was failing on the mixer-completion/
cmd/completion.go file on the master branch? This fixes it.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
LOCAL_BUNDLE_DIR is a new builder.conf field introduced in mixer 4.0.
It is included in the auto-generated conf from mixer init, and it is
referenced in the updated documentation.
Previously, however, it was marked as a "required" flag in the
ReadBuilderConf function. This meant that existing users, who did not
already have this value, were given an error message saying it was
missing, but no useful information about how to fix this.
This patch provides default fall-back value of the user's PWD +
"/local-bundles" if the field is missing. If this happens, a warning
message is output, informing the user to update their conf.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release fixes the mixer-completion to install to a given optional prefix
so it can be built in different environments.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Adds a '--path' override, allowing you to specify the full path
to where the destination file will be written.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This release introduces a major version update with several new features
and package additions.
The original C swupd-server https://github.com/clearlinux/swupd-server and
bundle-chroot-builder https://github.com/clearlinux/bundle-chroot-builder
are now integrated into mixer itself under the swupd/ and builder/ packages,
and used as libraries instead of standalone binaries. The rewrite closely ties
the swupd-server and chroot-builder functionality directly into mixer, while
providing the benefits of the Go language. The new functionality lives side
by side with the original implementation for the time being, until further
testing is completed to guarantee consistent, stable behaviour. However,
the new features may (and should) be used by providing --new-swupd and
--new-chroots with the appropriate mixer subcommands. They will become the
default as the old standalone build programs are deprecated.
The second major update is a new CLI written using the Cobra framework.
Existing commands are only subtly different:
old new
---- ----
mixer build-chroots mixer build chroots
mixer build-update mixer build update
mixer init-mix mixer init
-flags --flags
In short, the command hierarchy is changed such that things like 'build'
are top level commands, and the things they build are exposed under them,
rather than making many hyphenated commands. Flags are implemented as
regular short and long options, where long options use two hyphens
instead of one.
New commands have been added to make bundle lifecycle management easier,
and to ensure bundles are manipulated correctly without manually copying
things in specific folders. See all new commands by typing "mixer" *enter*.
Miscellaneous fixes to things such as init creating the builder.conf and
doing more upfront work for the user have also been implemented. The
goal of these updates is to streamline the operation of mixer and
improve usability.
Many tests were added for the code base; unit tests for the new
swupd and chroot libraries, BAT tests to cover the interface and
functionality of the Mixer itself, and extensive linting were added to
catch regressions and enforce cleaner code.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
The min version of these validation builds should be the FROM version to
mirror actual build process.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch introduces a new 'mixer-completion' executable that generates bash
and zsh completion files for mixer. The executable is built via 'make', but not
installed via 'make install'. Instead, 'make install' runs the executable,
which writes the bash and zsh completion files to /etc/bash_completion.d/ and
/usr/share/zsh/site-functions/, respectively.
This patch uses Cobra's built-in bash/zsh completion generation functionality.
Some Cobra applications choose to have this completion-file generation as a
permanent command off the root command (e.g., 'kubectl completion bash'). This
patch makes this a stand-alone executable to avoid cluttering the main 'mixer'
tool with this functionality. Many Cobra applications also leave creating/
installing these completion files up to the user as a manual, secondary step.
This patch completes these tasks at mixer install time.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
New command checks bundle definition files for validity. Only local bundle files are
checked; upstream bundles are trusted as valid. Valid bundles yield no output.
Any invalid bundles will yield a non-zero return code.
Basic validation includes checking syntax and structure, that the bundle has
a valid name, and that the header 'Title' matches the bundle filename. Commands
like 'mixer bundle edit' run basic validation automatically.
An optional '--strict' flag allows you to additionally check that the other
bundle header flags are parsable and non-empty.
Passing '--all-local' will run validation on all bundles in local-bundles.
This patch also adds tests to check that bundle header parsing is working
and that different types of validation errors are indeed being caught.
Basic validation is added throughout the various 'mixer bundle' commands.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Make the worker defaults lower for more sane systems with a comment
instructing users to increase these numbers on beefier systems.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This script creates builds from $1 to $2 if they exist upstream and
performs an os-install, bundle-add, bundle-remove, verify, and update on
each.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When files are subtracted from the manifest also subtract the size of
the file from manifest.Header.ContentSize. This makes the ContentSize
field only reflect the size of the files listed in the current manifest
and not subtracted files. One can then calculate the complete installed
size of that bundle by adding the ContentSize of all the includes
because there will be no overlap from subtracted files.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Replace the workflow of manually updating the two files to get a new
version. The 'mixer versions' prints the relevant versions and 'mixer
versions update' provides a way to update those versions.
When updating the command does check the upstream format and do not
cross format bumps. So it helps preventing people generating wrong
content. The commands will provide a point to do more checks in the
future, like if there is a problem with the local bundles after an
update.
Also make the download function previously used to get latest file
reusable for other purposes, and returning error for non-OK status
codes.
To keep commands consistent both 'versions update' and 'init' will
accept --clear-version or --upstream-version (they are equivalent).
Updates #145.
Updates #59.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Fail early if we can't parse those at ReadVersions time. Also remove
code in the individual functions that were doing these.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
This helps when doing more than one mix back-to-back or when packages
are installed in more than one bundle.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
We want information about the link, not the linked file, when doing a
stat on files in the update. The File.Info field holds FileInfo about
the file, not the linked file.
Doing a 'Stat' instead of an 'Lstat' caused an issue where a link was
changed in an update but the target file was not found in the full
chroot due to the link expecting the target to be in / (outside the full
chroot). If the target does not exist on the build system the 'Stat'
will cause a panic because the target file can not be found.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
In particular add a "done" message to help with progress indication
during parallelized chroot builds.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
These tests should be reporting on the structure and correctness of mixer
and its user interface. Due to them running very slowly on the Travis CI
environment, they are not run by default, but can be run locally with
make batcheck.
This patch also introduces a docker container to do the build, so we do not
have to worry about running this in different environments such as Travis
CI. The tests are thus confined to a strict, well-known workspace,
and we can avoid many dependency problems.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
And print the versions being used.
Fixes#123. (in combination with #153 that was already merged)
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Make the number of worker goroutines a parameter in delta creation and
add a flag to set that in Mixer. When zero (default) the flag sets the
number of workers to the number of CPUs.
Having a flag let the users to control better how much resources mixer
can/will take.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Make the number of worker goroutines a parameter in
swupd.CreateFullfiles and add a flag to set that. When zero (default)
the flag sets the number of workers to the number of CPUs.
Having a flag let the users to control better how much resources mixer
can/will take.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Use tabwriter to account for the size of all entries
printed (including the "included" ones that where not being accounted
before).
The output will be buffered (so it can calculate the columns) but for
the bundles list this is not an issue.
The code was also simplified to use bundle name directly instead of
taking the route of finding a *bundle and then using its name. They
should be the same.
Before:
bootloader (upstream)
c-basic (upstream)
desktop (upstream)
desktop-apps (upstream) (included)
desktop-assets (upstream) (included)
desktop-gnomelibs (upstream) (included)
desktop-locales (upstream) (included)
kernel-native (upstream)
libX11client (upstream) (included)
os-core (upstream)
os-core-update (upstream)
python3-basic (upstream) (included)
sysadmin-basic (upstream) (included)
After:
bootloader (upstream)
c-basic (upstream)
desktop (upstream)
desktop-apps (upstream) (included)
desktop-assets (upstream) (included)
desktop-gnomelibs (upstream) (included)
desktop-locales (upstream) (included)
kernel-native (upstream)
libX11client (upstream) (included)
os-core (upstream)
os-core-update (upstream)
python3-basic (upstream) (included)
sysadmin-basic (upstream) (included)
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
When 'mixer add-bundle' was first introduced, the list of bundles
being added was implemented as a single-argument comma-separated
list. It is more intuitive to specify bundles as separate arguments,
and this is the format all of the new 'mixer bundle ___' commands are
taking. (This also simplifies forthcoming mixer autocomplete code.)
This patch changes 'mixer bundle add' to support this new usage.
To make it more clear how bundles are to be specified, the "Usage"
string of help output has been updated for all commands to comply with
more standard conventions. The output now uses square brackets to
indicate optional elements, ellipses to indicate optional expansion,
and angle brackets to indicate variables as opposed to literal constants.
For example:
Usage: mixer bundle add <bundle> [<bundle>...] [flags]
and
Usage: mixer bundle list [mix|local|upstream] [flags]
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
If flag is passed we print the report with all the files in the
manifest and whether they were included or not. The report stays right
above the count of fullfiles and deltas.
The output looks like (modified to fit in the commit message):
Creating delta packs from 10 to 20
Creating delta pack for bundle caio from 10 to 20
Pack report:
/usr/bin/2to3 packed fullfile (from chroot)
/usr/bin/c_hash not packed (already in from manifest)
/usr/bin/clrtrust not packed (already in from manifest)
/usr/bin/corelist not packed (already in from manifest)
/usr/bin/cpan not packed (already in from manifest)
/usr/bin/ebrowse not packed (file deleted)
/usr/bin/emacs not packed (file deleted)
(...)
Creating delta pack for bundle os-core-update-index from 10 to 20
Pack report:
/usr/share/clear/os-core-update-index packed delta (10-20-3a480a802244c61eefd24009b0b24d1ba6d822aa4358ef51bfd4492244589a5e-74fba7851c7bb474588f0321fe4715c8e6f83bba8889cc917f6a0490f3a777b7)
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Fix an old TODO, this gives visibility in every build about the
compression algorithms being used.
Note the accounting "gzip" (and not "external-gzip") because at the
moment it is being used for links and directories. It probably should
just migrate to use the external gzip later.
Output looks like:
=> CREATE FULLFILES
- Already created: 0
- Not compressed: 0
- Compressed
- external-bzip2 40
- gzip 389
- external-xz 2520
- external-gzip 953
Total fullfiles: 3902
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
This is different than the old swupd-server. Added a note to the
imported test.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Constant was added in two different areas of the code, but has the
same purpose. Prefer the one with more descriptive name.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
If you update your upstream version number, the tool is supposed to
automatically pull the new definition files for you. There were two
places where it wasn't currently doing that yet, which is now fixed.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Unsetting the rename-to file type causes a file with no flags set (e.g.
....). Do not unset the file type when deprecating a rename-to file,
(e.g. F..r). Only unset the file type when deprecating a rename-from
file (e.g. .d.r).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Two issues were fixed here. The first issue was that the wrong manifest
subtraction function was being called for the recursive case.
The second issue was that subtractManifests was called for each bundle
in the same loop in which includes were being processed, so not all
nested includes would be processed by the time subtraction happened.
This was fixed by doing subtraction in a following loop.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Removes bundles from your mix by modifying the Mix Bundle List
(stored in the 'mixbundles' file). The Mix Bundle List is parsed, the bundles
are removed, and the resultant list is written back out in sorted order. If
bundles do not exist in the mix, they are skipped.
Passing '--local' will also remove the corresponding bundle definition file from
local-bundles, if it exists. Please note that this is an irrevocable step.
'--mix' defaults to true. Passing '--mix=false' will prevent the bundle from
being removed from your Mix Bundle List. This is useful when used in conjunction
with '--local' to *only* remove a bundle from local-bundles. If the bundle being
removed is an edited version from upstream, the bundle will remain in your mix
and now reference the original upstream version. If the bundle was custom, and
no upstream alternative exists, a warning will be returned.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Add 'create new bundle' functionality to 'mixer bundle edit'. Now, if a
bundle is not found in local-bundles or upstream-bundles, it is treated
as a new bundle, and a blank bundle definition file template is created
for the user to edit.
Also minor syntax optimization on editing switch.
Adds new command, 'mixer bundle edit', that allows a user to edit local
and upstream bundle definition files. This command will locate the
bundle (looking first in local-bundles, then in upstream-bundles), and launch
an editor to edit it. If the bundle is only found upstream, the bundle file will
first be copied to your local-bundles directory for editing. When the editor
closes, the bundle file is then parsed for validity.
The editor is configured via environment variables. VISUAL takes precedence to
EDITOR. If neither are set, the tool defaults to nano. If nano is not installed,
the tool will skip editing, and act as if '--copy-only' had been passed.
Passing '--copy-only' will suppress launching the editor, and will thus only copy
the bundle file to local-bundles if it is only found upstream. This can be
useful if you want to add a bundle to local-bundles, but wish to edit it at a
later time.
Passing '--add' will also add the bundle(s) to your mix. Please note that
bundles are added after all bunles are edited, and thus will not be added if any
errors are encountered earlier on.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
DNF already works in Clear and at the moment Yum is not working, so it
is a good time to stop calling it.
Drop the --packager, if an alternative version of DNF have to be used,
just put it in PATH before the system one.
Fixes#115.
When RunCommandOutput (or RunCommandSilent) fails, only add the STDOUT
and STDERR to the error string if they have any content (otherwise
omit).
Also add some text without a newline in the end of the string to allow
the underlying error to be printed correctly. If you've seen lines
with ": exit status NNN" this last change will make things looks
better.
In RunCommand, simplify the error message taking in account that the
wrap will append the underlying error when printed.
During pack creation, after linking, make sure that we don't have
completely duplicated delta entries. These were causing a race when
multiple goroutines were creating the deltas.
The patch brings the creation of delta.Path earlier, and use that as a
way to identify redundancies. Also make a small change to not compute
paths in createDelta if the file already exists.
Note that later steps might still want to deduplicate so we have only
one delta pointing to a HASH (since that's sufficient for the client),
but that should take into account the size of each of the diffs.
Some bundles have uppercase letters in their names. DO NOT make these
lowercase - this has the result of a manifest not being generated for
them because it cannot find the chroot to walk.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Tudor Marcu