65 Commits
Author SHA1 Message Date
Tudor Marcu 7a36abfa0e Release v4.0.2
This release changes basic bundle validation to support previously
non-passing bundles in upstream so mixes with previous versions are not
so strictly filtered. Hardlinking only happens if --keep-chroots is
passed to avoid race conditions and wasting time hardlinking when it
does not need to happen. Issues with bundle list were fixed, offline
mode was added for mixer to skip caching upstream bundle definitions, a
default value for local-bundle directory was added, and other updates to
variables to match the new naming schemes 4.0.* introduced.

Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
2018-03-02 01:00:34 -08:00
Kevin C. Wells 009751357e Mixer bundle basic validation change
Previously, "basic" bundle validation included checking that the
bundle filename (currently used as the "bundle name") matched the
"Title" field in the bundle header (currently not used), and that
the "Title" field itself was valid. This was a step toward future
usage, where mixer will use the header "Title" itself.

This caused mixer to be incompatible with previous versions of CLR,
for which these filename-to-title missmatches were allowed and thus
present.

This patch moves this checking exclusively to "--strict" validation,
and removes bundle content validation from "basic" altogether. Basic
validation now only checks that the bundle filename is valid and that
the bundle contents syntax can be parsed.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-03-02 00:51:48 -08:00
Tudor Marcu 28f1a71e23 builder.go: Hardlink only with --keep-chroots
Due to using nosync, hardlinking could start before all of the individual
bundle chroots are actually removed. However, we do not need to hardlink if
--keep-chroots is not passed, because the full chroot implicitly will not
have any duplicate files. In this case, just delete the individual bundle
chroots and don't hardlink.

Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
2018-03-01 17:46:23 -08:00
Kevin C. Wells 3b37f2b4da Fix Mix Bundle List file bug
Fixes a bug in the location for where the Mix Bundle List
(stored in the 'mixbundles' file) is read/write during 'mixer
bundle add' and 'mixer bundle remove' commands.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-03-01 10:39:51 -08:00
Kevin C. Wells 25d3b0bdf6 Add an "offline" mode for mixer
This patch adds a persistent "--offline" flag to the top-level mixer
command, which means every command can pass this flag to make mixer
work offline.

When mixer works offline, it skips caching upstream bundle definition
files. This has the implication that every bundle in the user's mix
must be available in local-bundles.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-03-01 10:39:00 -08:00
Kevin C. Wells b392839537 Add a default value for LOCAL_BUNDLE_DIR
LOCAL_BUNDLE_DIR is a new builder.conf field introduced in mixer 4.0.
It is included in the auto-generated conf from mixer init, and it is
referenced in the updated documentation.

Previously, however, it was marked as a "required" flag in the
ReadBuilderConf function. This meant that existing users, who did not
already have this value, were given an error message saying it was
missing, but no useful information about how to fix this.

This patch provides default fall-back value of the user's PWD +
"/local-bundles" if the field is missing. If this happens, a warning
message is output, informing the user to update their conf.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-03-01 10:34:53 -08:00
Tudor Marcu d30d3949ab mixer: update RPMDIR and REPODIR variables
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
2018-02-28 15:07:13 -08:00
Tudor Marcu 516c24d10c Release v4.0.1
This release fixes the mixer-completion to install to a given optional prefix
so it can be built in different environments.

Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
2018-02-26 20:50:27 +00:00
Tudor Marcu ca27a1c2a1 Release v4.0.0
This release introduces a major version update with several new features
and package additions.

The original C swupd-server https://github.com/clearlinux/swupd-server and
bundle-chroot-builder https://github.com/clearlinux/bundle-chroot-builder
are now integrated into mixer itself under the swupd/ and builder/ packages,
and used as libraries instead of standalone binaries. The rewrite closely ties
the swupd-server and chroot-builder functionality directly into mixer, while
providing the benefits of the Go language. The new functionality lives side
by side with the original implementation for the time being, until further
testing is completed to guarantee consistent, stable behaviour. However,
the new features may (and should) be used by providing --new-swupd and
--new-chroots with the appropriate mixer subcommands. They will become the
default as the old standalone build programs are deprecated.

The second major update is a new CLI written using the Cobra framework.
Existing commands are only subtly different:
old			new
----			----
mixer build-chroots	mixer build chroots
mixer build-update	mixer build update
mixer init-mix		mixer init

-flags			--flags

In short, the command hierarchy is changed such that things like 'build'
are top level commands, and the things they build are exposed under them,
rather than making many hyphenated commands. Flags are implemented as
regular short and long options, where long options use two hyphens
instead of one.
New commands have been added to make bundle lifecycle management easier,
and to ensure bundles are manipulated correctly without manually copying
things in specific folders. See all new commands by typing "mixer" *enter*.

Miscellaneous fixes to things such as init creating the builder.conf and
doing more upfront work for the user have also been implemented. The
goal of these updates is to streamline the operation of mixer and
improve usability.

Many tests were added for the code base; unit tests for the new
swupd and chroot libraries, BAT tests to cover the interface and
functionality of the Mixer itself, and extensive linting were added to
catch regressions and enforce cleaner code.

Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
2018-02-26 18:23:12 +00:00
Matthew Johnson 36328f4f8a Add a timer to delta pack creation
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
2018-02-22 14:56:43 -08:00
Kevin C. Wells 1e980e0174 Add 'mixer bundle validate' command
New command checks bundle definition files for validity. Only local bundle files are
checked; upstream bundles are trusted as valid. Valid bundles yield no output.
Any invalid bundles will yield a non-zero return code.

Basic validation includes checking syntax and structure, that the bundle has
a valid name, and that the header 'Title' matches the bundle filename. Commands
like 'mixer bundle edit' run basic validation automatically.

An optional '--strict' flag allows you to additionally check that the other
bundle header flags are parsable and non-empty.

Passing '--all-local' will run validation on all bundles in local-bundles.

This patch also adds tests to check that bundle header parsing is working
and that different types of validation errors are indeed being caught.

Basic validation is added throughout the various 'mixer bundle' commands.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-02-21 15:59:04 -08:00
Caio Marcelo de Oliveira Filho 0ee338938e mixer: add 'versions' and 'versions update' command
Replace the workflow of manually updating the two files to get a new
version. The 'mixer versions' prints the relevant versions and 'mixer
versions update' provides a way to update those versions.

When updating the command does check the upstream format and do not
cross format bumps. So it helps preventing people generating wrong
content. The commands will provide a point to do more checks in the
future, like if there is a problem with the local bundles after an
update.

Also make the download function previously used to get latest file
reusable for other purposes, and returning error for non-OK status
codes.

To keep commands consistent both 'versions update' and 'init' will
accept --clear-version or --upstream-version (they are equivalent).

Updates #145.
Updates #59.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-21 11:53:56 -08:00
Caio Marcelo de Oliveira Filho 8d52b70f82 builder: parse mix and upstream versions upfront
Fail early if we can't parse those at ReadVersions time. Also remove
code in the individual functions that were doing these.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-21 11:53:56 -08:00
Matthew Johnson eba3bd9383 Improve progress output for chroot building
In particular add a "done" message to help with progress indication
during parallelized chroot builds.

Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
2018-02-16 16:52:29 -08:00
Caio Marcelo de Oliveira Filho 196181d8ef mixer: add a default value for mix version in init
And print the versions being used.

Fixes #123. (in combination with #153 that was already merged)

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 16:00:50 -08:00
Caio Marcelo de Oliveira Filho 99f543e8bf Add --chroot-workers to build
Make number of worker goroutines a parameter for building chroots.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 14:54:16 -08:00
Caio Marcelo de Oliveira Filho 0063f7a5f3 Add --delta-workers to build
Make the number of worker goroutines a parameter in delta creation and
add a flag to set that in Mixer. When zero (default) the flag sets the
number of workers to the number of CPUs.

Having a flag let the users to control better how much resources mixer
can/will take.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 14:41:28 -08:00
Caio Marcelo de Oliveira Filho d083c9427c Add --fullfile-workers to build
Make the number of worker goroutines a parameter in
swupd.CreateFullfiles and add a flag to set that. When zero (default)
the flag sets the number of workers to the number of CPUs.

Having a flag let the users to control better how much resources mixer
can/will take.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 14:41:28 -08:00
Caio Marcelo de Oliveira Filho 1b553a24d3 mixer: fix output of list when have included bundles
Use tabwriter to account for the size of all entries
printed (including the "included" ones that where not being accounted
before).

The output will be buffered (so it can calculate the columns) but for
the bundles list this is not an issue.

The code was also simplified to use bundle name directly instead of
taking the route of finding a *bundle and then using its name. They
should be the same.

Before:

bootloader     (upstream)
c-basic        (upstream)
desktop        (upstream)
desktop-apps   (upstream) (included)
desktop-assets (upstream) (included)
desktop-gnomelibs (upstream) (included)
desktop-locales (upstream) (included)
kernel-native  (upstream)
libX11client   (upstream) (included)
os-core        (upstream)
os-core-update (upstream)
python3-basic  (upstream) (included)
sysadmin-basic (upstream) (included)

After:

bootloader        (upstream)
c-basic           (upstream)
desktop           (upstream)
desktop-apps      (upstream) (included)
desktop-assets    (upstream) (included)
desktop-gnomelibs (upstream) (included)
desktop-locales   (upstream) (included)
kernel-native     (upstream)
libX11client      (upstream) (included)
os-core           (upstream)
os-core-update    (upstream)
python3-basic     (upstream) (included)
sysadmin-basic    (upstream) (included)

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 14:30:40 -08:00
Caio Marcelo de Oliveira Filho 6f226e7223 mixer: add --report flag to build delta-packs
If flag is passed we print the report with all the files in the
manifest and whether they were included or not. The report stays right
above the count of fullfiles and deltas.

The output looks like (modified to fit in the commit message):

Creating delta packs from 10 to 20
  Creating delta pack for bundle caio from 10 to 20
    Pack report:
      /usr/bin/2to3                      packed fullfile (from chroot)
      /usr/bin/c_hash                    not packed (already in from manifest)
      /usr/bin/clrtrust                  not packed (already in from manifest)
      /usr/bin/corelist                  not packed (already in from manifest)
      /usr/bin/cpan                      not packed (already in from manifest)
      /usr/bin/ebrowse                   not packed (file deleted)
      /usr/bin/emacs                     not packed (file deleted)

(...)

  Creating delta pack for bundle os-core-update-index from 10 to 20
    Pack report:
      /usr/share/clear/os-core-update-index packed delta (10-20-3a480a802244c61eefd24009b0b24d1ba6d822aa4358ef51bfd4492244589a5e-74fba7851c7bb474588f0321fe4715c8e6f83bba8889cc917f6a0490f3a777b7)

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 14:00:57 -08:00
Caio Marcelo de Oliveira Filho f8e2184deb swupd: return information about fullfile creation
Fix an old TODO, this gives visibility in every build about the
compression algorithms being used.

Note the accounting "gzip" (and not "external-gzip") because at the
moment it is being used for links and directories. It probably should
just migrate to use the external gzip later.

Output looks like:

=> CREATE FULLFILES
- Already created: 0
- Not compressed:  0
- Compressed
  - external-bzip2       40
  - gzip                 389
  - external-xz          2520
  - external-gzip        953
Total fullfiles: 3902

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-02-16 10:37:00 -08:00
Kevin C. Wells 1a536eca69 Fix upstream bundle cache bug
If you update your upstream version number, the tool is supposed to
automatically pull the new definition files for you. There were two
places where it wasn't currently doing that yet, which is now fixed.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-02-15 15:41:08 -08:00
Matthew Johnson b7c2cf8d15 Add timing information for chroot building step
This timing should be printed to assist in evaluating the move to the
new chroot builder.

Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
2018-02-15 12:17:23 -08:00
Kevin C. Wells 2ecb969d62 Add mixer bundle remove command
Removes bundles from your mix by modifying the Mix Bundle List
(stored in the 'mixbundles' file). The Mix Bundle List is parsed, the bundles
are removed, and the resultant list is written back out in sorted order. If
bundles do not exist in the mix, they are skipped.

Passing '--local' will also remove the corresponding bundle definition file from
local-bundles, if it exists. Please note that this is an irrevocable step.

'--mix' defaults to true. Passing '--mix=false' will prevent the bundle from
being removed from your Mix Bundle List. This is useful when used in conjunction
with '--local' to *only* remove a bundle from local-bundles. If the bundle being
removed is an edited version from upstream, the bundle will remain in your mix
and now reference the original upstream version. If the bundle was custom, and
no upstream alternative exists, a warning will be returned.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-02-13 14:52:56 -08:00
Kevin C. Wells 502f6c2b5b Add 'create new bundle' functionality
Add 'create new bundle' functionality to 'mixer bundle edit'. Now, if a
bundle is not found in local-bundles or upstream-bundles, it is treated
as a new bundle, and a blank bundle definition file template is created
for the user to edit.

Also minor syntax optimization on editing switch.
2018-02-13 14:52:30 -08:00
Kevin C. Wells 424031e5aa Add mixer bundle edit command
Adds new command, 'mixer bundle edit', that allows a user to edit local
and upstream bundle definition files. This command will locate the
bundle (looking first in local-bundles, then in upstream-bundles), and launch
an editor to edit it. If the bundle is only found upstream, the bundle file will
first be copied to your local-bundles directory for editing. When the editor
closes, the bundle file is then parsed for validity.

The editor is configured via environment variables. VISUAL takes precedence to
EDITOR. If neither are set, the tool defaults to nano. If nano is not installed,
the tool will skip editing, and act as if '--copy-only' had been passed.

Passing '--copy-only' will suppress launching the editor, and will thus only copy
the bundle file to local-bundles if it is only found upstream. This can be
useful if you want to add a bundle to local-bundles, but wish to edit it at a
later time.

Passing '--add' will also add the bundle(s) to your mix. Please note that
bundles are added after all bunles are edited, and thus will not be added if any
errors are encountered earlier on.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-02-13 14:52:30 -08:00
Caio Marcelo de Oliveira Filho 9dfbf7f65f chroots: always use dnf
DNF already works in Clear and at the moment Yum is not working, so it
is a good time to stop calling it.

Drop the --packager, if an alternative version of DNF have to be used,
just put it in PATH before the system one.

Fixes #115.
2018-02-13 14:51:53 -08:00
Caio Marcelo de Oliveira Filho 836281baf1 builder: order the pack generation 2018-02-12 14:39:10 -08:00
Kevin C. Wells 2c9c9144c7 Refactor mix bundles
This commit refactors the way you specify what bundles are included
in your mix, as well as how the mix-bundles directory gets
generated.

Major changes in this commit:
- Introduces the Mix Bundles List, a newline-separated file in the
  working directory that lists the bundles that must be in the mix.
  Other bundles (included by those in the list) will be added
  automatically.
- Introduces the local-bundles directory that stores any bundles
  that the user created or that have been edited from upstream.
  Bundles in the local-bundles directory take precedence over
  upstream bundles of the same name.
- Generates the mix-bundles/ directory (used as input to the BCB)
  automatically on-the-fly when chroots are built. The Mix Bundles
  List is recursed to find all bundles needed for the mix, and the
  bundle definition files are copied from either the local or
  upstream bundles.
- If '--new-chroots' is passed, the mix-bundles/ directory is
  skipped entirely, as the new BCB is capable of reading the
  files from anywhere.
- Modifies the 'mixer bundle add' command to now edit the Mix
  Bundles List. Bundles are verified to exist in either the local
  or upstream bundles. The user is informed whether the added
  bundles came from local or upstream.
- Adds a new 'list' command to 'mixer bundle' that prints out
  the bundles in the Mix Bundle List (including those
  recursively included), all bundles available in local bundles,
  or all bundles available in upstream bundles. The mix bundle
  list, bundle names are annotated with information about their
  origin (local or upstream) and whether they are part of the
  Mix Bundle List or just included. For the local and upstream
  lists, bundle names are annotated with whether or not they are
  currently included in the mix. All three of the above list
  types support a '--tree' option that prints them as a formatted
  tree view, visually showing the include relationship for bundles
  and how the full list is being generated.

Minor changes in this commit:
- As the mix-bundles directory is now created on-the-fly, it is no
  longer tracked as a git repository. Instead, the entire mixer
  working directory is tracked, with temporary, tool-generated
  files ignored. This allows one to track and correlate the Mix
  Bundle List, mix version, and upstream version as they are changed.
- The above-mentioned git revision tracking is now optional, and
  is only set up if '--git' is passed to 'mixer init'. 'mixer
  bundle add' still supports '--git', which mirrors this behavior.
- The '--all' flags for 'mixer init' and 'mixer bundle add' have
  been split up into '--all-local' and '--all-upstream'. This
  allows a user to add or start with all local or upstream bundles,
  or both.
2018-02-08 18:34:34 -08:00
Kevin C. Wells 6f058e08bd Support (and default to) "latest" in mixer init clear-version
Adds support for the keyword "latest" for `--clear-version` when
running mixer init. This becomes the default value if the flag is
omitted.

'--clear-version latest' will fetch the latest published upstream
version number, according to the value found via 'upstreamurl'

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-02-08 10:59:51 -08:00
Kevin C. Wells 1cd3d083dd Rename, cleanup, and remove .mixer directory
This commit has three main side effects:
1) The .clearversion, .mixversion, and .clearurl files are no longer
hidden files. Deprecation code has been added to work with the old
versions if they exist, but print a deprecation notice.

They were likely originally hidden to declutter the workspace, but
they are (for now) files the user needs to edit directly, and thus
should not be hidden.

2) The clearversion and clearurl files has been renamed
upstreamversion and upstreamurl, as part of a larger goal to
accomodate derivatives-of-derivatives, for whom upstream is not
mainline CLR. (The internal fields on the Builder object have been
renamed as well.) Deprecation code has been added to work with the old
versions if they exist, but print a deprecation notice.

3) Per feedback, the .mixer directory has been removed.

Two of the goals the directory was introduced to solve will instead
be met in different ways:

i) Hiding temporary, sausage-making files. The mix-bundles/
directory will soon cease to exist entirely, once the BCB rewrite
moves out from behind UseNewChrootBuilder. upstream-bundles/ also
contains temporary, tool-generated files, but people have expressed
interest (or at least tolerance) of these not being hidden away.

ii) Organizing/decluttering the workspace. As there are numerous
different configuration files (including the mixversion,
clearversion, clearurl, and forthcoming mix bundle list), having
them in the .mixer directory provided a better form of decluttering
than having the files as hidden dot-files. However, ideally these
files will soon be merged into the builder.conf, when the that
file gets rewritten as a TOML file.

This commit also has minor side effects:
1) General code clean-up (e.g., use of filepath.Join())
2) Change Builder struct field names to match capitalization
conventions
3) Moves the location of InitMix within the file for organizational
purposes.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-02-08 10:47:48 -08:00
Kevin C. Wells 648994a5ff Refactor bundleset
Refactor bundleSet code to:
1) Loosen definition of bundleSet to simply a set, without
additional constraints. This allows the bundleSet to be built
incrementally, instead of all-at-once, which in turn allows us to
not duplicate bundle parsing that happens elsewhere in the code.
The constraints (completeness and cycle-free), as well as the
computation of the AllPackages field in each bundle, are moved to
a standalone "validateAndFillBundleSet" function.

2) Separate out bundle file parsing into a standalone function,
which allows code reuse elsewhere in the code.

3) Move completeness checking into the sort function, thus
eliminating the need to cycle through the set an additional time.

4) Add additional name validity checking, disallowing bundles with
the reserved names "full" and "MoM"

The above changes do not change the functionality of bundleset,
but improve efficiency and allow code reuse.

All tests have been updated, and additional tests of the bundle
file parsing function have been added.
2018-02-07 13:48:28 -08:00
Kevin C. Wells 64e3c26c1d Fix add all bundle during init bug
Fixes a bug in AddBundles where it didn't clear out the bundles
slice correctly if 'all' was passed AND a bundle list was passed
in. This case never happened with 'mixer bundle add --all', but
did happen with 'mixer init --all'.
2018-01-31 13:44:55 -08:00
Caio Marcelo de Oliveira Filho 0ba140fb67 mixer: accept --packager flag when building chroots
This lets override the default choice of packager for performing build
chroots. Makes easy to test dnf with --packager=dnf. The flag is
ignored when not using --new-chroots.

Also prints the actual command line being used.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-30 11:36:52 -08:00
Caio Marcelo de Oliveira Filho 6c347afc4c builder: fix generation of versions file when using dnf
Add one more entry to be skipped. This fixes the scenario of using dnf
by changing yum to be a symlink to it.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-29 11:14:05 -08:00
Caio Marcelo de Oliveira Filho ee63b720d2 builder: port chroot building to Go
This rewrites bundle-chroot-builder.py in Go to be part of Mixer
code. Mixer is the only user of that software, and both Mixer and
bundle-chroot-builder.py the same configuration file, with overlapping
fields.

Main differences from bundle-chroot-builder.py:

- New bundleset type was added, that cares about collecting as much
  information as possible from the bundles themselves. This type and
  related functions also sets us up for success when upcoming changes
  to how bundles are specified happen. There is no assumption all
  bundles are in the same directory.

- We are not using m4, instead a bundleset takes care of parsing. If
  format of individual bundle files change. The upside is that we can
  give nicer error messages, specially for the circular case.

- Read the configuration file directly (with go-ini) to peek at values
  that Mixer didn't read before. Done that to avoid conflicting with
  existing patch in-flight that parses configuration.

- Some individual steps were reordered for code clarity. E.g.: since
  we have bundleset, we can upfront generate all the *-include files.

- Fixed the output for versions file. Due to the way yum list output
  works, parsing it is not very friendly. Comments around the code
  tells the story.

- Removed the network testing step. It wasn't covering every case in
  the Python version, so I'm leaning to let the failure come from
  yum/dnf itself. I'm usually in favor of such early tests, but in
  this case the price of parsing yet another config file didn't felt
  worth.

- Removed the "yum clean all" step from the bootstrap. There isn't any
  cache at that point, and the next yum call will bootstrap the
  necessary files for yum to operate.

- Removed generation of files-* files (and the pkgmap-* files used to
  generate them). I couldn't find any tool or team making use of this
  information. Those (or their content) might be relevant in future
  changes to use a single chroot, but we should add when we need them.

- Added more detailed commentary to individual steps, collecting
  information from the developers of bcb and related software.

- The port still don't parallelize the work into multiple
  goroutines. I plan to do this in a similar way than what was done in
  CreateFullfiles, but in a separated patch.

Fixes #42.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-26 13:05:09 -08:00
Rodrigo Chiossi b2b8c66bc5 builder: propagate error to caller
This patch removes the last occurrences where an error would be handled
by builder itself and cause the program to Exit. Now the error is always
propagated to the caller.

Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
2018-01-24 12:14:55 -08:00
Rodrigo Chiossi ac1d616f78 Add builder.conf creation to init-mix
When running mixer init-mix, if no builder.conf exists in the provided
builderconf path, create a template in the given path and configure it
using the active directory as base path for the variables.
With this path, init-mix will also create the rpm/ and local/
directories and reference them in the generated builder.conf
2018-01-24 11:52:22 -08:00
Kevin C. Wells 94d71ff45a Remove 'mixer bundle get' command
The previous commits make upstream bundle definition caching
automatically handled by the mixer commands that need them,
and thus this command is obsolete.

This command also had dubious side effects: while it was
described as fetching upstream bundles, it *also* set up
the mix-bundles directory. This side effect behavior is
now taken care of by init.

This commit also makes cosmetic edit to 'mixer init-mix',
renaming it to 'mixer init'.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-01-23 13:39:40 -08:00
Kevin C. Wells 5c0064a965 Refactor download of upstream bundles
Split up the download/unpacking of upstream bundles and the
creation of the mix-bundles directory.

Fetching upstream bundles now lives in a standalone function that
is called by any function that relies on having upstream bundle
definitions. This means the tool automatically handles the caching
of upstream bundles, making the 'mixer bundle get' command
obsolete.

Creation of the mix-bundles directory has been moved to InitMix.

This patch also includes general cleanup and error handling,
especially for InitMix and AddBundles.

This patch has the following side-effects:
1) The upstream bundles are now considered temporary, sausage-
making data. As such, they now live in .mixer/upstream-bundles/,
and the .tar.gz file used to fetch them is cleaned up.
2) The method for fetching upstream bundles has an optional
parameter ('prune') that will clean up the bundle cache for other
versions of upstream. This should be considered the default value
as part of standard clean-up behavior; a value of false should only
be used in cases where a method needs to have multiple versions
of upstream bundles cached simultaneously.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-01-23 13:39:40 -08:00
Kevin C. Wells 69c49983e8 Replace exec to tar with native function
When unpacking upstream bundle defintions, previously an exec
was made to 'tar'. This patch replaces the exec call with a
native function that uses the Go tar and gzip packages instead.

Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
2018-01-23 13:39:40 -08:00
Mark D Horn 851a746073 Ensure required builder.conf variables are present
Check for required variables in the builder.conf when reading.

Fixes #63

Signed-off-by: Mark D Horn <mark.d.horn@intel.com>
2018-01-22 16:40:52 -08:00
Matthew Johnson 8f99f3feda Fix error message for mixer init-mix
Fixes #97
Print the correct error message with the correct flag names when
--clear-version or --mix-version is not supplied. Uses
cobra.MarkFlagRequired to mark these two flags as required.

Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
2018-01-18 18:02:23 -08:00
Matthew Johnson 90693da8fe Write update metadata to output directory
Instead of requiring the new swupd code to create the "format" and
"swupd-server-src-version" metadata files enable BuildUpdate to do so.

Because the mixer version is now used instead of a swupd-server version
write the mixer-tools version to a "mixer-src-version" file. Move the
Version const to builder.go so it can be accessed trivially. Update the
Makefile parsing to point to builder.go and update the access method in
the top-level mixer/cmd/root.go.

Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
2018-01-17 13:02:15 -08:00
Caio Marcelo de Oliveira Filho b1d5344dd0 builder, swupd: add mixer build delta-packs command
This is a --new-swupd replacement of mixer-pack-maker.sh script. There
are two ways to specify what delta packs to make: by setting a --from
version or by asking for (up to) K --previous-versions.

Unlike the script, only one --from is supported, if multiple specific
versions are needed, mixer must be called multiple times. The
rationale is the multiple --from was used to simulate
--previous-versions (by having the caller figuring out them), so not
very important anymore.

The implementation uses swupd.FindBundlesToPack to figure out
what (bundle, from, to) combinations it needs to build, then use
swupd.CreatePack to do the work.

The FindBundlesToPack was changed to take manifests instead of version
numbers and state dirs, the test program was updated accordingly.

Fixes #83.
Fixes #12.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-17 11:25:48 -08:00
Caio Marcelo de Oliveira Filho 0f38e81280 swupd: make CreatePacks take the chrootDir
In CreatePacks, instead of a specific full chroot, take the main
chrootDir and build the path to the full chroot. This will make easier
later to pass the config around (that will have those high level
paths).

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-17 11:25:48 -08:00
Caio Marcelo de Oliveira Filho 23c7879095 builder: create compressed tar for each manifest
For the Manifest.MoM, include the signature file. These compressed
files are used by swupd client.

Fixes #88.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-17 10:43:36 -08:00
Caio Marcelo de Oliveira Filho d7728d3e03 builder: split the old and new swupd usage
Create two separate functions for each code path. The major changes
were:

- Take advantage of the MoM structure given by the CreateManifests: in
  fullfile generation that now doesn't need to parse the full manifest,
  and in zero pack generation that doesn't need to parse the MoM.

- Simplify the code for zero pack generation, we don't need to
  FindBundlesToPack because we already know its all of them. That
  function will be useful for delta packs only.

- Move some path verification/creation to CreateFullfiles.

This commit also add some informational output when using new swupd
implementation.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-17 10:43:36 -08:00
Caio Marcelo de Oliveira Filho 747caeb606 builder: delete chroots later
Delete them right before the hardlink step (where it does make a
difference). It will make easier to split the two code paths right
above it (new and old swupd-server).

Neither signing, full file creation or zero pack should be affected by
it. So the final result should not change.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-17 10:43:36 -08:00
Caio Marcelo de Oliveira Filho 93fec93ee4 builder: use swupd package to create manifests when --new-swupd
Use swupd.CreateManifests function. The behavior should be the same as
running without --new-swupd, except that tar files are not generated
for the manifests yet and for now format is only numeric (both issues
opened).

Note that I'm _explicitly_ keeping the code inline until we add all
the pieces, so we can have a better vision on how it is used, how to
organize better some supporting data (like strings with directories
and so on) and review the functions and their arguments in swupd
packages.

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
2018-01-17 10:43:36 -08:00