This release includes multiple fixes to mixer's MCA along with updated
versions of mixer's dependencies.
Dependencies update:
- github.com/BurntSushi/toml: v0.3.0 -> v0.3.1
- github.com/spf13/cobra: 0.0.1 -> 0.0.5
- github.com/go-ini/ini: 1.32.0 -> 1.42.0
- github.com/pkg/errors: 0.8.0 -> 0.8.1
MCA Fixes and Improvements:
- Skip files with blacklisted characters
- Use resolveFileName function to remove duplicate code
- Use recursive includes list for file/pkg subtraction
- Serialize package downloads
- Minor statistics reporting improvements
- Sort bundles when printing statistics
- Add documentation
- Support repo/baseurl overrides for from and to versions
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Mixer does not create manifest entries for files with blacklisted
characters, so MCA should not include files with blacklisted
characters in the comparison.
Signed-off-by: John Akre <john.w.akre@intel.com>
When subtracting files and packages each manifest in the recursive
includes list must be checked to consistently subtract files/packages
correctly.
Signed-off-by: John Akre <john.w.akre@intel.com>
There was a race condition where the rpm command would try to query
corrupt (partially downloaded) packages. By serializing package
downloads, this race condition is no longer an issue. Also,
parallelized package downloads did not significantly improve
performance, so this change will not be a noticeable performance
regression.
Signed-off-by: John Akre <john.w.akre@intel.com>
The following minor statistics reporting changes were added to improve
statistics readability:
* Add colons to the added/deleted package sections
* Add space between '|' character and bundle name
Signed-off-by: John Akre <john.w.akre@intel.com>
In some cases a single DNF config file is not sufficient to correctly
download packages for the mix versions specified by the from and to
flags. The --from-repo-url and --to-repo-url flags were added to
temporarily override the baseurl value for the specified repo in the DNF
config when downloading packages for the from and to versions respectively.
Multiple instances of these flags can be used to override additional
repo/baseurl pairs. An example flag format is shown below:
--from-repo-url <repo>=<URL>
--to-repo-url <repo>=<URL>
Signed-off-by: John Akre <john.w.akre@intel.com>
This release contains a bug fix to wrong mixer.state initialization when
server_state_dir is outside the mix directory tree.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
MixState needs to know the server state dir in order to find the value
for previous version since its path will not always be relative to the
execution folder.
With this patch, MixState is always loaded after MixConfig. Since the
conversion of older formats of MixConfig account for format transfer,
changing the order does not prevent the format value to be properly
moved to the state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
New features:
- A new flag `--skip-format-check` was added to allow downstream users
to cross an upstream format bump manually. The manual process was unable
to proceed in this scenario since the format check would prevent the +20
build from completing
- Custom `os-release` files can now be provided as a replacement for the
default one. The path for the custom file can be configured in the new
`OS_RELEASE_PATH` option in `builder.conf`
- A new `latest_version` file is now published under `www/version/`
directory. This file allows the latest published version to be retrieved
without requiring the latest format to be known.
- A new subcommand `build validate` was introduced. This command
validates the difference between manifest contents of two versions by
comparing it with the difference in content of the rpms from those
versions.
Bug Fixes:
- Compression goroutines now use the user provided bundle-workers as the
default number number of goroutines. This change prevents mixer from
hitting the file descriptor limit - "Too many open files" error. This
change may have some performance impact on larger builds with the
default parameters, but the problem can be circumvented by manually
tuning the number of bundle-workers.
- Directories are now excluded from content size calculation. This
change allows builds to be reproducible since the directory size may
change even if the content is the same.
- Fix `--clean` flag for build subcommand. The clean flag did not roll
back the previous version of the mix, which caused deleted entries to be
missing when manifests were generated.
- Fix race conditions found by go's built in race detector.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When doing manual format bumps, the upstream version for +20 needs to be
updated to the upstream version in the new format, but since the format
differs, the build will fail.
This PR adds a new flag to `mixer versions update` and to `mixer build`
subcommands that allows this check to be skipped.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Directories can be installed with inconsistent sizes. To make mixes
reproducible, the size of directories is omitted from the contentsize
calculation.
Signed-off-by: John Akre <john.w.akre@intel.com>
If publishing an update, also update the www/version/latest_version.
This is useful for various consumers of swupd data: swupd-client can
easily discover the latest version (and not only the latest version in
a given format); swupd-inspector and swupd-extract will be able to
automatically figure the latest version of a mix given the update URL.
If --no-publish is used, the file will not be updated like the other
related files. Clear Linux itself uses that so it can decide later on
whether to publish or not an update to the end users.
Related to #467. With latest_version is possible to peek at
www/<latest_version>/format and figure its format.
Spawning a goroutine for each bundle when compressing files can cause the
system to reach the open file limit. This change allows the number of
goroutines used when compressing files to be configured by the adjusting the
bundle-workers argument.
Signed-off-by: John Akre <john.w.akre@intel.com>
/usr/lib/os-release is a special case file that must be updated by
Mixer for every mix with the exception of the +10 to +20. This file is
directly modified by Mixer and it is also part of the filesystem
package. As a result, a false positive error message will be generated
when /usr/lib/os-release is only modified by Mixer, but the error
message will not be generated when the filesystem package also modifies
the file. The false positive error message cannot be relied upon to
verify that /usr/lib/os-release was modified, so an additional error
message was added when the file is unchanged. In the case of a +10 to
+20 comparison, this error will be removed.
Signed-off-by: John Akre <john.w.akre@intel.com>
Reads to the Hashes variable must be inside of a reader mutex to prevent
race conditions with writes to the variable.
Signed-off-by: John Akre <john.w.akre@intel.com>
The bsdiff logger was shared globally between several goroutines and in
some cases, a goroutine would override the existing logger with a newly
created one.
This change centralizes the logger creation so that it is only created
once which prevents logger creation race conditions.
Signed-off-by: John Akre <john.w.akre@intel.com>
The scope of the err variable was not limited to its local function, so
it was possible for outside errors from other goroutines to set it. This
change limits the scope of the err variable to the local function which
prevents this type of race condition.
Signed-off-by: John Akre <john.w.akre@intel.com>
The `--clear-version` flag in mixer init accepts a string as parameter
to allow the user to set it to `latest`, which will auto fetch the
upstream version. For all other cases, this flag must be provided an
integer number. This patch clarifies the valid values for this flag both
in the error message and in the help description.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The state file uses relative file paths, but Mixin does not create a
mix inside the current working directory. Changing the current working
directory to the mix directory solves relative file path issues for the
state file.
Signed-off-by: John Akre <john.w.akre@intel.com>
The PREVIOUS_MIX_VERSION value from mixer.state has replaced LAST_VER
for setting the previous manifest header field. The LAST_VER value was
unable to properly set the previous header field when rebuilding a mix
with the clean flag. This change resolves the issue with the clean flag.
Signed-off-by: John Akre <john.w.akre@intel.com>
The mixer.state file now tracks the previous mix version with the
PREVIOUS_MIX_VERSION field. When this value is not set, it will default
to the LAST_VER or 0 when LAST_VER is invalid.
Format bumps create an important exception for the value of
PREVIOUS_MIX_VERSION. During a format bump, PREVIOUS_MIX_VERSION will be
overridden by the LAST_VER which will prevent PREVIOUS_MIX_VERSION from
altering format bump behavior.
Additionally, the mixer.state version was incremented and tests for
state file version conversions were added.
Signed-off-by: John Akre <john.w.akre@intel.com>
If custom bundle name matches upstream bundle name, use only custom bundle manifest in MoM.
Otherwise swupd picks up first manifest from MoM (upstream) and fails to validate manifest of custom bundle.
When any manifest file checked by MCA has a version change without a
hash change, the bundle is marked as a minversion and will have a
'minversion bump detected' field in the results summary.
Signed-off-by: John Akre <john.w.akre@intel.com>
Format bumps require additional special case file handling for the
os-core and os-core-update bundles. Error messages are generated when
any special case files are missing and warning messages are generated
when they may be missing. The warning messages are necessary because
there currently is not enough information to determine the +20 version
which has file exceptions. Also, this change adds bundle deletion
support.
Signed-off-by: John Akre <john.w.akre@intel.com>
RPMs in the local RPM repo override upstream RPMs which can impact the
results of MCA. MCA will print a warning message to make users aware that
their local RPMs may override the upstream RPMs used by MCA.
Signed-off-by: John Akre <john.w.akre@intel.com>
MCA compares two versions to validate that the manifest file changes
align with corresponding package changes. All manifest file changes (except
for a few special cases) will be compared against resolved package file
changes. Any mismatches will be reported as errors. When there are no
errors, package statistics will be displayed.
The following command runs the MCA test:
mixer build validate --from <version> --to <version>
Signed-off-by: John Akre <john.w.akre@intel.com>
To make downloadRpms more generic, it no longer cleans the DNF cache
after failures and returns the DNF output.
Signed-off-by: John Akre <john.w.akre@intel.com>
New param added in builder.conf Mixer section to support
customize os-release file.
Example to specify modified os-release file path in Mixer section:
[Mixer]
OS_RELEASE_PATH = "/tmp/modified-os-release"
Note: Either use empty string or do not specify the OS_RELEASE_PATH indicated
Mixer to use default os-release file.
Signed-off-by: sweeaun <swee.aun.khor@intel.com>
Logger default date/time are prefixes and don't play well when
printing multiple lines. What usually matters is the amount of time
elapsed since the start, so print that instead (when each new entry
that is not the first is starting).
The bundle validation count only needs to account for the bundles that
have been directly added to the mix. With the fragmentation of the
default bundles into smaller bundles that are used as includes, the test
must ignore the included bundles and just count the ones that were
directly added.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release reintroduce manifest delta support from the old
swupd-server.
Changes:
- Enable create Manifest deltas in mixer
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
swupd-server used to support building bsdiff based deltas of manifest
files. This change adds that ability back for bundle manifests with
the 'mixer build delta-manifests' command.
This release contains bug fixes for `--offline` option and for the
download timeout error when building large mixes.
BugFixes:
- Fail init offline with no clear version.
- Enforce offline mode in `mixer versions`.
- Skip format bump check in offline mode.
Changes:
- Deprecate new-swupd and new-config flags.
- Use cdn to download content on mixer.
- Increase Clear repo default timeout.
- Retry RPM downloads on failure.
- Add clean flag to mixer build all command.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Rodrigo Chiossi