Mixer does not create manifest entries for files with blacklisted
characters, so MCA should not include files with blacklisted
characters in the comparison.
Signed-off-by: John Akre <john.w.akre@intel.com>
When subtracting files and packages each manifest in the recursive
includes list must be checked to consistently subtract files/packages
correctly.
Signed-off-by: John Akre <john.w.akre@intel.com>
There was a race condition where the rpm command would try to query
corrupt (partially downloaded) packages. By serializing package
downloads, this race condition is no longer an issue. Also,
parallelized package downloads did not significantly improve
performance, so this change will not be a noticeable performance
regression.
Signed-off-by: John Akre <john.w.akre@intel.com>
The following minor statistics reporting changes were added to improve
statistics readability:
* Add colons to the added/deleted package sections
* Add space between '|' character and bundle name
Signed-off-by: John Akre <john.w.akre@intel.com>
In some cases a single DNF config file is not sufficient to correctly
download packages for the mix versions specified by the from and to
flags. The --from-repo-url and --to-repo-url flags were added to
temporarily override the baseurl value for the specified repo in the DNF
config when downloading packages for the from and to versions respectively.
Multiple instances of these flags can be used to override additional
repo/baseurl pairs. An example flag format is shown below:
--from-repo-url <repo>=<URL>
--to-repo-url <repo>=<URL>
Signed-off-by: John Akre <john.w.akre@intel.com>
MixState needs to know the server state dir in order to find the value
for previous version since its path will not always be relative to the
execution folder.
With this patch, MixState is always loaded after MixConfig. Since the
conversion of older formats of MixConfig account for format transfer,
changing the order does not prevent the format value to be properly
moved to the state file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When doing manual format bumps, the upstream version for +20 needs to be
updated to the upstream version in the new format, but since the format
differs, the build will fail.
This PR adds a new flag to `mixer versions update` and to `mixer build`
subcommands that allows this check to be skipped.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If publishing an update, also update the www/version/latest_version.
This is useful for various consumers of swupd data: swupd-client can
easily discover the latest version (and not only the latest version in
a given format); swupd-inspector and swupd-extract will be able to
automatically figure the latest version of a mix given the update URL.
If --no-publish is used, the file will not be updated like the other
related files. Clear Linux itself uses that so it can decide later on
whether to publish or not an update to the end users.
Related to #467. With latest_version is possible to peek at
www/<latest_version>/format and figure its format.
Spawning a goroutine for each bundle when compressing files can cause the
system to reach the open file limit. This change allows the number of
goroutines used when compressing files to be configured by the adjusting the
bundle-workers argument.
Signed-off-by: John Akre <john.w.akre@intel.com>
/usr/lib/os-release is a special case file that must be updated by
Mixer for every mix with the exception of the +10 to +20. This file is
directly modified by Mixer and it is also part of the filesystem
package. As a result, a false positive error message will be generated
when /usr/lib/os-release is only modified by Mixer, but the error
message will not be generated when the filesystem package also modifies
the file. The false positive error message cannot be relied upon to
verify that /usr/lib/os-release was modified, so an additional error
message was added when the file is unchanged. In the case of a +10 to
+20 comparison, this error will be removed.
Signed-off-by: John Akre <john.w.akre@intel.com>
The bsdiff logger was shared globally between several goroutines and in
some cases, a goroutine would override the existing logger with a newly
created one.
This change centralizes the logger creation so that it is only created
once which prevents logger creation race conditions.
Signed-off-by: John Akre <john.w.akre@intel.com>
The scope of the err variable was not limited to its local function, so
it was possible for outside errors from other goroutines to set it. This
change limits the scope of the err variable to the local function which
prevents this type of race condition.
Signed-off-by: John Akre <john.w.akre@intel.com>
The PREVIOUS_MIX_VERSION value from mixer.state has replaced LAST_VER
for setting the previous manifest header field. The LAST_VER value was
unable to properly set the previous header field when rebuilding a mix
with the clean flag. This change resolves the issue with the clean flag.
Signed-off-by: John Akre <john.w.akre@intel.com>
The mixer.state file now tracks the previous mix version with the
PREVIOUS_MIX_VERSION field. When this value is not set, it will default
to the LAST_VER or 0 when LAST_VER is invalid.
Format bumps create an important exception for the value of
PREVIOUS_MIX_VERSION. During a format bump, PREVIOUS_MIX_VERSION will be
overridden by the LAST_VER which will prevent PREVIOUS_MIX_VERSION from
altering format bump behavior.
Additionally, the mixer.state version was incremented and tests for
state file version conversions were added.
Signed-off-by: John Akre <john.w.akre@intel.com>
When any manifest file checked by MCA has a version change without a
hash change, the bundle is marked as a minversion and will have a
'minversion bump detected' field in the results summary.
Signed-off-by: John Akre <john.w.akre@intel.com>
Format bumps require additional special case file handling for the
os-core and os-core-update bundles. Error messages are generated when
any special case files are missing and warning messages are generated
when they may be missing. The warning messages are necessary because
there currently is not enough information to determine the +20 version
which has file exceptions. Also, this change adds bundle deletion
support.
Signed-off-by: John Akre <john.w.akre@intel.com>
RPMs in the local RPM repo override upstream RPMs which can impact the
results of MCA. MCA will print a warning message to make users aware that
their local RPMs may override the upstream RPMs used by MCA.
Signed-off-by: John Akre <john.w.akre@intel.com>
MCA compares two versions to validate that the manifest file changes
align with corresponding package changes. All manifest file changes (except
for a few special cases) will be compared against resolved package file
changes. Any mismatches will be reported as errors. When there are no
errors, package statistics will be displayed.
The following command runs the MCA test:
mixer build validate --from <version> --to <version>
Signed-off-by: John Akre <john.w.akre@intel.com>
To make downloadRpms more generic, it no longer cleans the DNF cache
after failures and returns the DNF output.
Signed-off-by: John Akre <john.w.akre@intel.com>
New param added in builder.conf Mixer section to support
customize os-release file.
Example to specify modified os-release file path in Mixer section:
[Mixer]
OS_RELEASE_PATH = "/tmp/modified-os-release"
Note: Either use empty string or do not specify the OS_RELEASE_PATH indicated
Mixer to use default os-release file.
Signed-off-by: sweeaun <swee.aun.khor@intel.com>
Logger default date/time are prefixes and don't play well when
printing multiple lines. What usually matters is the amount of time
elapsed since the start, so print that instead (when each new entry
that is not the first is starting).
swupd-server used to support building bsdiff based deltas of manifest
files. This change adds that ability back for bundle manifests with
the 'mixer build delta-manifests' command.
Download failures can occur when downloading large sets of RPMs from a
server with a poor connection. This change adds retry functionality to
attempt RPM downloads multiple times. Also, the retries flag was added to
the following Mixer build commands.
retries:
Fixes: #531
Signed-off-by: John Akre <john.w.akre@intel.com>
Since the default Clear repo mirror changed to the cdn, dnf timeouts
occur more frequently. Increasing the timeout to 45 seconds from the
default timeout of 30 seconds.
Signed-off-by: John Akre <john.w.akre@intel.com>
There is no information available in offline mode to determine if a
format bump is required. This change also allows to manually set the
value for upstream version without directly editing the file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
For `mixer versions` subcommand and `mixer versions update` when
--upstream-version flag was provided, builder would try to fetch
information about upstream about current version and format.
This patch provides an alternative output for `mixer versions` and
also skips upstream checks for `versions update` when in offline mode.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
From format 27 onwards, Mixer will recognize bundles with "[Status] : Experimental"
as experimental bundles and mark their status flag as "e" in the Manifest.MoM.
i.e. These bundles will include an "e" in the 2nd position of the manifest flags,
as shown here:
Me.. 4b91c3122e7e32f1e3edb597c6f89ff32cfffd977afb5e5 10 some-bundle
Fixes#497
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
When creating manifests for a new build, also create to-Manifests
for each bundle that has changes in this version.
The goal on publishing to-Manifests is to reduce the download size
on updates, according to what was discussed on
clearlinux/swupd-client#498
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
`bundle add` will now add valid bundles and skip only the invalid ones,
thereby fixing issue #489.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Some testcases showed these checks were incomplete and not handling
corner cases. Improve the logic of these checks to comply with the new
tests.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of requiring upstream bundles be read from a child directory
under where mixer is currently run read them from underneath the mixer
workspace (builder.Config.Mixer.VersionPath) the same way local-bundles
is handled.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Start to add some unit tests for the bundle_control file. There is much
work to still be done as far as testing goes for these methods.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
With the removal of the editor feature in `bundle edit`,
the term `edit` is not relevant anymore. Hence, changing it to `create`.
In order to prevent breakage of existing use-cases,
`edit` is aliased to `create`and `--suppress-editor` is made hidden and deprecated.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
`bundle edit` will no longer open the editor. It will just create new bundles or copy existing bundles.
This command will locate the bundle by first looking in local-bundles, and then in upstream-bundles.
If the bundle is only found upstream, the bundle file will be copied to the local-bundles directory.
If the bundle is not found anywhere, a blank template will be created with the correct name.
This commit also fixes issue #448 and increases test coverage for `bundle validate`.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Updating mixver should not require a network connection, nor reach out
to the server to find out any more information than is needed.
Fixes#282
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fixes#482
Since clients can't update over format bump boundaries anyways do not
attempt to create delta-packs covering that jump.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This commit fulfills enhancement #58. When building an image without providing a template,
if the default image template is not present in the mix directory, mixer will do the following:
- Copy image template from ister template "/usr/share/defaults/ister/release-image-config.json".
- Update image template with the bundle list from "mixbundles" file.
In case the "mixbundles" file is empty or not present, use ister template as is.
- Inform the user about the above steps.
- Continue the existing build process.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
John Akre