This release includes the following updates:
- Revert bundles:Update os-release file with mix info
- Add minversion header to MoM
When the minVersion argument is provided to createManifests
populate a header in the MoM with this version. If no flag is
provided and the previous MoM had a minversion set populate the
new MoM with this version.
Write the minversion to the manifest header when the minversion
!= 0. Since we only set minversion for the MoM the minversion
will only be written to the MoM and not to bundle manifests.
- Fix warnings message in checkNotContains() for helpers_test
- Update comment parsing for package files to ignore # and trim
white space
Critical Updates:
----------------
* New format bump process script (afb.sh) and test added to describe
the new format bump flow that should be followed. It outlines a much
simpler way to do bumps, and does not require moving back and forth
between formats or tooling.
* Do not update deletes over minversions
Minversions should not update deleted files to the minversion as
this may cause swupd to attempt to delete content twice. This
exacerbates issues where a directory (/usr/local/bin) was
mistakenly deleted in a version because it was mistakenly added
earlier. Updating the delete to the minversion then causes the
directory to be deleted off the client system again.
* Enable backwards compatible build update
This commit enables mixer to write different manifest formats
based on the configured format value. When introducing a
breaking change to a manifest it will be registered with
the format it requires. Existing template is registered to
everything through format25 to remain compatible with Clear
Linux OS format. Everything above the current format gets the
new template, in this case with the minversion added to the MoM.
This means that format bumps do not need two different versions
of mixer installed since it will always know how to build the
correct format based on the configured value.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This script is intended as a reference and architecture document for a
new format bump flow. This format bump flow is much simpler than the
original/current flow which requires jumping back and forth between the
+20 and +10 builds to create the updates.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Fixes#455
Minversions should not update deleted files to the minversion as this
may cause swupd to attempt to delete content twice. This exacerbates
issues where a directory (/usr/local/bin) was mistakenly deleted in a
version because it was mistakenly added earlier. Updating the delete to
the minversion then causes the directory to be deleted off the client
system again.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Make package file parsing ignore '#' and anything after that until the
end of the line. Also trim spaces. This makes the comment syntax the
same as bundle files.
At this point, one could consider trying to merge both parsing
functions, but given how simple the rules for this one are, kept them
separate. We can revisit this later.
This commit enables mixer to write different manifest formats based on
the configured format value. When introducing a breaking change to a
manifest it will be registered with the format it requires. Existing
template is registered to everything through format25 to remain
compatible with Clear Linux OS format. Everything above the current
format gets the new template, in this case with the minversion added to
the MoM.
This means that format bumps do not need two different versions of mixer
installed since it will always know how to build the correct format
based on the configured value.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The minversion header should only show up in the MoM and should carry
forward to the next version's MoM if the --min-version flag is not
supplied on the commandline. The minversion header should not be
displayed if the minversion is 0.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
An upcoming improvement to swupd-client requires the minversion header
to exist in the MoM in order to determine the strategy with which to
update files in the update.
When the minVersion argument is provided to createManifests populate a
header in the MoM with this version. If no flag is provided and the
previous MoM had a minversion set populate the new MoM with this
version.
Write the minversion to the manifest header when the minversion != 0.
Since we only set minversion for the MoM the minversion will only be
written to the MoM and not to bundle manifests.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes a couple bugs that block the format bump process:
1) Don't add deleted bundles to MoM
When a bundle gets deleted, its entry still exists in the previous
MoM but it shouldn't exist in the new MoM. Since the deleted bundle
will not exist in the bundle list anymore, it must be also verified
to determined if an missing entry should be propagated to the new MoM.
2) Skip format bump check if running offline.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Format bump check requires fetching format for current upstream which
cannot be done if there is no network available. If `--offline` flag is
provided, this check must be skipped.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When a bundle gets deleted, its entry still exists in the previous MoM
but it shouldn't exist in the new MoM. Since the deleted bundle will not
exist in the bundle list anymore, it must be also verified to determined
if an missing entry should be propagated to the new MoM.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Bug fixes:
- Do not user container for `build image`.
The build image command cannot run in a container because there are no
loopback devices to mount in a container environment. This change
forces mixer to run in native mode when build image is called.
- Revert download timeout to default values.
The timeout for downloads was causing mixer to fail in slow networks
because it was too short. This fix reverts the value back to default.
- Properly handle upstream URL with subpath.
Mixer were unable to properly handle downloads when the upstream URL
included subpaths since the subpath was discarded. This fix changes the
internal algorithm to preserve the subpath.
- Enforce --Offline flag.
During mixer initialization, some commands relied on a separate check to
determine if an online check was required. This check ignored the value
of the --offline check, causing some scenarios where mixer would try to
connect even when the flag was set. This fix gives priority to the flag
to make sure it is always respected when set.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
There are a couple mixer commands that don't require network and thus
skip any network check. This behavior is also intended when the
--offline flag is used, but the flag is never checked in that scenario.
This patch unifies builder.Offline (which is set by --offline) and
networkCheck variable to make sure the network check is also skipped if
--offline flag is used.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When building the upstream URL, if the b.UpstreamURL was set do a url
with a subpath included, the subpath was being discarded by
URL.ResolveReference(). In order to preserve the subpath in the base,
the provided subpath must be manually added to the base URL object.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This reverts commit 9cf854d8d6.
The reduction to the timeout has caused failures for users on slow
networks, so this must be reverted to the default values.
The reduction to the timeout was added to improve user experience when
connection failed due to a proxy error. It would take mixer too long to
fail and the error message didn't indicate what the problem could be.
The error is now properly indicated, so even if it takes a while for the
error to occur the first time, the user should be able to identify the
problem on the spot.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The build image command cannot run in a container because there are no
loopback devices to mount in a container environment. Force running with
native if build image is called.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
For a clear linux mix, the os-release file will be updated as follows:
- NAME="Clear Linux Mix"
Eventually this value should be read from user input when #424 is implemented.
- ID=<based on above NAME field>
E.g. ID=clear-linux-mix
- VERSION="<mix-ver> (<upstream-ver>)"
E.g. VERSION="10 (24750)"
- PRETTY_NAME="<NAME> <mix-ver> (<upstream-ver)"
E.g. PRETTY_NAME="Clear Linux Mix 10 (24750)"
- VERSION_ID=<mix-ver>
E.g. VERSION_ID=10
- BUILD_ID=<upstream-ver>
E.g. BUILD_ID=24750
- MIX_UPSTREAM_URL="<upstream-url>"
This is a new field in os-release.
E.g. MIX_UPSTREAM_URL="https://download.clearlinux.org"
These automated tests will check if conversion between multiple older
versions to the current version is working properly.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
With the automatic conversion on load time, mixer ill always use the new
config format. Even if the mix is initialized with the legacy config, it
will be converted right away. As such, there is no need to allow the
creation of legacy configs anymore nor to check if it is being used.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch introduces automatic conversion of config files to the latest
format. This will seamlessly transition all users to the latest format
without the need for manually converting the config with the `mixer
config convert` command.
This patch also introduces the config_conver.go file, which centralizes
all conversion operations and also the parsing for the legacy config in
order to keep config.go clean and focused on current config features.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
All the values inside buildBundlesConfig are already parsed and
available in the code through builder.Config, so there is no need to
parse them again from the config file.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release fixes many bugs in the tooling:
- Fix double parsing of the config file
- Do not add default bundles when 'init' called with offline
- Reduce download timeout to 10 seconds instead of 30 on file download
- Improve download upstream error message
- Enforce UTF-8 locale when running DNF commands in mixer
- Ignore possible %ghost files in RPMs
- Fix global --offline usage to completely block all outbound network
checks on all commands, even when running in container mode
- Fix error in concurrent manifest creation code to properly exit on
first error hit
- Use --bundle-workers for manifest generation like other commands
- Fix to use passed in minversion for +10 format bump build
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
When generating the manifest for bundles, use the user defined
--bundle-workers to determine the number of goroutines to use instead of
spawning one goroutine per bundle.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When there is an error in one of the worker goroutines, the error is
read but before it is returned, the variable is reused inside the
following for loop to check the error of addFilesFromChroot(). Since we
always have a "full" bundle, it is always overwritten.
Another issue is that the err variable was shared among the goroutines
and was set directly instead of through the error channel. This would
allow one routine to overwrite the error from another.
Lastly, there is a problem with the loop that feeds bundleChan with
data. The second select statement calls 'break' when an error is found,
but 'break' will exit the 'select' block, not the 'for' loop. In order to
stop the loop, we move the break to an if statement that checks for a
condition set on error. This change ensures that the worker goroutines
stop once an error is found.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When --offline is passed, any command executed should not reach over the
network. This introduces a tricky scenario when it is passed to a build
command with --native=false (default), because the command in the
container will be run with --offline, but the native mixer binary will
reach over the network to pull/update docker containers before executing
in them. This patch also makes that process offline only, attempting to
use a cached docker image so that it can run offline fully, but exits if
none exist.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Due to the way that we use repoquery, we are returned files that
may be %ghost files in RPMs. These files may be missing and this
is legal and compliant.
While Clear Linux OS does not use %ghost files in any of the RPMs
created, there is nothing wrong with permitting other vendor RPM
files that may list %ghost files, and mixer should just ignore
these files.
This change does just that: If repoquery returns files that are
not existing in the fullimage, prints a warning and continues with
out dropping the manifest for that bundle.
In some situations dnf may try to combine files from different
encodings. If the command is not running is a UTF-8 locale, this
combination will be missing a new line, causing and error.
An example of this occurrence is the command `dnf repoquery --quiet -l
systemd ca-certs-static ...` which combines a file list from systemd and
ca-certs-static
This patch enforces UTF-8 locale on every dnf execution.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The error when mixer failed to download the upstream version was too
confusing since there were many layers of wrapped errors.
This patch removes the excessive wrapping and suggests a missing proxy
configuration to the user in case this error occurs.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running 'mixer init' with the --offline flag, there is no upstream
to fetch the default bundles from, so the mix should be initialized
without the default bundles.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running any 'mixer config' sub-command, the config will be parsed
in the command itself, so there is no need to pre-parse them in the pre
execution. Those commands also don't rely on network nor cause any
format conflict, so it is safe to skip all the remaining checks like it
is done for the 'init' command.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
We were assigning a map to a map to assign it to a string slice. Instead
just assign the map keys to the string slice directly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When ranging over the syncmap to add packages to a bundle we were
mistakenly ranging over the entire set of bundles and adding all bundle
packages to the same bundle (nondeterministically, since we were doing
concurrent writes to a syncmap). In reality we just need to range over
the repoPkgMap itself and not load it from the syncmap at all. Do this
operation before even storing it.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The default runtime in upstream Clear will be set to the kata runtime,
which does not support host networking. This is needed for mixer, so we
must force --runtime=runc during all docker runs.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes a data race which cause concurrent map read/writes
when resolving packages and bundles. Versioning is also moved into the
Makefile so it does not have to be updated in the source code anymore,
and added at compile time.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Two issues, first is resolved by renaming the error variable so a
goroutine isn't sharing err from outer scope with all other goroutines.
The second issue is resolved by using a sync.Map to safely read and
write to a map via the Range function.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This change moves the setting of mixer version string from source code
to a variable defined during compilation. This approach is preferable
because:
1- builder.go does not need to change just to change version though
functionally is the same.
2- simplifies logic in Makefile, no need to retrieve value from source.
Tudor Marcu