Don't download the linter unless a special CI env setting is set. Also
use a different cache to avoid permissions issues with the rootfs in
CI.
Signed-off-by: William Douglas <william.douglas@intel.com>
This release includes minor bug fixes and enhancements.
- Fix generated warning for delta packs.
- Show upstream-format command in help.
- Remove delta file in case bspatch test fails.
- Update warning message for missing state file.
- Update to GitHub Actions v2.1.0
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes minor bug fixes along with implementation of
logging.
Enhancements:
- Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`.
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Other Changes:
- Remove the --no-publish flag.
Mixer will now always update the LAST_VER file after `build update`.
- Update github actions to v2 to handle CI re-runs.
- Update Makefile to build man pages as a prerequisite for the
`build` target.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Update man page to include `format` flag.
Also, update Makefile to build man pages as a prerequisite for the
`build` target.
Fixes#557
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes performance improvements, minor bug fixes and
removal of the Mixin feature.
Changes:
- Remove the Mixin feature as Mixer now supports creation of 3rd party content.
- Calculate the hash for each file just once.
- Parallelize bundle processing.
- Parallelize bundle file resolving.
- Update mix variables when crossing upstream formats.
- Fix MCA bundle deletion during format bump.
- Add more debug messages for DNF errors.
- Fix permission issue in bat test.
- Replace Travis CI with GitHub Actions.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes minor enhancements and bug fixes.
Enhancements:
- Initialize local yum repo even when its empty as newer versions of
DNF requires that yum repos be initialized.
- Install special case files in the end so that they will not create
directories with potentially conflicting permissions.
The order of installation is now packages, contents and then special
case files.
Bug Fixes:
- Set file permissions in the chroot after creating it and setting the
file owner. This is necessary because umask can prevent newly created
files from acquiring the source file's permissions.
- Don't follow symlinks when comparing files between content and full
chroot.
- Skip format bump check in MCA.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes support for building 3rd party content along with
other enhancements and bug fixes.
New features (beta) for 3rd party support:
- Create a minimal mix using just an os-core bundle that can be empty.
- Add BUILD_ID field in os-release file that specifies the upstream
version from which the mix is created.
- Build bundles from contents by providing the content path using a new
keyword "content()" in the bundle definition.
- Introduce a new export flag "x" in bundle manifests to indicate exportable
files in "/bin/", "/usr/bin/" and "/usr/local/bin/".
The user can override this flag by specifying the files using a new
keyword "un-export()" in the bundle definition.
Other changes:
- Fix error to handle bundles that contain only optional bundles.
- Fix build errors due to stale dnf cache by cleaning the dnf cache before using it.
- Clean mix workspace before building bundles.
- Add --local flag for bundle creation that skips the upstream check and
creates empty local bundles.
- Improve performance by writing mixbundles file just once for the resolved bundle set.
- Make bundle messages consistent.
- Print delta pack results together to prevent interleaved results.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes minor enhancements and bug fixes.
Changes:
- Add root permission check to inform the user when certain commands
require root permission.
- Refactor error handling in go channels to prevent potential hanging of
the application in certain scenarios.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes transitioning to `go mod` to manage dependencies along with
minor enhancements and bug fixes.
Changes:
- vendor: Use `go mod` to manage dependencies.
- Makefile: Remove unused local gopath and set default gopath.
- Makefile: Install go linters if it's not present.
- MCA: Refactor diff results analysis section to make it easier to handle
special case manifest files.
- MCA: Add support for bundle tracking file.
- swupd: Retain previously deleted file entries when the previous manifest's
filelist is longer than the new manifest's list.
- repo: Add the option to set a repo’s priority using the `set-priority` repo command.
- Detect deprecated bundles when crossing an upstream format bump in a downstream mix.
- When packages cannot be resolved during `build bundles`, indicate the unresolved
packages in the error message.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
Currently, with the use of go modules, packaging using autospec expects
all the vendored dependencies to be available as go modules in koji.
Since that is not the case, the vendor directory is still maintained in
this repo.
By default, go commands like `go install` ignore the vendor directory
when in module mode. The -mod=vendor flag instructs the go commands to
use the main module's top-level vendor directory to satisfy dependencies.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes using `go mod` to manage dependencies along with
MCA enhancements and bug fixes.
Changes:
- MCA: Refactor diff results analysis section to make it easier to handle
special case manifest files.
- MCA: Add support for bundle tracking file.
- MCA: Print warnings to stderr.
- swupd: Retain previously deleted file entries when the previous manifest's
file list is longer than the new manifest's list.
- vendor: Use `go mod` to manage dependencies.
- Makefile: Remove unused local gopath and set default gopath.
- Makefile: Install go linters if its not present.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
A local gopath is no longer needed to build if the code is outside the
go src/ folder. This change also allows `make lint` to work if the code
is outside the go tree.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release includes improved MCA output format along with improved
performance for repos with multiple sub directories.
Changes:
- vendor: Add tablewriter.
- MCA: Refactor statistics table.
- repo: Improve performance for multiple repo subdir.
- Fix lint errors.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes improved MCA performance along with other bug fixes.
With all the bundles and package repositories on the local system,
the execution time for MCA (mixer build validate) has improved
from ~106 min to ~15 min.
Changes:
- MCA: Instead of querying packages downloaded by `dnf install`,
the packages are resolved and queried with `rpm`.
When the packages are in local repos, the query operation
is much faster.
- MCA: Added support to resolve packages that do not follow NVRA name format.
- MCA: Suppress verbose Mixer output.
- Return more package metadata during package resolution.
- Remove creation of `os-packages` file as it is empty and not being
used.
- repo: Added support to handle packages within sub directories of a
repo baseurl. Although, for better performance, it is still recommended
to store the packages directly within the repo baseurl for local repos.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes bug fixes to maintain feature parity with v5.9.1
along with the MCA change to take into account sha256 hashes.
Changes:
- Skip hash comparisons between md5 and sha256 during `build validate`
to avoid incorrect hash mismatches. rpm 4.12 used md5 file hashes
by default and rpm 4.14 switched the default to sha256.
- Stop building bundles and exit if package dependencies are not resolved.
This re-enables feature parity with v5.9.1.
- Allow users to set multiple repos correctly by removing the
constraint to use only "clear" and "local" repos. This re-enables
feature parity with v5.9.1.
- Fix the incorrect error message when trying to remove a non-existent
repo.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes a minor bug fix in the zero pack creation.
Changes:
- Remove shared variable in the zero pack go routine.
The err variable was being shared and updated in the zero pack go routine.
This could cause race conditions and undesirable results including segmentation fault.
This fix creates a local variable within the zero pack go routine.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes bug fixes and improved logging along with
replacing ister with clr-installer to build images.
Changes:
- Replace ister with clr-installer to build images.
Also generates an iso image.
- Preserve config filename during conversion.
- Use "dnf repoquery" to determine rpm filename instead of assuming the
naming convention.
- Identify missing rpms early on instead of waiting till extraction.
- Fix array out of bounds exception while downloading rpms.
- Fix incorrect logging during rpm extraction.
- Improve logging.
Signed-off-by: Reagan Lopez <reagan.lopez@intel.com>
This release includes major performance improvements to mixer build
process and removes some of its legacy systems.
On the performance side, RPM extraction is now performed by rpm2archive
and tar instead of dnf. This cuts a significant amount of time from the
overall build. Selective compression for fullfiles have also been
introduced and the default compression method is set to xz only. This
change reduces fullfiles generation time to 30% of the previous time at
the cost of 0.3% increase in build size for a mix with all upstream
packages. Lastly, zero packs creation have been parallelized to further
reduce build time.
Iterative manifests are now gone. Our tests showed that they didn't
present any performance advantage in comparison with delta manifests
which are already being used in production. The 'bundle index' bundle
has also been removed. This feature was never used for its intended
purpose and caused builds to fail frequently due to hash mismatches
during its generation. Existing mixes with this bundle will just have it
propagated as a empty bundle until the next format bump. At that point,
the bundle will be automatically removed.
Built in support for Docker has also been removed. The `--native` flag
has been deprecated and all commands now run in a native setup. Docker
support was introduced to allow auto-format-bump feature to be
implemented, but current mixer implementation does not require it
anymore to work. External docker images, like mixer-ci, should be used
as a replacement in use cases where the built-in docker support was used
to create mix in incompatible host OSes.
Mixer now provides the possibility of customizing the URL where the
bundles are downloaded from. Previously that was hardcoded to upstream
bundles, which prevented mixes from being created based of downstream
releases. This option can be configured in 'builder.conf'.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Gometalinter has been deprecated in favor of golangci-lint.
This patch updates `make lint` to use the proper linter.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release includes a fix to delta pack generation and adds signing
support to the `latest` and `latest_version` files.
Changes:
- Return error instead of panic while generating delta
- Signing the latest file in version/formatXX dir
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release includes an optimization in the way file records are
generated in manifests. Instead of adding all files from a bundle and
later removing the files that already exist in that bundle dependencies,
mixer now preemptively checks if a file is part of a dependency before
generating its record. This greatly reduces the number of generated
records and speed build time.
Included patches:
- swupd: Skip included files when creating file records
- MCA: Parse rpm output with \a character
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release reverts the PR which removed the index bundle. This PR is
triggering a build error that is noticed on multi-core machines.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release introduces the also-add feature to bundle files and removes
the auto-generated index bundle.
Changes:
- Delete index manifest
- Support also-add bundles in manifests
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release includes multiple fixes to mixer's MCA along with updated
versions of mixer's dependencies.
Dependencies update:
- github.com/BurntSushi/toml: v0.3.0 -> v0.3.1
- github.com/spf13/cobra: 0.0.1 -> 0.0.5
- github.com/go-ini/ini: 1.32.0 -> 1.42.0
- github.com/pkg/errors: 0.8.0 -> 0.8.1
MCA Fixes and Improvements:
- Skip files with blacklisted characters
- Use resolveFileName function to remove duplicate code
- Use recursive includes list for file/pkg subtraction
- Serialize package downloads
- Minor statistics reporting improvements
- Sort bundles when printing statistics
- Add documentation
- Support repo/baseurl overrides for from and to versions
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release contains a bug fix to wrong mixer.state initialization when
server_state_dir is outside the mix directory tree.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
New features:
- A new flag `--skip-format-check` was added to allow downstream users
to cross an upstream format bump manually. The manual process was unable
to proceed in this scenario since the format check would prevent the +20
build from completing
- Custom `os-release` files can now be provided as a replacement for the
default one. The path for the custom file can be configured in the new
`OS_RELEASE_PATH` option in `builder.conf`
- A new `latest_version` file is now published under `www/version/`
directory. This file allows the latest published version to be retrieved
without requiring the latest format to be known.
- A new subcommand `build validate` was introduced. This command
validates the difference between manifest contents of two versions by
comparing it with the difference in content of the rpms from those
versions.
Bug Fixes:
- Compression goroutines now use the user provided bundle-workers as the
default number number of goroutines. This change prevents mixer from
hitting the file descriptor limit - "Too many open files" error. This
change may have some performance impact on larger builds with the
default parameters, but the problem can be circumvented by manually
tuning the number of bundle-workers.
- Directories are now excluded from content size calculation. This
change allows builds to be reproducible since the directory size may
change even if the content is the same.
- Fix `--clean` flag for build subcommand. The clean flag did not roll
back the previous version of the mix, which caused deleted entries to be
missing when manifests were generated.
- Fix race conditions found by go's built in race detector.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release reintroduce manifest delta support from the old
swupd-server.
Changes:
- Enable create Manifest deltas in mixer
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release contains bug fixes for `--offline` option and for the
download timeout error when building large mixes.
BugFixes:
- Fail init offline with no clear version.
- Enforce offline mode in `mixer versions`.
- Skip format bump check in offline mode.
Changes:
- Deprecate new-swupd and new-config flags.
- Use cdn to download content on mixer.
- Increase Clear repo default timeout.
- Retry RPM downloads on failure.
- Add clean flag to mixer build all command.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release includes the following:
- Fix iterative manifests leaking on minversion
- Enable iterative manifests with 0 file count
- Add support for experimental bundles
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This release includes the following fixes and additions:
- Fix MoM verification with OpenSSL 1.1 requring a purpose to be passed
- Make --config global
- Break root preRun() into smaller parts, with better & simplified logic
- builder: only check online to update upstreamver
- Removed editor option for bundle edit; simply creates new bundles or
copies existing ones now
- Added bundle_control unit tests
- Convert to using go mod vendor instead of our full vendor directory
- Enable iterative manifest creation for more optimal update content
creation
- Misc formatting and error fixes
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
William Douglas