- No Kernel Panic
- Successfully activates verity device in initramfs
- Booting on qemu
- Authentication token manipulation error at login due to read-only
rootfs
- Used kernel cmdline cryptdevice=UUID=*:root root=/dev/mapper/root with correct UUID.
- Unsuccessful.
- Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
- Used kernel cmdline cryptdevice=UUID=*:root root=/dev/mapper/root
- Unsuccessful.
- Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Add a file->bundle index file to go in a manufactured
os-core-update-index bundle for faster swupd search operations on
client. Instead of downloading all manifests in the current version the
client can just bundle-add this bundle and use the file as its search
helper.
Because this file is part of a bundle and regular update it will be a
candidate for renames and delta updating. Only present files and
symlinks, not directories or deleted files, are added to this file. The
file contains a tab-separated list of filename to bundlename mappings,
sorted first by filename and secondarily by bundle size.
This manifest has to be processed after all other manifests have been
processed because the "subtracted" file lists have to be complete. This
means we have to post-munge the MoM and full manifests with the new
index file and new index bundle.
An exception in bundle includes reading is also necessary because the
first time includes are read this bundle will not exist. This means that
bundles that include this index bundle will not perform manifest
subtraction with that bundle. In this specific case it is okay because
the only file in the manifest lives at a path that is present in every
bundle (/usr/share/clear/...). If this changes in the future the impact
will be small because this bundle will only ever provide one file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This reverts most of commit
cb80386f01. At the moment swupd-client
wouldn't handle this, so would end up downloading the fullfiles.
Part of the benefit of this optimization we are going to get from
renames when the hash don't change.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Make pack creation call delta generation, so the delta packs have
deltas inside instead of only fullfiles.
To achieve this, the delta generation was modified in following ways:
- Return a list of all the deltas, not only the failed ones, this is
useful when packing. Otherwise a combination of peeking at DeltaPeer
and the failed list was needed.
- Simplify the goroutine logic: since we have a "home" for all the
error values (in Delta struct), we can just feed all of the deltas
to be processed and wait for the goroutines to finish.
- Instead of calling linkPeerAndChange, call a variant that does not
change the versions, that allows the manifest to be used later on
for the rest of packing.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Import test from original swupd-server. It checks if deleted files are
correctly marked, taking includes into account. I've created some new
checks based on swupd-server output.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Update the test so a smaller format is used when creating the next
version, and verify that it fails.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Import test from original swupd-server. It performs basic checks of a
complete execution for a new version: creating fullfiles and a
zeropack.
Incremented mustValidateZeroPack to check for presence of staged/ and
delta/ and their properties.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Import test from original swupd-server. It checks direct includes
cause a version bump, and indirect ones don't.
This test was already partially present, the new version adds in more
checks to cover everything was being checked by the original tests and
some more (e.g. validating contents of the packs).
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Import test from original swupd-server. It checks whether contentsize
is adding (and not adding) values correctly.
At the moment directories are getting accounted in contentsize (should
they), so use an empty bundle to get a baseline. So I checks
independent of that factor. Also avoided checking the exact figure for
os-core and full, and only do check the size increase between two
versions for those.
It might make sense to have a separate test that verify the contents
of os-core match what is desired, but seemed out of scope of this one.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
In the same spirit of testFileSystem struct. It also embeds the
testFileSystem, so only one needs to be used in a test. Note that
cleanup function will not delete the temporary directory if the test
fails, to allow inspection (it also prints the directory name).
Two tests were modified to illustrate how it should be used, one that
was already using fs and another that wasn't.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Fixes a bug in AddBundles where it didn't clear out the bundles
slice correctly if 'all' was passed AND a bundle list was passed
in. This case never happened with 'mixer bundle add --all', but
did happen with 'mixer init --all'.
swupd-extract is a program that reads a swupd update repository and
extracts the files for bundles of a specific version. It does a
similar job as "swupd verify --install --no-scripts --no-boot-update".
There is code for maintaining the client state, but at the moment it
only care about zero packs. Verification of the Manifest.MoM signature
is done using "openssl" external program.
It has some niceties when we are extracting Clear Linux content
// List the latest clear linux bundles.
$ sudo swupd-extract clear
// Extracts os-core into "output/" directory.
$ sudo swupd-extract clear os-core
// Extracts editors and all its included bundles into mydir.
$ sudo swupd-extract --output mydir clear/20540 editors
but instead of clear, a proper URL with version can be used instead,
i.e. it is easy to use it to extract mixes content. It also works fine
with local paths. The included help enumerates the rest of the features.
There is some duplicate code for compressedTarReader which I intend to
make into an internal package further on (as well as consolidating
other related archive code).
The main motivation was to allow users of distros other than Clear
Linux to use mkosi to generate Clear Linux images, but the program can
also be useful for validation.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
This lets override the default choice of packager for performing build
chroots. Makes easy to test dnf with --packager=dnf. The flag is
ignored when not using --new-chroots.
Also prints the actual command line being used.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Add one more entry to be skipped. This fixes the scenario of using dnf
by changing yum to be a symlink to it.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
- Add tests for the logic of figuring out which bundles need delta
packs between two versions.
- Add tests for creating zero packs, with data from different sources,
as well as incomplete data (so fallback logic kicks in), validating
the contents of the generated pack.
- Add testFileSystem helper struct, to make filesystem-like operations
in a test directory. By using methods of the struct, we reduce a bit
the noise of passing t and dir around.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
This rewrites bundle-chroot-builder.py in Go to be part of Mixer
code. Mixer is the only user of that software, and both Mixer and
bundle-chroot-builder.py the same configuration file, with overlapping
fields.
Main differences from bundle-chroot-builder.py:
- New bundleset type was added, that cares about collecting as much
information as possible from the bundles themselves. This type and
related functions also sets us up for success when upcoming changes
to how bundles are specified happen. There is no assumption all
bundles are in the same directory.
- We are not using m4, instead a bundleset takes care of parsing. If
format of individual bundle files change. The upside is that we can
give nicer error messages, specially for the circular case.
- Read the configuration file directly (with go-ini) to peek at values
that Mixer didn't read before. Done that to avoid conflicting with
existing patch in-flight that parses configuration.
- Some individual steps were reordered for code clarity. E.g.: since
we have bundleset, we can upfront generate all the *-include files.
- Fixed the output for versions file. Due to the way yum list output
works, parsing it is not very friendly. Comments around the code
tells the story.
- Removed the network testing step. It wasn't covering every case in
the Python version, so I'm leaning to let the failure come from
yum/dnf itself. I'm usually in favor of such early tests, but in
this case the price of parsing yet another config file didn't felt
worth.
- Removed the "yum clean all" step from the bootstrap. There isn't any
cache at that point, and the next yum call will bootstrap the
necessary files for yum to operate.
- Removed generation of files-* files (and the pkgmap-* files used to
generate them). I couldn't find any tool or team making use of this
information. Those (or their content) might be relevant in future
changes to use a single chroot, but we should add when we need them.
- Added more detailed commentary to individual steps, collecting
information from the developers of bcb and related software.
- The port still don't parallelize the work into multiple
goroutines. I plan to do this in a similar way than what was done in
CreateFullfiles, but in a separated patch.
Fixes#42.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
This patch removes the last occurrences where an error would be handled
by builder itself and cause the program to Exit. Now the error is always
propagated to the caller.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
When running mixer init-mix, if no builder.conf exists in the provided
builderconf path, create a template in the given path and configure it
using the active directory as base path for the variables.
With this path, init-mix will also create the rpm/ and local/
directories and reference them in the generated builder.conf
When initializing the mix workspace, if a file that would be initialized
already exists, it should be preserved. This patch introduces a generic
implementation of CopyFile that accepts custom flags and provides two
interfaces with pre-defined flags: CopyFile, which keeps the original
implementation and CopyFileNoOverwrite, which adds the new functionality.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Add test cases for directory and symlink.
Removed the check for mustNotExist, as there may be the case the same
hash is requested in the current version as in a previous
version. This happens for example in case of directories with same
permissions from different versions (case "G" and "H").
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
The previous commits make upstream bundle definition caching
automatically handled by the mixer commands that need them,
and thus this command is obsolete.
This command also had dubious side effects: while it was
described as fetching upstream bundles, it *also* set up
the mix-bundles directory. This side effect behavior is
now taken care of by init.
This commit also makes cosmetic edit to 'mixer init-mix',
renaming it to 'mixer init'.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Running 'mixer bundle add --all' fails with an array
out of bounds error, because the args array is still
being split, even if it is not passed.
I thought this bug was already found and fixed during
the CLI rewrite, but apparently it was missed or there
was a regression.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Split up the download/unpacking of upstream bundles and the
creation of the mix-bundles directory.
Fetching upstream bundles now lives in a standalone function that
is called by any function that relies on having upstream bundle
definitions. This means the tool automatically handles the caching
of upstream bundles, making the 'mixer bundle get' command
obsolete.
Creation of the mix-bundles directory has been moved to InitMix.
This patch also includes general cleanup and error handling,
especially for InitMix and AddBundles.
This patch has the following side-effects:
1) The upstream bundles are now considered temporary, sausage-
making data. As such, they now live in .mixer/upstream-bundles/,
and the .tar.gz file used to fetch them is cleaned up.
2) The method for fetching upstream bundles has an optional
parameter ('prune') that will clean up the bundle cache for other
versions of upstream. This should be considered the default value
as part of standard clean-up behavior; a value of false should only
be used in cases where a method needs to have multiple versions
of upstream bundles cached simultaneously.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
When unpacking upstream bundle defintions, previously an exec
was made to 'tar'. This patch replaces the exec call with a
native function that uses the Go tar and gzip packages instead.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
unchanged is a slice of []*Files in linkManifestAndChange that is not
used except by appending to. Remove for now, if it is needed later it
can be added at that time.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Delta peers are only necessary when the file name is the same, but the
contents have change. Avoid setting every identical delta peer and only
set when the files are not the same.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When parsing a file, fill in the Name field based on the
filename, essentially taking whatever comes after "Manifest.". If
doesn't match, just leave Name empty like before.
Small test added so we can keep track of edge cases in the future.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
The flag attributes are already exported, but the constants were not,
so export them. Re-order them in the file to be consistent with the
flag order. Flags are now represented by a single byte.
One option in the future is to map the constants directly to their
real values in the file (e.g. 'F', 'D'), instead of having a new
enumeration starting from zero.
Opted not to do the same with rename (fourth byte) since it is
currently mapped in a different way (boolean), but might make sense to
treat it as a modifier instead (as other extensions might reuse that
byte).
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Fixes#97
Print the correct error message with the correct flag names when
--clear-version or --mix-version is not supplied. Uses
cobra.MarkFlagRequired to mark these two flags as required.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Move go-ini/ini package (at version 1.32.0) to the toplevel vendor
folder. This library will be used by the build chroot code.
Also remove the inner swupd vendor folder now unused, and move the
README.vendor file to the toplevel, now that dep is used there.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Instead of requiring the new swupd code to create the "format" and
"swupd-server-src-version" metadata files enable BuildUpdate to do so.
Because the mixer version is now used instead of a swupd-server version
write the mixer-tools version to a "mixer-src-version" file. Move the
Version const to builder.go so it can be accessed trivially. Update the
Makefile parsing to point to builder.go and update the access method in
the top-level mixer/cmd/root.go.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Also add GetHashForFile and GetHashForBytes functions. Tests of
genHash were changed to test the equivalent GetHashForBytes function.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Expose a swupd.Hash type that can be written to and then asked for the
hash -- so no need to have all the contents bytes. Export a struct
with the metadata we need for hash calculation. This will help
calculating hash values based on the content of a tar file.
Also add some more details to how the hash work.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
The flag --cpu-profile let the caller specify a filename to write the
CPU profile information to. That can be later read with
$ go tool pprof FILENAME
More details on how to use the tool is in
https://blog.golang.org/profiling-go-programs.
The new flag is hidden from the default help message -- since it is
mostly a feature for developers.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
This is a --new-swupd replacement of mixer-pack-maker.sh script. There
are two ways to specify what delta packs to make: by setting a --from
version or by asking for (up to) K --previous-versions.
Unlike the script, only one --from is supported, if multiple specific
versions are needed, mixer must be called multiple times. The
rationale is the multiple --from was used to simulate
--previous-versions (by having the caller figuring out them), so not
very important anymore.
The implementation uses swupd.FindBundlesToPack to figure out
what (bundle, from, to) combinations it needs to build, then use
swupd.CreatePack to do the work.
The FindBundlesToPack was changed to take manifests instead of version
numbers and state dirs, the test program was updated accordingly.
Fixes#83.
Fixes#12.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
