This release adds some new features and optimizations to udpates.
* Apply all deltas from the pack, not just the renames identified in the
manifest.
* Do not retry 404 downloads.
* Allow updates to specific version with the -m flag.
Additionally, some other minor issues were fixed, including a bug in
time stat reporting.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Change load_mom to have a variant that can propagate the error code,
and check for that. Use that variant when downloading the
server_manifest. This approach could later be applied to other retries
that involve downloading.
With this patch, "update -m" to a non-published version will not retry
many times before failing.
Allow updating to a version greater than current using -m. Also
supports 'latest' that has the same behavior as not passing any flag.
If the requested version is smaller than the server version (which
is the latest one available in the current format), use that. If not,
use the latest available and re-execute.
Care was taken so that a "update -m LAST_OF_A_BUMP" will not trigger a
re-execution of update.
At the moment version_url is still used, but a future improvement
could avoid it by first looking at the format of the given version
first.
Fixes#257.
Makes life easy to add extra flags without much churn or globals, also
follows what other commands do. This will free main.c to be used by
the real main() of swupd.
Add "swupd verify" options that were missing from the documentation
(swupd.1.rst and swupd.1) but listed by the tool when calling
"swupd verify -h". The order was also slightly changed to be
identical to the order used by "swupd verify -h".
Signed-off-by: Geoffroy Van Cutsem <geoffroy.vancutsem@intel.com>
The two last uses of rename flag were:
- When creating the update list, include old renames present in the
new manifest if the file mentioned in the rename is present in the
current manifest. These would be used to link renames (and try to
unpack relevant deltas).
Why is it OK to remove? Linking renames doesn't happen anymore,
since we get the deltas from other means. The new versions of files
will be included in the update, so we are not missing anything. This
was just making more files available to later steps.
- When consolidating files (i.e. removing duplication), the code
explictly had cases to handle when rename happen or not in case only
one file was deleted.
Why is it OK to remove? In the current code, the side-effect of
having a rename or not doesn't alter the result. This can be seen by
the fact that, at the point cases 2-5 were evaluated, either file1
or file2 are deleted, and that is the only factor that decides which
file to pick. As the code evolved from the time of the comment, this
became more evident.
Updated the commentary in consolidate_files() replacing the table with
a more detailed description of each case (now that we have less
cases). Added some explanations about how we decide with file to pick
when both are present or not present
Consider all deltas from a pack instead of only those who are involved
in a rename. This will allow the server to make better packs
regardless of the from/to version, with the possibility of a delta to
a file that was not necessarily marked with the rename flag.
This makes deltapeer member of file unused, so it was removed. The
peer member continues to be used for its original role: mapping the
same files (same filename) between two manifests, usually the state of
the local system and some new version to be installed.
The test case added is like test/functional/update/rename, but without
the rename flags present. Since in that test there are some fullfiles
missing, it will only pass if swupd-client applies the deltas in the
pack.
This release fixes some more memory management issues with
un-initialized variables and cloned lists and removes the unused
libswupd library.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The clone of strings lists was creating a copy of the items and adding
to the list, but still keeping the old ones around. Change the
function to just clone the items. This was causing invalid reads when
doing swupd search for a term that has matches, and causing crashes
for some users.
For the record, one of the valgrind reports:
==30978== Invalid read of size 1
==30978== at 0x51FF680: __strcmp_ssse3 (in /usr/lib64/haswell/libc-2.27.so)
==30978== by 0x407C1B: calculate_size (search.c:194)
==30978== by 0x407C62: calculate_size (search.c:219)
==30978== by 0x407C62: calculate_size (search.c:219)
==30978== by 0x4075CD: apply_size_penalty (search.c:240)
==30978== by 0x4075CD: do_search (search.c:715)
==30978== by 0x4075CD: search_main (search.c:898)
==30978== by 0x50B4B36: (below main) (libc-start.c:308)
==30978== Address 0x65484b0 is 16 bytes before an unallocated block of size 0 in arena "client"
This release fixes the 2TB contentsize limit check and adjusts the
swupd-client.timer unit to check for updates more frequently.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
We know these are cheap and our CDN takes care of the actual
bulk of the bandwith.
Spread the update checks between 45min0sec and 1hr33mins
effectively.
This release makes some bug and memory management fixes.
* swupd-add-pkg bug that prevented creating a local bundle with the same
name as the package it includes was fixed.
* A bug that caused bundle-list to print garbage for systems with a
large number of bundles was fixed.
* The swupd completion code was updated and filename completion for
swupd hashdump was added.
* Protect against bogus filecount heap corruption.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Make swupd save a copy of the current MoM in a readable location, so
the completion code for bundle-add has access to it.
Stop generating the completion code, just use the file directly.
Remove the short option from the completion code, whilst they are
useful for an expert there is little point typing minus tab and then a
letter rather than minus and the letter.
Add crude filename completion for hashdump.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
The directory reading routines do not promise that the entries remain
valid whilst the directory is open. In particular if the directory is
more than 4k (one stdio buffer) in size then the names will be invalid.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
When we run out of memory call abort() for consistency with
string_or_die.
This doesn't address the issue that nF * sizeof(struct filerecord)
might not fit into size_t. For this to be a problem we would need more
than 400,000,000 files in the manifests on a 32bit system.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Fixes#412
A maliciously constructed manifest could arrange for an insanely large
number of files to be claimed. This parameter was then being passed to
malloc without any further validation, potentially causing heap
corruption if the value was such that when multiplied by
sizeof(struct file) it would overflow an integer. Clamp the value to
no more than 4 million to avoid this attack.
Reported by ktwo@ktwo.ca
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Create a test-bundle manifest with a stupid number of files, see that
it is rejected. Unfortunately because of retries this takes a long
time (70 seconds) time to run.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
The previous value - 13,805,671,819 is bogus, and we want to be able
to test if the value is reasonable.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
This release updates the swupd-add-pkg script to work with the latest
changes to the mixer tooling it relies on.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Fix the script to propagate the old (unchanged) custom bundles to the new
merged MoM manifest. Small errors were fixed as well with reading the correct
- initially unmerged - manifest, and not adding a package to a bundle if it
already exists.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release fixes mixer integration with the bundle list command
(previously it would try to use the server MoM for the local mix
version).
Swupd search is improved to report real recursive contentsize for bundle
results as a sum of included bundles. Search now displays all results by
default with an optional --top NUM flag to only display NUM results per
bundle. Another flag -m, --csv outputs all results in CSV format so they
can be more easily parsed by a machine.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of silently truncating search results to 5 files per 5 bundles
make the default search output a full list grouped by bundle. This
output will only be sorted by increasing bundle size.
Add a -t, --top=NUM argument to tell search to truncate results at NUM
files per bundle. This option attempts to display the top results based
on a set of heuristics.
When truncating results due to --top add a message when the file results
are cut off.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Since bundle size might be inaccurate in this case just do not display
it, or even calculate the size.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When reporting bundles for swupd search include their included bundles'
contentsizes as well for more accurate size reporting. For bundles that
are not installed on the client system, this skips any installed
includes and only reports contentsize for uninstalled includes. For
bundles that are installed on the system calculate the entire
contentsize for all includes and report this as the "installed size" of
the bundle.
When the scope is 'o' (one hit only) do not print size information.
Since only the first hit was taken it is unlikely that contentsize was
calculated for all that bundle's includes.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release adds the 'swupd clean' command to clean up old files from
the statedir, adds an indication to 'swupd search' to show if a bundle
is installed, updates the copyright header, and updates mixer
integration to use the new mixer-tools 4.0.x interface.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This command removes old files from statedir. The interface is a bit
opaque so there will be room to modify the policy later.
By default it will delete staged files that are older than
DAYS_TO_KEEP_FILES. For manifests this heuristic is not good as older
manifests might still be used for the current OS version being run, so
also ensure that those do not get deleted. This prevents 'swupd
search' to redownload files.
The default behavior should be suitable to use in combination with
automatic updates, to control the size of state dir. There is also an
--all option to remove all the state files regardless of dates and
usage.
To avoid "disasters" in case some paths are set wrong, the command
explicitly delete patterns of files create by swupd.
When displaying a search result for a bundle display "installed" next to
bundles that have been added to the system.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes the "Hardlink target error" when extracting packs and
improves swupd search output.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Remove some of the magic numbers, add some comments, fix some code
style, remove an unnecessary string duplication, free dynamically
allocated memory in response to feedback on PR #387.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
While reviewing this I typed down a few notes as I was looking at how
this was implemented. Keep these notes as comments in the code.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This patch improves the output of swupd search by grouping search
results by bundle. Each file and bundle result is ranked according to a
set of heuristics such as bundle size and path name. The top five bundle
results (and the top five file results in each bundle) are displayed
while the rest of the output is truncated.
In some cases packs may include files with entries of type "hard
link", that refer to previous files in the pack. Libarchive was
failing since it couldn't find the previous file in disk to archive,
because we change its output path.
Adjusting the hardlink target path (when it exists) fix the
issue. This is also what bsdtar does.
This patch was tested by simulating the update that goes from 19460 to
19580 with the desktop pack installed (that contains one hardlink in
the pack used by this update).
Fixes#351.
This release fixes various bugs
* Remove invalid memory reads when listing local bundles
* Fixes init/deinit of all bundle list variants
* Sorts output of installable bundles
* Fixes invalid call to stat when FALLBACK_CAPATHS are empty
* Fixes mixer integration bundle-add/remove
* Fixes mixer integration upstream url handling
* And fixes other minor internal issues
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Matthew Johnson