pypi is deprecating the API and doesn't recommend its usage for
integrity checks so stop trying to make use of it in autospec.
Signed-off-by: William Douglas <william.douglas@intel.com>
These patterns are being removed as they are not used or are going to
be reworked.
This change also removes support for multi-versioned packages which
were considered for go and cargo purposes.
Signed-off-by: William Douglas <william.douglas@intel.com>
All sdist tarballs available on pypi appear to have both an md5sum and
sha256sum listed, so switch to verify the stronger hash (sha256)
instead.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Many different components require the target download path and it is
needed early. Previously code existed to attempt to autodetect its
location and so defining the path was enforced later in the build
process. This created problems with ordering functions based on this
requirement but since the target path is now certain to be defined at
startup (due to previous changes) we can better align the location and
initialization of the data to the Config class.
As part of this change also avoid requiring the temporary working
directory to be setup as part of the Build class. This is done as a
cleanup due to the only real user of this location being the Content
class.
This also drops the stand-alone pkg_integrety main function that was
initially used as a one off test to avoid needing to run autospec to
validate the integrity of a package. This feature was seldom used and
doesn't function well without handling configuration so it was
dropped.
The config module had a large amount of globals that were being
touched across many modules that would import. This made changes to
config very fragile as figuring out what would be modified in any
given call chain was difficult to diagnose.
It also made testing fragile as one would need to reset a given
module's config import to the best of their knowledge before rerunning
another test.
To get away from that (and to try and reduce the number of globally
modified variablies in autospec as a whole), refactor the config
module to provide its state as part config class. The long running
goal of changes like this is to better track what content can be
updated by a particular function (if a function would have access
to the config instance is now the hint rather than the config module
getting imported).
There are only two GPG packet types that autospec needs to know about
for now ("signature" and "user ID"), so update `parse_gpg_packets` to
only add those packets for tracking.
Also, update unit tests to account for packets that are now skipped,
thus reducing the size of the packets list.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The new QtIoVerifier class is largely based on GnomeOrgVerifier.
Also add a unit test for the successful verification case.
(Most of qt package tarballs are large, so I picked the smallest one,
qtspeech, which is 98KB.)
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The download code for licenses, tarballs, and signature files are
similar enough to warrant consolidation into a unified interface, so add
a new function do_curl() that satisfies the needs of all the callers.
Note that I made one change that noticeably changes the output,
hopefully to reduce confusion: The signature download loop no longer
prints error messages, since the end user should not need to care at all
unless a signature cannot be downloaded and verification is required. If
the latter case is true, autospec will still exit and print the fatal
error, so the issue should be straightforward to diagnose after seeing
that error.
A couple of bugs are fixed as a result of using the new function:
- The FAILONERROR pycurl option is now set for tarball downloads. This
prevents certain undesirable side-effects like downloading 404
response pages.
- Responses from FTP servers are now handled better, since a more
common "success" code from FTP servers is 226, not 200.
And unit tests have been updated:
- Added new tests for download.do_curl().
- Replaced "mock" import with "unittest.mock" for pkg_integrity tests.
- Revised license and pkg_integrity tests to reflect the new download
paths.
- Added missing mocks for the download paths in PyPIVerifier and
GEMShaVerifier.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Due to the test->check file rename, breakages in testing for
pkg_integrity were found. This uncovered actual problematic leftovers
in the GPGVerifier from the move to a stand alone download_file
function for all signatures.
Tests then needed to be updated to avoid running downloads as part of
their validation but we should add tests mocking the pycurl calls and
testing the rest of download_file at some point.
Instead of probing the url with HEAD, download the first available
signature, if any, and pass it to validator.
The concept of "validation from url" is, thus, gone. Downloading the
signature file is no longer part of the validation.
Keys now must be stored in the individual packages. autospec will import them
there (as opposed to this central locations). Packages were updated with their
current pub keys where possible.
Create a mock function that copies files from the testfiles
directory to a tmp directory, when attempting to download
files from the network.
Create a mock function for head_request that returns 404 for
a few specified urls, and 200 for the rest.
The mock calls will allow the functionality of the pkg_integrity
program to be tested and not dependent on networking. Tests will
no longer need to be skipped in travis with the removal of
network dependency.
Removed a few tests that were not specific to pkg_integrity, but
instead were testing the importing and exporting abilities
of the gpg key server.
Added mock as a requirement to the requirements.txt file.
Signed-off-by: Gabi Beyer <gabib@live.com>
Debian is down again, and we've seen them go down in the past. We
shouldn't be running these tests in Travis.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
These tests regularly fail in Travis for unknown reasons. Skip them
until the tests can be rewritten.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of creating in the current directory, which assumes autospec is
being run via a make target from the package directory, create
options.conf in the target directory. This insures the configuration
file will be placed correctly when autospec is run directly with a
different target directory provided.
Implemented support for hash verification of packages from
download.gnome.org. This commit includes tests for domain
based verification (pypi ang gnome)
Public key import was not cleanning after running. With this
change the state of the keyring will be reverted to the
original state after the test runs.
With this change python packages in pypi will be
verified using the MD5 hash posted in pypi along
the package if all the other verifications are
not possible.
This patch adds handling for two cases for interactive
mode. 1- when non pty, the interactive mode is disabled
, and 2- timeout input function, when user fails to type
an input in interactive mode the script will continue
after a timeout.
This flag will be used for package verification interactive
mode, currently package verification will attempt to import
a public key if a public key id is found. With the addition
of this flag is possible to disable the default interactive
mode and instead fail the verification if the public key is
not already in the keyring.
In some environments it might be neccesary to pass
http_proxy env var if this is not set otherwise the
attempt of find a key will fail since gpg will not
be able to find a keyserver if behind proxy.
When autospec fails to find a key it just quits
even when a keyid exists. With these changes users
will have the option to import a key without
quiting autospec.
Add the package signature URL as a Source99 file to include it with the
source rpm. This allows the signature to be more traceable and
accessible even without a network connection.
Since pkg_integrity now depends on config.py, it is necessary to change
how imports are done in the tests so mocking can be achieved. This also
required renaming the tests/pkg_integrity.py file to
tests/test_pkg_integrity.py to avoid a name collision. Before each test
the necessary config option is set correctly and the configuration file
rewrite is mocked away.
William Douglas