Modify license skips to handle the common tempfile prefix of '/tmp/*'
and normalize the case where lines lead with a slash vs not.
Also remove a duplicate test for globlike match.
Signed-off-by: William Douglas <william.douglas@intel.com>
Add support for a new config file with glob like handling of paths as
some projects contain license files that are not the license of the
project.
scan_for_licenses now looks at the configuration and does glob like
matching for all the potential license files it tries to add.
This change caused a few structural adjustments to the code and a
slight style change for importing just the toplevel util submodule
that are superficial.
Signed-off-by: William Douglas <william.douglas@intel.com>
Many different components require the target download path and it is
needed early. Previously code existed to attempt to autodetect its
location and so defining the path was enforced later in the build
process. This created problems with ordering functions based on this
requirement but since the target path is now certain to be defined at
startup (due to previous changes) we can better align the location and
initialization of the data to the Config class.
As part of this change also avoid requiring the temporary working
directory to be setup as part of the Build class. This is done as a
cleanup due to the only real user of this location being the Content
class.
This also drops the stand-alone pkg_integrety main function that was
initially used as a one off test to avoid needing to run autospec to
validate the integrity of a package. This feature was seldom used and
doesn't function well without handling configuration so it was
dropped.
The tarball module had a number of globals that were referenced by
many other modules and has ordering dependencies with the config
module for some values. This made deciding on where certain values get
initialized difficult but before the initialization can be addressed a
refactor is helpful.
This change moves the global state (and functions that needed to
act on that global state) into a Content class. The goal of this work
is to better track what can be updated by a particular function, load
data in a sensible order and have it owned by a sensible component.
I expect that future work will see the tarball, buildpattern and some
portions of the config moved around to better match what component
should own what data (and to better deliniate between the data that
has a similar use but comes from different sources).
The config module had a large amount of globals that were being
touched across many modules that would import. This made changes to
config very fragile as figuring out what would be modified in any
given call chain was difficult to diagnose.
It also made testing fragile as one would need to reset a given
module's config import to the best of their knowledge before rerunning
another test.
To get away from that (and to try and reduce the number of globally
modified variablies in autospec as a whole), refactor the config
module to provide its state as part config class. The long running
goal of changes like this is to better track what content can be
updated by a particular function (if a function would have access
to the config instance is now the hint rather than the config module
getting imported).
We may get a message "<package>.license not found" even
if the file exists.
The <package>.license file is manually created but the
licenses in the file may be silently rejected due to being
blacklisted.
For example adding "GPL" or "BSD" to abc.license while autospecing
a package "abc" can produce the error:
"[FATAL] Cannot find any license or abc.license file!"
eventhough the abc.license manifestly exists.
This patch modifies the original error message and also prints out
some additional messages, such as:
[WARNING] abc.license: blacklisted license GPL ignored.
[WARNING] abc.license: blacklisted license BSD ignored.
...
[FATAL] Cannot find any license or a valid abc.license file!
Note: With this patch "add_license" only fails (returns False) if trying
to add a blacklisted license, it always returns True for non-blacklisted
licenses. This required to change the unit test "test_add_license_present"
accordingly.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
The download code for licenses, tarballs, and signature files are
similar enough to warrant consolidation into a unified interface, so add
a new function do_curl() that satisfies the needs of all the callers.
Note that I made one change that noticeably changes the output,
hopefully to reduce confusion: The signature download loop no longer
prints error messages, since the end user should not need to care at all
unless a signature cannot be downloaded and verification is required. If
the latter case is true, autospec will still exit and print the fatal
error, so the issue should be straightforward to diagnose after seeing
that error.
A couple of bugs are fixed as a result of using the new function:
- The FAILONERROR pycurl option is now set for tarball downloads. This
prevents certain undesirable side-effects like downloading 404
response pages.
- Responses from FTP servers are now handled better, since a more
common "success" code from FTP servers is 226, not 200.
And unit tests have been updated:
- Added new tests for download.do_curl().
- Replaced "mock" import with "unittest.mock" for pkg_integrity tests.
- Revised license and pkg_integrity tests to reflect the new download
paths.
- Added missing mocks for the download paths in PyPIVerifier and
GEMShaVerifier.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The additional test uncovered a syntax error in the print_fatal call in
license_from_copying_hash, which was fixed as well. An unnecessary mock
was removed from another test.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of manually searching for identifying strings in package
licenses, rely on the configuration file and license server to set the
license information. Remove unit tests that tested these functions.
Convert license_blacklist back to dict for faster lookup times.
Several changes to the license module including changes to where local
hashes are stored. Tests were added as well. Detailed changes below.
config:
- Move the license hashes from an internal repo to a config file
(license_hashes) in autospec.
test_license:
- Unit tests added
- Unit test target added to autospec Makefile
- COPYING_TEST file added with GPL-3.0 text for test purposes
license:
- scan_for_licenses: separate out search strings from conditional to
make the code more readable and remove unused parameter (required a
change to where autospec.py called the function).
- remove duplicate licenses from license_translation list.
- convert license_blacklist dict to list (values were unused, it was
being treated as a list anyways).
- separate out search strings from code in license_from_copying_hash
and license_from_doc.
- move curl code out of `with open(file) as fname:` clause in
license_from_copying_hash.
William Douglas