Use a regular expression in order to exclude all directives at the
beginning of the filename, including %doc.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When cleaning directories from the package file lists do not attempt to
clean "%dir" prefixed files. Although autospec does not currently
support empty directories, it could in the future.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Clean directories from package file lists. If directories are
encountered, print a warning, add the directory to the blacklist, and
re-run.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Create a mock function that copies files from the testfiles
directory to a tmp directory, when attempting to download
files from the network.
Create a mock function for head_request that returns 404 for
a few specified urls, and 200 for the rest.
The mock calls will allow the functionality of the pkg_integrity
program to be tested and not dependent on networking. Tests will
no longer need to be skipped in travis with the removal of
network dependency.
Removed a few tests that were not specific to pkg_integrity, but
instead were testing the importing and exporting abilities
of the gpg key server.
Added mock as a requirement to the requirements.txt file.
Signed-off-by: Gabi Beyer <gabib@live.com>
For github URLs, capture and identify repo name separately from package
name. This is a building block for resolving cases where the package
name is actually different than the github repo name.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
For github URLs in tests/packageurls, check that we produce a giturl
that looks reasonably correct.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
Since moving output to a temporary directory, the "output" directory has
fallen out of use. Remove the output directory from autospec.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Tests for the %files section use bare file names. Since rpmbuild
requires leading slashes anyway, adding slashes to some of the tests.
Adding new tests for filenames with white space and/or rpm directives.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
Test almost 2000 URLs for proper version and name detection. This list
is sorted by the second column (expected name).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Rename set_mock to get_mock_cmd, and return the value instead of
setting a global variable.
This patch should not change any behavior.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Use 'sudo' unless /usr/bin/mock points to consolehelper. This is a
simpler heuristic than checking for the group membership and will let
us use the group membership in Clear Linux to give sudo permissions.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Adobe font packages have -nndpi as part of the name. Add tests and a
regular expression to capture these.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Debian is down again, and we've seen them go down in the past. We
shouldn't be running these tests in Travis.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This prevents collisions when two instances are trying to build the same
package in parallel. Unit tests added as well.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
These tests regularly fail in Travis for unknown reasons. Skip them
until the tests can be rewritten.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This allows python3 unittest to properly generate the tests when running
all the tests in the tests directory.
Also adds a fix to test_test.py to reset buildpattern.default_pattern
every test. This was a bug introduced by running all tests at once -
this was not being cleared correctly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
with the imlpementation of the write_out functionality,
some unittests need to be changed to use that fuction.
Signed-off-by: Gabi Beyer <gabrielle.n.beyer@intel.com>
Minor clean up of buildreq module, including the following changes:
- Remove unused import "tarball"
- Add function docstrings
- Convert to list-driven pattern checks instead of a long list of
conditionals.
- Rename variable names that were overriding keywords (file, buffer,
etc.)
- Remove unused setup_py_python3 function
- Remove unused arguments and variables from scan_for_configure function
(and corresponding function calls in test_buildreq.py and autospec.py)
Most changes made at pylint and pycodestyle suggestions.
Updates include generating an individual test for each url to be tested
instead of doing it all in one test and appending to an error list where
appropriate. This has the benefit of showing failures for each
individual test case in a recognizable manner.
Tests are also added for most of the remaining functions.
These tests do not pass without the accompanying tarball module cleanup
and restructure.
Some changes have recently been pushed to autospec sources without
updating the unit tests. This patch brings the unit tests up-to-date
again.
test_build.py: Remove go buildtool bits (was deprecated)
test_specdescription.py: Newline added to post-scripts
test_specfile.py: Reset license_translations dict
files.py: Check dict with get() in case of missing key
Add unit tests for the very large count.py module/script. These unit
tests attempt to test every regular expression present in the parse_log
function and helped catch several bugs in the original regexes ported
from perl.
Clean up the specdescription module, including resolving two bugs and
making the code more readable and more robust.
BUGS:
- description_from_pkginfo
A string was checked for a substring using the string.find() method
but was treated as a boolean, assuming that find() returned 0 when the
substring was not found. This was causing a section end to be
immediately detected and the section was not parsed. The call was
replaced with the `in` keyword.
- description_from_spec
The default description was overwritten regardless of whether the
detected description had any content. A check was added to only
overwrite the default if there was a detected description.
Other general cleanup:
- Remove unused default_group global
- Remove unused argument in summary_from_R
- Add assign_summary/description methods to reduce global usage and
reduce code repetition
- Add function docstrings
- Convert to `with open()` blocks when working with files instead of
explicitly handling closing the files
- Wrap open() calls in try...catch blocks
Unit tests added for the specdescription module. At the time of this
commit, there are two failing tests due to bugs in the module.
Failures:
- test_description_from_pkginfo
This test fails because a string.find() method is checked as a boolean
as if find returned 0 if the substring was not present. find()
actually returns -1 in this case. Because find() was being used as a
guard to find the end of a section, the end was found immediately and
the section was not parsed.
- test_description_from_spec_no_info
This test fails due to the default description being overridden, even
when the override is just an empty string.
Shortcomings were identified in setup.py parsing for python build
requirements. These tests more exhaustively test the cases we have been
seeing in the wild.
This patch makes some simplifications to the buildreq module and related
modules (autospec and config). These are mostly readability changes and
code cleanup.
The complexity of the buildpattern has been significantly reduced with
the move to a centralized specfiles class to handle specfile writes.
This patch tests the remaining functionality and fixes a typo in
buildpattern.py (pattern_strengh -> pattern_strength).
Instead of creating in the current directory, which assumes autospec is
being run via a make target from the package directory, create
options.conf in the target directory. This insures the configuration
file will be placed correctly when autospec is run directly with a
different target directory provided.
The lib_to_so boolean option in options.conf for a specific package
causes files ending with .so to be added to the lib subpackage instead
of the dev subpackage. This is helpful for packages with a plugin
architecture that use .so files as their plugins. Specific examples of
the packages this is helpful for is mesa and elfutils.
Functional tests updated.
A proxy redirect to localhost prevents package builds from accessing the
internet (they will receive a connection refused). Package build access
to the internet is a security issue if successful, and a very long
timeout if the build is behind an actual proxy.
Update the functional tests to reflect this change.
Implemented support for hash verification of packages from
download.gnome.org. This commit includes tests for domain
based verification (pypi ang gnome)
Update functional tests to reflect the recent version and name detection
changes, such as the addition of the [configuration] section of the
options.conf file.
Due to an 'nargs' specification in the argument parser, the name
argument was being read as a list. The list was then always accessed as
name_argument[0] to access the name. This patch removes the 'nargs'
specifier so name is always a string and removes the logic to parse the
list.
If both version and name arguments exist, set them and skip automated
detection. If one is not set, continue with the automated detection
since the URL parsing for both are done together, override at the end.
Autospec is unable to accurately parse every type of URL for the correct
name and version of the package. When this is the case, the detected
version can be overwritten by the --version argument on the command
line.
Instead of manually searching for identifying strings in package
licenses, rely on the configuration file and license server to set the
license information. Remove unit tests that tested these functions.
Convert license_blacklist back to dict for faster lookup times.
Several changes to the license module including changes to where local
hashes are stored. Tests were added as well. Detailed changes below.
config:
- Move the license hashes from an internal repo to a config file
(license_hashes) in autospec.
test_license:
- Unit tests added
- Unit test target added to autospec Makefile
- COPYING_TEST file added with GPL-3.0 text for test purposes
license:
- scan_for_licenses: separate out search strings from conditional to
make the code more readable and remove unused parameter (required a
change to where autospec.py called the function).
- remove duplicate licenses from license_translation list.
- convert license_blacklist dict to list (values were unused, it was
being treated as a list anyways).
- separate out search strings from code in license_from_copying_hash
and license_from_doc.
- move curl code out of `with open(file) as fname:` clause in
license_from_copying_hash.
Public key import was not cleanning after running. With this
change the state of the keyring will be reverted to the
original state after the test runs.
The files module was being treated like a class, in that it was
necessary to keep state throughout the entire autospec run. Convert the
files module to a class and make some other general clean ups. This
changed how the files module was instantiated and called by the main
autospec module and other modules such as tarball and config.
Test the files module with unit tests and add the target to the
makefile.
Autospec previously used an unsecure public temporary directory as its
working directory. This patch creates a secure temporary directory only
accessible by the creating user ID.
Matthew Johnson