This prevents collisions when two instances are trying to build the same
package in parallel. Unit tests added as well.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
These tests regularly fail in Travis for unknown reasons. Skip them
until the tests can be rewritten.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This allows python3 unittest to properly generate the tests when running
all the tests in the tests directory.
Also adds a fix to test_test.py to reset buildpattern.default_pattern
every test. This was a bug introduced by running all tests at once -
this was not being cleared correctly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
with the imlpementation of the write_out functionality,
some unittests need to be changed to use that fuction.
Signed-off-by: Gabi Beyer <gabrielle.n.beyer@intel.com>
Minor clean up of buildreq module, including the following changes:
- Remove unused import "tarball"
- Add function docstrings
- Convert to list-driven pattern checks instead of a long list of
conditionals.
- Rename variable names that were overriding keywords (file, buffer,
etc.)
- Remove unused setup_py_python3 function
- Remove unused arguments and variables from scan_for_configure function
(and corresponding function calls in test_buildreq.py and autospec.py)
Most changes made at pylint and pycodestyle suggestions.
Updates include generating an individual test for each url to be tested
instead of doing it all in one test and appending to an error list where
appropriate. This has the benefit of showing failures for each
individual test case in a recognizable manner.
Tests are also added for most of the remaining functions.
These tests do not pass without the accompanying tarball module cleanup
and restructure.
Some changes have recently been pushed to autospec sources without
updating the unit tests. This patch brings the unit tests up-to-date
again.
test_build.py: Remove go buildtool bits (was deprecated)
test_specdescription.py: Newline added to post-scripts
test_specfile.py: Reset license_translations dict
files.py: Check dict with get() in case of missing key
Add unit tests for the very large count.py module/script. These unit
tests attempt to test every regular expression present in the parse_log
function and helped catch several bugs in the original regexes ported
from perl.
Clean up the specdescription module, including resolving two bugs and
making the code more readable and more robust.
BUGS:
- description_from_pkginfo
A string was checked for a substring using the string.find() method
but was treated as a boolean, assuming that find() returned 0 when the
substring was not found. This was causing a section end to be
immediately detected and the section was not parsed. The call was
replaced with the `in` keyword.
- description_from_spec
The default description was overwritten regardless of whether the
detected description had any content. A check was added to only
overwrite the default if there was a detected description.
Other general cleanup:
- Remove unused default_group global
- Remove unused argument in summary_from_R
- Add assign_summary/description methods to reduce global usage and
reduce code repetition
- Add function docstrings
- Convert to `with open()` blocks when working with files instead of
explicitly handling closing the files
- Wrap open() calls in try...catch blocks
Unit tests added for the specdescription module. At the time of this
commit, there are two failing tests due to bugs in the module.
Failures:
- test_description_from_pkginfo
This test fails because a string.find() method is checked as a boolean
as if find returned 0 if the substring was not present. find()
actually returns -1 in this case. Because find() was being used as a
guard to find the end of a section, the end was found immediately and
the section was not parsed.
- test_description_from_spec_no_info
This test fails due to the default description being overridden, even
when the override is just an empty string.
Shortcomings were identified in setup.py parsing for python build
requirements. These tests more exhaustively test the cases we have been
seeing in the wild.
This patch makes some simplifications to the buildreq module and related
modules (autospec and config). These are mostly readability changes and
code cleanup.
The complexity of the buildpattern has been significantly reduced with
the move to a centralized specfiles class to handle specfile writes.
This patch tests the remaining functionality and fixes a typo in
buildpattern.py (pattern_strengh -> pattern_strength).
Instead of creating in the current directory, which assumes autospec is
being run via a make target from the package directory, create
options.conf in the target directory. This insures the configuration
file will be placed correctly when autospec is run directly with a
different target directory provided.
The lib_to_so boolean option in options.conf for a specific package
causes files ending with .so to be added to the lib subpackage instead
of the dev subpackage. This is helpful for packages with a plugin
architecture that use .so files as their plugins. Specific examples of
the packages this is helpful for is mesa and elfutils.
Functional tests updated.
A proxy redirect to localhost prevents package builds from accessing the
internet (they will receive a connection refused). Package build access
to the internet is a security issue if successful, and a very long
timeout if the build is behind an actual proxy.
Update the functional tests to reflect this change.
Implemented support for hash verification of packages from
download.gnome.org. This commit includes tests for domain
based verification (pypi ang gnome)
Update functional tests to reflect the recent version and name detection
changes, such as the addition of the [configuration] section of the
options.conf file.
Due to an 'nargs' specification in the argument parser, the name
argument was being read as a list. The list was then always accessed as
name_argument[0] to access the name. This patch removes the 'nargs'
specifier so name is always a string and removes the logic to parse the
list.
If both version and name arguments exist, set them and skip automated
detection. If one is not set, continue with the automated detection
since the URL parsing for both are done together, override at the end.
Autospec is unable to accurately parse every type of URL for the correct
name and version of the package. When this is the case, the detected
version can be overwritten by the --version argument on the command
line.
Instead of manually searching for identifying strings in package
licenses, rely on the configuration file and license server to set the
license information. Remove unit tests that tested these functions.
Convert license_blacklist back to dict for faster lookup times.
Several changes to the license module including changes to where local
hashes are stored. Tests were added as well. Detailed changes below.
config:
- Move the license hashes from an internal repo to a config file
(license_hashes) in autospec.
test_license:
- Unit tests added
- Unit test target added to autospec Makefile
- COPYING_TEST file added with GPL-3.0 text for test purposes
license:
- scan_for_licenses: separate out search strings from conditional to
make the code more readable and remove unused parameter (required a
change to where autospec.py called the function).
- remove duplicate licenses from license_translation list.
- convert license_blacklist dict to list (values were unused, it was
being treated as a list anyways).
- separate out search strings from code in license_from_copying_hash
and license_from_doc.
- move curl code out of `with open(file) as fname:` clause in
license_from_copying_hash.
Public key import was not cleanning after running. With this
change the state of the keyring will be reverted to the
original state after the test runs.
The files module was being treated like a class, in that it was
necessary to keep state throughout the entire autospec run. Convert the
files module to a class and make some other general clean ups. This
changed how the files module was instantiated and called by the main
autospec module and other modules such as tarball and config.
Test the files module with unit tests and add the target to the
makefile.
Autospec previously used an unsecure public temporary directory as its
working directory. This patch creates a secure temporary directory only
accessible by the creating user ID.
This shortcoming was identified when a github package URL was changed to
http from https. The 's' character is made optional when parsing the
github URL and two test cases are added to the packageurl list for
testing.
When running build_and_run for test_autospec if subprocess.check_output
throws an exception, output was a byte encoded string and a later test
to see if a unicode string was contained in that byte encoded string
would throw an exception.
Instead of doing the decode in a try block put it in finally. This does
add the posibility of utf-8 decode failing but that's fatal later on
either way.
Version detection could be considerably simplified using builtin python
functions such as strip, replace, and list comprehension. The logic was
also modified a bit to provide more consistent version strings
containing only numbers and periods '.'. More unit tested urls were
added to tests.
9.11.0-P1 -> 9.11.0.1
1.0.0-rc1 -> 1.0.0.1
7_4_2 -> 7.4.2
2.1-3 -> 2.1.3
Some non-digit strings are conserved for upstream version update detection:
2.1-beta1 -> 2.1.beta1
2.1-pre8 -> 2.1.pre8
3.5.0b2 -> 3.5.0b2
This patch allows users to specify requires_add and requires_ban to
augment the automatic runtime requirement handling.
As part of this change, the handling of python requires has been
modified so that they are managed under generic requires instead of
their own special mechanism. python requires detection was also improved
by adding install_requires section contents when the section is detected
in setup.py.
The primary change aside from python handling is to remove main_requires
handling in files.py and consolidate requires fully in buildreq.py.
With this change python packages in pypi will be
verified using the MD5 hash posted in pypi along
the package if all the other verifications are
not possible.
This patch adds handling for two cases for interactive
mode. 1- when non pty, the interactive mode is disabled
, and 2- timeout input function, when user fails to type
an input in interactive mode the script will continue
after a timeout.
Matthew Johnson