When the user chooses to import a public key during the autospec run
display information on that key in the commit message.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Improve rust/cargo support by:
1) Better handling only toplevel Cargo configuration files for dependency
lookup.
2) Adding runtime requirements for buildreq packages in order to get
dependencies from non directly dependent packages resolved.
3) Add cargo as a buildreq for rust packages to handle the rustc/cargo
split.
4) Add cargo detection from failed_commands.
5) Fix rust source directory destination to not append version in case
the package name already had version information in it.
6) Get name and version information from crates.io package files.
configparser.ConfigParser attempts to interpolate "%" characters in the
values of the key-value pairs when parsing. This is absolutely unwanted
behavior, especially when parsing URLs such as
<sourceforge url>/joe-editor/files/JOE%20sources/<joe tar>
which cause an uncaught exception.
Turn off interpolation by passing the interpolation=None argument to
ConfigParser.
Add a unit tests to report when ConfigParser(...) is called in the
codebase without setting interpolation to None.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This test suite is extremely hard to keep up-to-date and is no longer as
valuable now that unit tests have been written for much of the code
base. Actual functional testing can be done against package repos
without requiring them to be included in the autospec repository.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Get the source url to set the url if one is not passed into
autospec. This allows a user to use an infile file as the
source for the tarball, and prevents them having to manually
find the url. It also requires that the variable PV and ROS_SPN
are replaced with their correct strings (version and name,
respectively).
Changed the name of the infile_parsers.py file to infile_bb_parser
since it is specific to recipe files.
Have string read into lists and joined to fix spacing issues.
Signed-off-by: Gabi Beyer <gabib@live.com>
Create 3 separate infile files, the infile_handler reads in the
inputs and determnes how to parse them. The --infile argument now
allows a url, file, or directory of files to be passed as the
input. The infile_handler determines that type of input, calls the
correct parser depending on the filetype, and calls the update to
the specfile.
The infile parsers contains the scraping and parsing for all file
types. Currently it scrapes both .bb and .inc files and stores the
data into one dictionary. Because these files are both for recipes
their data corresponds and the .inc file takes presedence.
The infile update spec file udpates aspects of the specfile that
have been scraped and stored into a dict from the infile parsers
file. If there exists a value for ROS_SPN in the .bb file, use that
when replace the ROS_SNP value in the replace PV function.
Signed-off-by: Gabi Beyer <gabib@live.com>
Add comments to the update functions (summary, license, build
deps) within the funcion.
Add an update_license function that appends any licenses scraped
from the bb file to the autospec licenses list if they don't
already exist.
Also added unittests for functionality
Signed-off-by: Gabi Beyer <gabib@live.com>
Add --infile argument that takes a url, path, or directory
to a bitbake/recipe file. It scrapes the .bb/.inc file for
configuration data and stores it in a dictionary. This
architecture can be used for multiple file formats.
Also adds some initial unit tests for scraping functionality.
Signed-off-by: Gabi Beyer <gabib@live.com>
In this case, the project uses one GitHub repo to distribute different
packages. The existing extraction rules are not enough
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
The additional test uncovered a syntax error in the print_fatal call in
license_from_copying_hash, which was fixed as well. An unnecessary mock
was removed from another test.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The package metadata is now written to options.conf so the Makefile
writing is no longer needed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Use a regular expression in order to exclude all directives at the
beginning of the filename, including %doc.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When cleaning directories from the package file lists do not attempt to
clean "%dir" prefixed files. Although autospec does not currently
support empty directories, it could in the future.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Clean directories from package file lists. If directories are
encountered, print a warning, add the directory to the blacklist, and
re-run.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Create a mock function that copies files from the testfiles
directory to a tmp directory, when attempting to download
files from the network.
Create a mock function for head_request that returns 404 for
a few specified urls, and 200 for the rest.
The mock calls will allow the functionality of the pkg_integrity
program to be tested and not dependent on networking. Tests will
no longer need to be skipped in travis with the removal of
network dependency.
Removed a few tests that were not specific to pkg_integrity, but
instead were testing the importing and exporting abilities
of the gpg key server.
Added mock as a requirement to the requirements.txt file.
Signed-off-by: Gabi Beyer <gabib@live.com>
For github URLs, capture and identify repo name separately from package
name. This is a building block for resolving cases where the package
name is actually different than the github repo name.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
For github URLs in tests/packageurls, check that we produce a giturl
that looks reasonably correct.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
Since moving output to a temporary directory, the "output" directory has
fallen out of use. Remove the output directory from autospec.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Tests for the %files section use bare file names. Since rpmbuild
requires leading slashes anyway, adding slashes to some of the tests.
Adding new tests for filenames with white space and/or rpm directives.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
Test almost 2000 URLs for proper version and name detection. This list
is sorted by the second column (expected name).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Rename set_mock to get_mock_cmd, and return the value instead of
setting a global variable.
This patch should not change any behavior.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Use 'sudo' unless /usr/bin/mock points to consolehelper. This is a
simpler heuristic than checking for the group membership and will let
us use the group membership in Clear Linux to give sudo permissions.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Adobe font packages have -nndpi as part of the name. Add tests and a
regular expression to capture these.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Debian is down again, and we've seen them go down in the past. We
shouldn't be running these tests in Travis.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This prevents collisions when two instances are trying to build the same
package in parallel. Unit tests added as well.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
These tests regularly fail in Travis for unknown reasons. Skip them
until the tests can be rewritten.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This allows python3 unittest to properly generate the tests when running
all the tests in the tests directory.
Also adds a fix to test_test.py to reset buildpattern.default_pattern
every test. This was a bug introduced by running all tests at once -
this was not being cleared correctly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
with the imlpementation of the write_out functionality,
some unittests need to be changed to use that fuction.
Signed-off-by: Gabi Beyer <gabrielle.n.beyer@intel.com>
Minor clean up of buildreq module, including the following changes:
- Remove unused import "tarball"
- Add function docstrings
- Convert to list-driven pattern checks instead of a long list of
conditionals.
- Rename variable names that were overriding keywords (file, buffer,
etc.)
- Remove unused setup_py_python3 function
- Remove unused arguments and variables from scan_for_configure function
(and corresponding function calls in test_buildreq.py and autospec.py)
Most changes made at pylint and pycodestyle suggestions.
Updates include generating an individual test for each url to be tested
instead of doing it all in one test and appending to an error list where
appropriate. This has the benefit of showing failures for each
individual test case in a recognizable manner.
Tests are also added for most of the remaining functions.
These tests do not pass without the accompanying tarball module cleanup
and restructure.
Some changes have recently been pushed to autospec sources without
updating the unit tests. This patch brings the unit tests up-to-date
again.
test_build.py: Remove go buildtool bits (was deprecated)
test_specdescription.py: Newline added to post-scripts
test_specfile.py: Reset license_translations dict
files.py: Check dict with get() in case of missing key
Add unit tests for the very large count.py module/script. These unit
tests attempt to test every regular expression present in the parse_log
function and helped catch several bugs in the original regexes ported
from perl.
Clean up the specdescription module, including resolving two bugs and
making the code more readable and more robust.
BUGS:
- description_from_pkginfo
A string was checked for a substring using the string.find() method
but was treated as a boolean, assuming that find() returned 0 when the
substring was not found. This was causing a section end to be
immediately detected and the section was not parsed. The call was
replaced with the `in` keyword.
- description_from_spec
The default description was overwritten regardless of whether the
detected description had any content. A check was added to only
overwrite the default if there was a detected description.
Other general cleanup:
- Remove unused default_group global
- Remove unused argument in summary_from_R
- Add assign_summary/description methods to reduce global usage and
reduce code repetition
- Add function docstrings
- Convert to `with open()` blocks when working with files instead of
explicitly handling closing the files
- Wrap open() calls in try...catch blocks
Unit tests added for the specdescription module. At the time of this
commit, there are two failing tests due to bugs in the module.
Failures:
- test_description_from_pkginfo
This test fails because a string.find() method is checked as a boolean
as if find returned 0 if the substring was not present. find()
actually returns -1 in this case. Because find() was being used as a
guard to find the end of a section, the end was found immediately and
the section was not parsed.
- test_description_from_spec_no_info
This test fails due to the default description being overridden, even
when the override is just an empty string.
Shortcomings were identified in setup.py parsing for python build
requirements. These tests more exhaustively test the cases we have been
seeing in the wild.
This patch makes some simplifications to the buildreq module and related
modules (autospec and config). These are mostly readability changes and
code cleanup.
Matthew Johnson