These patterns are being removed as they are not used or are going to
be reworked.
This change also removes support for multi-versioned packages which
were considered for go and cargo purposes.
Signed-off-by: William Douglas <william.douglas@intel.com>
The generated packageurl name/version tests were only testing one of
four content states, leading to a gap in test coverage, since some names
or versions might be mis-parsed in a state that was not tested.
Fix the issue by testing all content states for every package URL in the
tests/packageurls file. And ensure that test failure output remains
actionable by using a subTest() context manager and setting the `state`
argument, which will print the value of `state` in the test failure
output header.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When not handling a multiple version package, there was a case where if
the version wasn't set the multi_version map would be the
config.versions map and that's not intended to outside of multiple
version use case.
So to fix this, make a new map when setting a default version for the
multi_version map and keep config.versions as its own map.
Signed-off-by: William Douglas <william.douglas@intel.com>
The pattern must be passed to the source class to handle pecl archive
files correctly. It would be good to have the configuration files be
loaded before the sources are processed but that's a later work item
and is unlikely to cause problems in this case.
Fixes#617.
The buildpattern is a bit of an odd standalone module that was a
vestiage of the pre-refactor days where modules owned global
data. These days are coming to a close so move this data to a more
sensible owner in config.
This resolves#612 though likely more will be done as part of #131.
Many different components require the target download path and it is
needed early. Previously code existed to attempt to autodetect its
location and so defining the path was enforced later in the build
process. This created problems with ordering functions based on this
requirement but since the target path is now certain to be defined at
startup (due to previous changes) we can better align the location and
initialization of the data to the Config class.
As part of this change also avoid requiring the temporary working
directory to be setup as part of the Build class. This is done as a
cleanup due to the only real user of this location being the Content
class.
This also drops the stand-alone pkg_integrety main function that was
initially used as a one off test to avoid needing to run autospec to
validate the integrity of a package. This feature was seldom used and
doesn't function well without handling configuration so it was
dropped.
The build module had a number of globals that were referenced by many
other modules and has ordering dependencies with the config and
tarball module for some values. This made deciding on where certain
values get initialized difficult but before the initialization can be
addressed a refactor is helpful.
This change moves the global state (and functions that needed to act
on that global state) into a Build class. The goal of this work is
to better track what can be updated by a particular function, load
data in a sensible order and have it owned by a sensible component.
Some of this data is likely to be removed from the build class in the
future as initialization of download and base paths should take place
in other modules most likely.
The tarball module had a number of globals that were referenced by
many other modules and has ordering dependencies with the config
module for some values. This made deciding on where certain values get
initialized difficult but before the initialization can be addressed a
refactor is helpful.
This change moves the global state (and functions that needed to
act on that global state) into a Content class. The goal of this work
is to better track what can be updated by a particular function, load
data in a sensible order and have it owned by a sensible component.
I expect that future work will see the tarball, buildpattern and some
portions of the config moved around to better match what component
should own what data (and to better deliniate between the data that
has a similar use but comes from different sources).
The config module had a large amount of globals that were being
touched across many modules that would import. This made changes to
config very fragile as figuring out what would be modified in any
given call chain was difficult to diagnose.
It also made testing fragile as one would need to reset a given
module's config import to the best of their knowledge before rerunning
another test.
To get away from that (and to try and reduce the number of globally
modified variablies in autospec as a whole), refactor the config
module to provide its state as part config class. The long running
goal of changes like this is to better track what content can be
updated by a particular function (if a function would have access
to the config instance is now the hint rather than the config module
getting imported).
This patch consolidates the code that manages the sources
(.zip, .tar, etc) in Source class. This class is written in a way
that allows the reuse of code for all current supported types of
sources and at the same time, it allows the easy addition of new
ones.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
Add a special archives list (go_archives), so we can remove Go specific
methods and simply make use of regular methods.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
- Make use of tarfile library
- Update test for this function adding multipl test cases.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
- Make use of zipfile library.
- Update test for this function adding multiple test cases.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
Update support for handling go packages by handling missing go
dependencies. Make use of go modules and the go proxy (see
https://tip.golang.org/cmd/go/#hdr-Module_proxy_protocol for more
details on the proxy implementation) to discover what artifacts are
needed.
A new go-dep build_pattern that will package all versions that the
go-proxy finds is also added as part of this change. Additional rework
of the golang build_pattern (and Makefile build_pattern as a number of
go packages use Makefiles) to use module vendoring when module
configuration is detected was added as well.
The new functionality is also intended to be used by a higher level
tool that will wrap autospec calls to build missing dependencies as
well as the requested package.
The actual output of the unzip -q -l commands used in build_zip()
doesn't include the archive name. Removing it from the mock UNZIP_OUT so
unit tests match reality.
The package metadata is now written to options.conf so the Makefile
writing is no longer needed.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
For github URLs, capture and identify repo name separately from package
name. This is a building block for resolving cases where the package
name is actually different than the github repo name.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
For github URLs in tests/packageurls, check that we produce a giturl
that looks reasonably correct.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
This allows python3 unittest to properly generate the tests when running
all the tests in the tests directory.
Also adds a fix to test_test.py to reset buildpattern.default_pattern
every test. This was a bug introduced by running all tests at once -
this was not being cleared correctly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Updates include generating an individual test for each url to be tested
instead of doing it all in one test and appending to an error list where
appropriate. This has the benefit of showing failures for each
individual test case in a recognizable manner.
Tests are also added for most of the remaining functions.
These tests do not pass without the accompanying tarball module cleanup
and restructure.
Due to an 'nargs' specification in the argument parser, the name
argument was being read as a list. The list was then always accessed as
name_argument[0] to access the name. This patch removes the 'nargs'
specifier so name is always a string and removes the logic to parse the
list.
If both version and name arguments exist, set them and skip automated
detection. If one is not set, continue with the automated detection
since the URL parsing for both are done together, override at the end.
Autospec is unable to accurately parse every type of URL for the correct
name and version of the package. When this is the case, the detected
version can be overwritten by the --version argument on the command
line.
The files module was being treated like a class, in that it was
necessary to keep state throughout the entire autospec run. Convert the
files module to a class and make some other general clean ups. This
changed how the files module was instantiated and called by the main
autospec module and other modules such as tarball and config.
Test the files module with unit tests and add the target to the
makefile.
Version detection could be considerably simplified using builtin python
functions such as strip, replace, and list comprehension. The logic was
also modified a bit to provide more consistent version strings
containing only numbers and periods '.'. More unit tested urls were
added to tests.
9.11.0-P1 -> 9.11.0.1
1.0.0-rc1 -> 1.0.0.1
7_4_2 -> 7.4.2
2.1-3 -> 2.1.3
Some non-digit strings are conserved for upstream version update detection:
2.1-beta1 -> 2.1.beta1
2.1-pre8 -> 2.1.pre8
3.5.0b2 -> 3.5.0b2
Make several small improvements to package name and version detection.
Unit test now passes for a sampling of >100 package URLs. This patch
also adds several package URLs for testing.
Some of the improvements to name/version detection include:
- Reordering of patterns so specific patterns come first - these are our
edge cases but they will not be checked if they are at the end of the
list.
- Add conversion from 7_4_2 formatted versions to the more-common 7.4.2
format.
- Remove release candidate tag (-rc#) from the end of version tags.
- Remove leading 'r' or 'v' even if it is capitalized.
- Remove package name from beginning of version if present.
This unit test reads test cases from tests/packageurls and attempts to
determine the version and name from the url specified. tests/packageurls
defines these cases and their expected results in the format:
<package url>,<expected name>,<expected version>
There are currently some test cases that fail. This unit test will make
it easier to make changes to the tarball.name_and_version() function.
William Douglas