This release introduces safeguards to remove directories from the
package file list to protect against top-level directories being
included in packages (such as /usr). The build.log files are now saved
to the target directory for each failed build attempt. The urlban is now
respected when writing metadata to options.conf.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This also requires moving the read_config_opts() call to after
autospec.conf is read so the urlban is populated.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Use a regular expression in order to exclude all directives at the
beginning of the filename, including %doc.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When cleaning directories from the package file lists do not attempt to
clean "%dir" prefixed files. Although autospec does not currently
support empty directories, it could in the future.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Clean directories from package file lists. If directories are
encountered, print a warning, add the directory to the blacklist, and
re-run.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This is useful for debugging purposes. The file must be saved to the
target directory since the results directory is wiped on each round.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release makes some small updates to our README documentation,
improves tarball.py and pkg_integrity.py testing, fixes a bug that
allowed an empty file to remain on failed downloads, and adds a regular
expression for build.log test scanning.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Create a mock function that copies files from the testfiles
directory to a tmp directory, when attempting to download
files from the network.
Create a mock function for head_request that returns 404 for
a few specified urls, and 200 for the rest.
The mock calls will allow the functionality of the pkg_integrity
program to be tested and not dependent on networking. Tests will
no longer need to be skipped in travis with the removal of
network dependency.
Removed a few tests that were not specific to pkg_integrity, but
instead were testing the importing and exporting abilities
of the gpg key server.
Added mock as a requirement to the requirements.txt file.
Signed-off-by: Gabi Beyer <gabib@live.com>
Fixes#26
Remove empty file created when a download fails. The empty file caused
autospec to fail on consecutive runs due to autospec attempting to reuse
the file when it exists.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
For github URLs, only use the repo name as the package name if it's more
descriptive than what was extracted from the tarball name. Otherwise,
filter off prefix "release-" or a numeric suffix.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
For github URLs, capture and identify repo name separately from package
name. This is a building block for resolving cases where the package
name is actually different than the github repo name.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
Extend the regular expressions used to match github URLs. Covers
additional cases found in the wild.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
For github URLs in tests/packageurls, check that we produce a giturl
that looks reasonably correct.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
This release improves automated commit message guesses using the git
shortlog for the updated package. It also adds a --no-prep option that
downloads the sources, does the basic name and version detection, and
puts metadata in the <targetdir>/workingdir directory for human or tool
consumption before exiting without writing to the spec file.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Also move the prep work into its own function to make it easier to
manage if it continues to grow.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Since moving output to a temporary directory, the "output" directory has
fallen out of use. Remove the output directory from autospec.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
For non-autospec enabled packages this option can be used to download
the upstream tarball, any specified archives, extract them and put the
archives at their destination, and update the upstream file, but not
actually attempt to build a specfile.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The previous ignore, import not at top of file (E402) was unneeded. Bare
excepts are handy when we want to fail gracefully from any error. This
is a script, not an imported library.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
In %files sections, double-quote any filenames containing a space or
tab. Uses a regex to detect and not quote rpm directive prefixes.
Fixes#32.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
Tests for the %files section use bare file names. Since rpmbuild
requires leading slashes anyway, adding slashes to some of the tests.
Adding new tests for filenames with white space and/or rpm directives.
Signed-off-by: Brett T. Warden <brett.t.warden@intel.com>
This release greatly improves version and package name detection and
adds over 1900 package URL tests for this feature. Package metadata is
now written to options.conf under the 'package' section. A Makefile is
still being written to, but this is deprecated and will be removed in
the future. The mock command is now detected in a manner compatible with
linux distributions other than Clear Linux.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Fix up several regular expressions and remove unused ones. Move as much
string replacing to string.replace() functions instead of slow regular
expressions.
Remove the pypi.python.org -> pypi.debian.net url conversion
(pypi.debian.net keeps going down).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Test almost 2000 URLs for proper version and name detection. This list
is sorted by the second column (expected name).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Rename set_mock to get_mock_cmd, and return the value instead of
setting a global variable.
This patch should not change any behavior.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Use 'sudo' unless /usr/bin/mock points to consolehelper. This is a
simpler heuristic than checking for the group membership and will let
us use the group membership in Clear Linux to give sudo permissions.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Instead of requiring the url argument and guessing where to put the
results from an autospec run, read metadata from options.conf. The
'package' section in options.conf defines the package name, url, and
archives when available.
If options.conf is present and there is at least one match between
detected metadata from the source tarball and the metadata in the
configuration, assume that the target directory is the current
directory. This is overridable with the -t/--target option. If nothing
matches the target directory is <current_dir>/<package_name>.
This functionality allows user to run autospec within the target
directory without passing any arguments, including the previously
required "url" argument if the metadata is present.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Deprecate writing to a Makefile in the package directory and write to
the options.conf file instead. Any other infrastructure that needs a
Makefile should write it itself.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Adobe font packages have -nndpi as part of the name. Add tests and a
regular expression to capture these.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Debian is down again, and we've seen them go down in the past. We
shouldn't be running these tests in Travis.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release adds the "-C/--cleanup" flag to clean up the mock chroots
after a build. It also ensures that mock will build unique chroots for
each build, useful for automation environments where autospec is run in
parallel.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Add -C/--cleanup to tell mock to clean up the chroot after it completes
the build. This is useful for wrapping programs which run several
instances of autospec automatically. Use this flag to determine when we
actually need a unique uniqueext (automation environments) and when
those uniquexts can be reused (developer running one-offs).
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This prevents collisions when two instances are trying to build the same
package in parallel. Unit tests added as well.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Allow users to specify requires that are not dropped during existence
detection and os-packages list checking.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Matthew Johnson