Debuginfo packages now install binary debuginfo to /usr/share/debug and
source debuginfo to /usr/share/debug/src.
The purpose for the migration is for these new locations to serve as
"lookaside" directories used by future versions of clr-debug-info,
enabling select debuginfo packages to be installed in bundles and safely
coexist with the FUSE mounts.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
- Run git commands within the newly created package repo
- Only run these commands if autospec succeeds
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Some packages in the distro do not generate debuginfo subpackages, so
make sure to skip the debuginfo install in this situation.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The caches are not package-specific, and --scrub=all will clean all
caches even with the --uniqueext option. For that, we have 'make
mockclean' instead.
Using both --clean and --scrub=chroot is also unnecessary, as --clean
already removes the chroot.
Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Currently, we use the traditional BIOS flag "-bios" to specify the OVMF
firmware device file. Ken proposes changing to pflash type BIOS and separate
BIOS code and variable.
-drive file=OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \
-drive file=OVMF_VARS.fd,if=pflash,format=raw,unit=1 \
Ken has following reasons:
1. -bios can only support single flash binary, but the actual BIOS binary
is consisted by many parts like SEC, PEI(Boot Block), DXE(Main Block)
and Variable area as non-volatile storage for BIOS settings. During
OMVF booting, a file called NvVars.bin will be created at
<Disk>/<ESP Partition>/NvVars.bin for BIOS variable storage.
This will pollute original Clearlinux-xxx.img.
2. In future, we can add secure signed SEC BIOS part to enable full
secure boot for Clear Linux via pflash.
3. Some version of open stack, it requires separate Var Store NV instead
of OVMF.fd. Ken meet with this issue recently. So if can show the separate
pflash BIOS file via start_qemu.sh as example.
Suggested-by: "Lu, Ken" <ken.lu@intel.com>
As from 0c69bfb9a9 ("Add OVMF.fd to the repo"), the CODE and VARS will
be used by start_qemu.sh.
Note that these version of OVMF_* are a copy from /usr/share/qemu
in Clear Linux OS 25840.
$ sha512sum /usr/share/qemu/OVMF_*
efea56944ad8153ed956390863c30c1a61eb4e52c43a8a0e3e6e3a71e56796f3c6a1c32e107a3d86e7a2e59faa462c7ffb10916b62a73d110bd0b0b84a60e02a /usr/share/qemu/OVMF_CODE.fd
448412fd7ba267b4180db8ade6edb67af467e5b9b3e3ff8dfd583a2fded4070f6951667297e6896ce8bd9f4d2ec3dd8a5a70b6e9a2f686efec9a57124fec512a /usr/share/qemu/OVMF_VARS.fd
The repos are in an unusual state for 25600 (just like for 25590) in
that two swupd-client packages were added. The tooling for updating this
packages file does not expect that condition, so revert the change.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For testing purposes only, if developers want to install RPMs they just
built onto their development system, potentially overwriting content
that is tracked by swupd, they can now run 'make install-local' to
install all of the RPMs, or 'make install-debuginfo-local' to just
install the debuginfo package.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The user may have run the user-setup script previously, or they may
already belong to the 'kvm' group to begin with, so avoid running the
usermod command in these situations.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For some commands, it is useful to be able to add custom prerequisites.
We already support this for `make koji` and `make build`, and there was
just a request to add it for `make autospec` and `make autospecnew`.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Make the SPDX puller code more robust by reading from JSON sources
rather than parsing HTML. Also, keep all deprecated license identifiers
for now, since they are still present in the spdx list.
Four identifiers were removed since the last refresh, but we are keeping
them for backwards compatibility. The removed license IDs now reside in
the licenses-extra file. At
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Several commands should be using PKGS_LOCAL, which is the list of all
package repos that are present on the system, since the user might not
have all repos from PKGS cloned.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This fixes integration with the GNU Make jobserver such that the
-j/--jobs value passed to make is actually honored.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Document includes a description for pumping the release, but what
if there is a completely new upstream version? This patch adds a
section that describes how to update the upstream url for a
package and update the package information accordingly.
Signed-off-by: Kevron Rees <kevron.m.rees@intel.com>
Documentation only describes how to tweak autospec behavior by
modifying one file ("buildreq_add"). A new developer might not be
aware that there are several other files that can be modified that
will change autospec behavior. Link to the autospec documentation
so that developers can learn of the other files and how autospec
uses them.
Signed-off-by: Kevron Rees <kevron.m.rees@intel.com>
In case git repos cloned by autospec into results/ contain RPMs, we do
not want to hardlink those files to rpms/.
Fix the issue by limiting the find depth to 1 level so that the tooling
only hardlinks rpms created by Mock.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Previously, we always emitted an error and failed when the specfile or
upstream target was requested, breaking the -B switch. Now, we only fail
if the file doesn't actually exist, not just that it was requested via
make.
Since binary RPMs are now stored in the "rpms" directory, follow a
similar method for storing them in the local repo, too.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To improve visibility of which RPMs a user may want to install after a
build, copy (hardlink) them to a separate "rpms" directory.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
I often find myself running "make clean proper", when by default, I
think the old build artifacts should be removed so that the results
directory is pristine at the start of a new build.
As a result, if users wish to keep all build artifacts from a previous
"make build" or "make autospec", they must keep manual backups, as the
tooling no longer keeps them around.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Mock will append the binary rpm build logs to the srpm logs, but this is
confusing. After the srpm finishes building, rename the log files to
avoid this behavior.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Because Mock dumps its logs and and resulting rpms in the same directory
(results), shuffling the logs and rpms only after a successful build is
confusing, since they will be untouched for the error case.
It's simpler to avoid moving the files entirely. The only exceptions
will be to avoid clobbering existing log files. Changes of this nature
will appear in later commits and in autospec.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The failed_pattern_dir contains a path to a location containing
additonal configuration files for failed_commands, ignored_commands,
maven_jars, gems, license_hashes, license_translations,
license_blacklist, and qt_modules.
In Mock version 1.4.4, the chroot group was changed from "mockbuild" to
"mock". This creates a problem for running Mock on Clear Linux because
(a) the distro already has group "mock", and (b) Mock tries to `groupdel
mock` within the chroot and fails, since the group is defined in
/usr/share/defaults/etc/group and not in /etc/group. Mock treats the
groupdel failure as fatal. Fortunately there is a workaround: set the
"chrootgroup" config to a group not already defined in the OS. I am
setting the group to "mockbuild" in the default config to restore the
behavior of previous Mock versions. Thanks to Mark Horn for adding this
feature upstream! It was merged for Mock 1.4.7.
Also, disable the new bootstrap container feature, added in Mock version
1.4.1. It was disabled by default in version 1.4.2, but keep the
explicit setting in case upstream enables it in the future.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Fixes#6
Not all Clear Linux packages have debuginfo subpackages, so ignore
errors when attempting to move them to the results/debuginfo/ directory.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When running 'make koji', the tooling will attempt to create a new git
tag with a name constructed from the spec file's NVR.
However, if there are uncommitted changes to the Name, Version, or
Release fields (collectively, NVR) in that file, the new tag's name will
not match the NVR in the spec file referenced by commit HEAD.
Protect against the mismatch by exiting early from a 'make koji' if any
spec file modifications are not yet committed to the repo, and alert the
user what the problem is.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The debuginfo RPMs are stored in a separate directory now, so the rpm
copy pipeline needed a slight adjustment.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The logs, srpms, and debuginfo should all be split into their own
directory, so it is less confusing about which RPMs can be used when
creating mixes. They are mainly for extra metadata, and should not be
left in the main RPM results location.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fixes#4
Because start_qemu.sh references OVMF.fd, it is convenient for OVMF.fd
to live alongside that script for ease of use.
Note that this version of OVMF.fd is a copy of /usr/share/qemu/OVMF.fd
in Clear Linux OS builds 22930 and newer from the edk2 package
(edk2-2-8, built using the recent UDK2018 release).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Fixes#3
The other shell scripts in this repo are executable, so there's no
reason this one should not be.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
DNF doesn't fall back to /var/tmp if it cannot write to the cachedir
location specified in dnf.conf (unlike YUM). Removing the explicit
cachedir setting in dnf.conf enables the fallback behavior, which we
will need for autospec use.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since the user-setup script does not clone package repos by default
anymore, 'make pull' can be made more user-friendly by not printing
warnings if some/all/most of the package repos are not already present
in the packages/ directory.
Instead, only populate the package "pull" list with names from the
packages file that are currently present in the packages/ directory.
Package repos that are not present can be cloned as before: 'make clone'
to clone all missing package repos, or 'make clone_PKG' to clone a
selected package repo.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Not setting "keepcache=1" results to DNF removing cached RPMs after
every local build.
Also, naming the cachedir "/var/cache/dnf" is important for the cache to
stick around and be stored in /var/cache/mock/clear/dnf_cache.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The new dnf.conf is very similar to the yum.conf, but because the
in-tree Mock config specifies to use DNF, we should track a dnf.conf
here as well and use it by default.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The "image-creator" and "koji-client-files" subdirectories are named as
such to reflect their purpose from long ago. In the present, the names
are confusing, so I am shuffling files around a bit to reflect current
usage.
All conf files (or conf file templates) have been moved to "conf"
instead, and some documentation has been added to explain what the files
are used for, and how they can be customized.
Also, after moving the conf files, the last file remaining in one of the
old subdirectories is "start_qemu.sh", so move it to the toplevel dir.
We could eventually move the miscellaneous scripts into a dedicated
directory too, but I will save that for another time.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The common tooling behavior changed a while back with respect to the
mock config usage, so update the user-setup script to reflect the
changes. Also add documentation about how to customize the mock config,
which can be leveraged to revert back to the original behavior as well.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To enable future shared workflows between the common tooling and mixer,
create a directory named 'mix' within the tooling workspace.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Support both absolute and relative paths for the cert options, instead
of relative paths only, by using realpath(1) to resolve the arguments.
(Note that the script cannot reference the path arguments verbatim
because much of the script operates in the workspace's toplevel
directory, which is a different location than where the script is
invoked. In that situation, relative paths would not resolve.)
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
After running 'make autospecnew', the freshly created repo only exists locally,
so the repo remote is not yet initialized. The 'git pull --rebase' (via 'make
koji') will then fail due to the uninitialized remote. Improve this case by
only rebasing when the remote is initialized.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If the user tries to create a new package with 'make autospecnew', a
package repo will be created from scratch, even if a repo exists with
the same name at PKG_BASE_URL.
This problem stems from 'make autospecnew' assuming that all tracked
package repos are cloned already, but is a bad assumption, given the
recent change of behavior for user-setup.sh (not cloning package repos
by default).
Resolve it by first trying to clone the repo with "make clone_PKGNAME";
clone_PKGNAME targets are automatically defined for all packages listed
in the projects/common/packages file. As long as users have an
up-to-date packages file (i.e. are running 'make pull' regularly), then
this solution works well. The case not covered is when packages are
added, then later removed; package repos will already exist remotely in
this case. However, this is a relatively rare case, so I opted to only
handle the common case where a package is added and never removed from
the distro at a later point.
(Side note: I originally wanted to 'git ls-remote REPO' to check remote
repo existence, but this does not work well with Github, as Github
prompts for username/password for remotes that appear to not exist,
since they might be private repos. Prompting a Github username/password
in this case is not very user-friendly. Using the Github API looks
viable, but I would rather not special case the handling of Github repos
in the common tooling.)
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The DOWNLOAD_MIRROR server, the SOURCES_URL server, and upstream
source tarballs may be hosted in many different environments.
Allow users to supply additional curl options via a config to use
whenever connecting to the DOWNLOAD_MIRROR and SOURCES_URL servers and
work better with their particular hosting environment. Because upstream
source tarballs can come from many different locations, do not offer a
global config and instead allow users to define this manually as needed
when the rule is used.
Signed-off-by: George T Kramer <george.t.kramer@intel.com>
Eventually developers - when they know what they're doing - want to
manipulate a little bit the mock arguments to i.e skip a few stages
in the mock build cycle while they're hacking a package.
One example is when developers are testing changes to a spec file but
not touching dependencies and they're [re]running the build multiple times
it would be nice to not cleanup before the build and run a rebuild
for example with: --rebuild --no-clean
With this patch one should only do:
make build MOCK_OPTS="--rebuild --no-clean"
Signed-off-by: Leandro Dorileo <leandro.maciel.dorileo@intel.com>
Users are encouraged to run the user-setup script as documented in the
"Automated setup" section. To clarify that the "Manual setup" section is
an alternative to "Automated setup", move those docs to the end of the
README instead.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Because the package repo cloning process can take a lot of time, do not
clone them by default. Instead, add a new --clone-packages|-a option to
perform this task.
Also, immediately before the script exits, document some possible next
steps when *not* cloning package repos (the new default).
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To allow setting up multiple workspaces in the same directory, or if the
name "clearlinux" is not the desired name, add a --directory|-d option
to let the user configure it.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since Clear Linux has many package repos, cloning them all in serial can
take a long time. Add a -j option to the script to let users speed up
the process.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
My previous change to handling of the 'upstream' file assumed that the
file exists already, but this need not be true.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When SOURCES_URL is not defined, the current behavior of extracting
source files from SRPMs in the latest published Clear Linux release has
proven to not work so well: It is not optimal when users are adding new
packages, modifying existing packages, not building from latest master,
etc.
Instead, run 'generateupstream' automatically so that source files are
downloaded from the remote locations listed in the spec file. This
target creates the 'upstream' file too, which becomes useful if the user
later defines SOURCES_URL, when 'upstream' is actively consumed.
In the event that download errors occur, as a fallback mechanism, try to
find the corresponding SRPM from a published release of Clear Linux OS.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
I will reusing this code (largely unmodified) in the next commit, so
move it into a separate target to easily inspect changes.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If any of the curl commands fail, it's better for the entire target to
fail. In this way, developers will be properly notified.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The curl --no-proxy hint is not needed anymore, since Clear Linux OS has
an autoproxy, and curl will honor proxy environment variables.
Also, pass -L to curl to follow redirects, similarly to other curl calls
in this makefile.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Improve the detection of source URLs by matching only non-commented
Source* lines with "://" present.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The 'git branch' command fails at this stage because the master branch
is not initialized locally, but because the remote is also not
initialized, additional special incantations are needed to make this
work. Revert it for now so that users do not see an error message from
the 'git branch' command. A proper fix will follow later.
This reverts commit 71d9699bdd.
Because 'make koji' expects that master should track origin/master,
initialize the tracking branch during the autospecnew.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If you run `newgrp kvm` after the user-setup script completes, and then
proceed to run `make build`, `make autospec`, etc. within a package
repo, mock will fail because it tries to `groupadd mockbuild` in the
package chroot with the wrong group ID. Logging out and logging back in
results in correct behavior, so leave that as the sole recommendation.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
In my previous commit, one of the changed lines used a combination of
tabs and spaces, but it should have used tabs only.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Over time, the autospec options used for 'autospec' and 'autospecnew'
commands have diverged. They are meant to be in sync, so do that here,
and format the recipe in a way that eases future changes.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The echo commands at key points of the pullrebase should be sufficient
for users to know what action pullrebase performed.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If a new package repo is created with make autospecnew, and then make
autospec is run immediately afterward, a git fetch is performed, but
the remote repo is not likely to exist yet.
Address this issue by only running the git fetch if the remote origin is
initialized locally via a git push, git pull, etc.
In addition to properly supporting the workflow described above, this
change also enables developers to build packages that are *not* intended
to be included in Clear Linux OS, thus not destined to appear in the
'packages' file. Package repos of this nature might have different,
manually configured remotes, or the remote may never be initialized at
all if builds are only run locally.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
if ! grep -q "^$$LICENSE$$"$(TOPLVL)/projects/common/licenses-spdx&& ! grep -q "$$LICENSE$$"$(TOPLVL)/projects/common/licenses-exceptions;thenFAIL=yes;echo"ERROR: License '$$LICENSE' is not an allowed SPDX license ID";fi;\
if ! grep -qx"$$LICENSE"$(TOPLVL)/projects/common/licenses-{spdx,exceptions,extra};thenFAIL=yes;echo"ERROR: License '$$LICENSE' is not an allowed SPDX license ID";fi;\
done;test -z "$$FAIL"
#help cvecheck: Checks
@@ -60,7 +62,7 @@ clean:
#help proper: Brings your copy of the package to a nearly clean git checkout state.
echo"Error with koji installation or configuration" >&2
exit1
fi
echo"Testing koji installation . . ."
if koji moshimoshi;then
echo -en "\n************************\n\n"
echo"Koji installed and configured successfully"
else
echo -en "\n************************\n\n"
echo"Error with koji installation or configuration" >&2
exit1
fi
fi
echo -en "\n************************\n"
echo'Workspace has been set up in the "clearlinux" directory'
echo'NOTE: logout and log back in or run "newgrp kvm" to finalize the setup process'
echo"Workspace has been set up in \"$WORKSPACE\""
if[ -z "$PACKAGE_REPOS"];then
echo"NOTE: To clone all package repos, run \"cd $WORKSPACE; make [-j NUM] clone-packages\""
echo"NOTE: To clone a single package repo with NAME, run \"cd $WORKSPACE; make clone_NAME\""
fi
if[ -n "$NEEDS_KVM_GROUP"];then
echo'NOTE: logout and log back in to finalize the setup process'
fi
# vi: ft=sh sw=2 et sts=2
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
clrbuilder