Compare commits

..
101 Commits
41 .. 50
Author SHA1 Message Date
Patrick McCarty bca3bdefda Release v50
- Convert clr_debug_prepare to a bash script, also incorporating a few
  improvements and bug fixes into the rewrite.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2019-06-05 15:34:59 -07:00
Patrick McCarty 179392c601 Convert clr_debug_prepare to a bash script
For easier parallelization, I opted to replace the C program with a bash
script that uses GNU Parallel, find, and GAWK for the core control flow.

For compatibility, running the shell script without arguments has the
same general behavior as the C program. Some bugs have been fixed as
well, so the end result should be an improvement over the original.

Bugs fixed:

- A race condition in the C program with the "unsymlink" operation has
  been addressed: instead of replacing the valid symlinks within
  debuginfo.raw, the symlink targets are added directly to the automatic
  tarballs, and tar's "transform" feature is used to rewrite the filename.

- Running system("tar ... &") to create the tarballs often had the side
  effect of bogging down the system, since the number of active tar
  processes had no upper bound. The shell script rewrite uses GNU
  Parallel, which limits (by default) the number of parallel jobs.

- Directory tarballs were unintentionally re-created whenever content
  within the directories were modified, due to the stat() mtime changing.
  With the new script, directory tarball creation only occurs for those
  directories that do not already exist at the destination.

Once the script begins to create tarballs, performance is largely the
same as the C program. The script has a "precompute" phase where it
determines the list of needed tarball creations up-front, which is
different (and probably a bit slower) than the on-the-fly stat()s used
by the C program, but the runtime cost of this new precompute phase is
negligible compared to the tarball creation phase.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2019-06-05 15:28:20 -07:00
Patrick McCarty b802948209 Release v49
- List tarball contents prior to extraction to improve robustness
- Plug some memory leaks
- Print errors to stderr instead of stdout

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2019-05-24 13:04:07 -07:00
Auke Kok 4495b923a9 Redo tar extraction - try and make this safe(r).
Instead of extracting to a tmp folder, just tar `tf` it first. This
keeps extracting to the proper locations trivial, at the cost of
decompression the content twice.
2019-05-10 11:08:06 -07:00
Auke Kok 8f7289f028 Test result of tar extraction before trusting the result.
We extract the tarball to a temporary file, before actually allowing
gdb to use this file. This allows us to make sure that gdb doesn't
see the file if it is corrupt. Only if tar exceeds, we rename() the
temp result into the actual needed file. If tar fails, we throw away
the file.
2019-05-07 16:19:36 -07:00
Auke Kok dc376d5e8b Update to v48 2019-01-15 12:47:32 -08:00
Auke Kok c831ecb902 Use select() to time out, and exit.
With socket activation, we can now select() and time out before accept()
and exit, releasing all resource we used back to the system.

The default timeout is set for 10 minutes. If no more connections
arrive, the daemon exits.

Socket activation is done through the clr_debug_daemon.socket unit, which
the clr_debug_fuse service requires, so this should be extra robust when
starting up.

The daemon ignores stat() requests for trash that gnome sends when it
sees the fuse system mounted, and returns the underlying stat() for
the root node instead of a remote value, because there is no value
in fetching this info over curl.
2019-01-09 12:58:42 -08:00
Auke Kok 0978d429b2 Clean up some of the socket activation stuff. 2019-01-08 12:37:56 -08:00
Patrick McCarty 6653992aba Release v47
Fixes a bug with clr_debug_prepare's handling of broken symlinks.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2018-12-10 17:10:44 -08:00
Patrick McCarty 0af0308dd2 Revert "have tar follow symlinks"
After closer inspection, I've determined that prior to this commit,
clr_debug_prepare was behaving properly with respect to the handling of
symlinks in debuginfo packages. Reverting this commit fixes the issue.

Details:

 - Symlinks under /usr/lib/debug/.build-id that are not broken should be
 unsymlinked so that the actual debuginfo is added to the automatic
 debuginfo tarballs. In the debuginfo packages, these files have a
 .debug extension and point to files under /usr/lib/debug/usr.

 - Symlinks under /usr/lib/debug/.build-id that *are* broken cannot be
 unsymlinked, and therefore, tar should not follow these symlinks,
 because fatal errors (and empty tarballs) are the result. These
 symlinks lack the .debug extension and point to the corresponding
 libraries or binaries under /usr when exposed via clr_debug_fuse. So it
 is correct for the symlinks to be broken when packaged by
 clr_debug_prepare, since they point to non-debuginfo files, and the
 debuginfo.raw directory tree only contains debuginfo content.

This reverts commit 7dcddfd0ca8f5539ef34028a1c680f6e82dcf246.
2018-12-10 17:04:05 -08:00
Patrick McCarty 8ace503438 Release v46
This release fixes a build issue when not passing -Wl,--copy-dt-needed-entries
to the compiler.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2018-05-25 21:56:43 -07:00
Patrick McCarty 2c87f83adc build: add LIBSYSTEMD_LIBS to link line
For the regular Clear Linux build, the -lsystemd option was not needed at link
time due to -Wl,--copy-dt-needed-entries being used. Without that option
though, the linking of clr_debug_daemon will fail.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2018-05-25 21:55:28 -07:00
Patrick McCarty 67b9b3acc8 Update gitignore
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2018-05-25 21:38:06 -07:00
Arjan van de Ven d108c0af64 add cdn-alt 2018-05-17 13:54:16 +00:00
Auke Kok 98e802b4cc Also install socket unit. 2018-04-24 13:04:33 -07:00
Auke Kok 612801b0c8 Make -daemon socket activated. 2018-04-23 15:15:33 -07:00
Arjan van de Ven 3c2ad26baa have tar follow symlinks 2017-12-23 00:49:24 +00:00
Arjan van de Ven d20d585e5c much shorter timeouts 2017-12-10 14:25:07 +00:00
Icarus Sparry 1f3a0f0be4 Extract tar file as dbginfo
Make /var/cache/debuginfo and its contents owned by dgbinfo. Create
the directories as this user, and switch to run as this user.

Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
2017-10-31 13:35:29 -07:00
Icarus Sparry d2d67bdb1d Change tmpfiles.d configuration
Use the dbginfo user for the directories.

Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
2017-10-31 09:47:39 -07:00
Auke Kok 60db7bab0f v43 2017-06-06 13:28:11 -07:00
Auke Kok ef6b11498a DefaultDependencies keyword is only valid in [Unit] context. 2017-06-06 13:27:20 -07:00
Auke Kok efeed3e83f v42. 2017-06-01 16:22:00 -07:00
Auke Kok f5542e7fbb Fetch timestamp from debuginfo server.
We trigger redownloads from the server if the timestamp of
downloaded tar files is too new. An easy way around this issue
is to request the timestamp from the server instead of using
the download time. A `touch` of the tar then assures that
the downloaded file has the same timestamp as the server has.
2017-06-01 16:19:15 -07:00
Arjan van de Ven be8a99da53 allow service to start early 2017-05-07 15:42:30 +00:00
Ikey Doherty c9304402e2 Bump v38 to resync configure + tags
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2017-03-09 17:48:59 +00:00
Ikey Doherty 6e202f13f7 Bump v34
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2017-03-09 17:46:10 +00:00
Ikey Doherty 6735f8f6ed nica/files: Ensure we really do break on a read error
Previously nc_copy_file would return true regardless of a source read
error, flagged in analysis. Ensure we bypass the set of ret to true and
return the correct value in all instances.

Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2017-03-09 15:44:09 +00:00
Ikey Doherty 8b3777ab99 server: Remove useless assignment of prefix
This particular assignment is never used, as if this path fails, we go to
the thread end. We then reassign prefix after we split the input string.

Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2017-03-09 15:38:29 +00:00
Ikey Doherty 84350939ef nica/hashmap: Fix signature issue & item dereference
The signature was incorrect for inserting buckets, as we used an int, not
a boolean. Another issue resolved with this change is the potential
dereferencing of a null pointer by not having checked first if item was
NULL when setting the next pointer.

Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2017-03-09 15:35:29 +00:00
Arjan van de Ven 72b1120a40 add --no-same-permissions as well 2017-03-09 14:52:37 +00:00
Auke Kok d3d5fab301 Tag v33 2016-11-01 09:20:19 -07:00
Auke Kok e377433a9d Return on error here too.
Missed a 'return' catching an error.
2016-11-01 09:15:12 -07:00
Auke Kok afa92c77b1 Add reasonable connection timeout limits.
Adds a 30second connection timeout, and a low bandwith timeout
value (at 1kb/sec over 30 seconds) where the attempt will fail.
This provides some feedback to users that network issues are
preventing debug info from being loaded.
2016-11-01 09:15:12 -07:00
Arjan van de Ven a01ce4c329 dumb down due to limited infra 2016-11-01 13:17:49 +00:00
Arjan van de Ven 1eaaac582c use http/2 2016-11-01 12:56:35 +00:00
Arjan van de Ven 935dbee3c7 use CDN urls 2016-11-01 12:53:04 +00:00
Arjan van de Ven 0065ef2670 don't spew the journal 2016-11-01 12:52:17 +00:00
Ikey Doherty b5b4030f7c Release v32
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-18 19:21:20 +01:00
Ikey Doherty de9e0ce5c5 server: Disallow gdb -p attach, drop CAP_SYS_ADMIN
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-18 15:03:28 +01:00
Ikey Doherty 9e400bd149 Use correct types in absence of stdatomic.h
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-18 14:37:07 +01:00
Ikey Doherty f5a65358af Use pthread mutex in the absence of stdatomic C11 atomics
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-18 14:33:26 +01:00
Ikey Doherty c82a007960 Add configure output, and check for stdatomics
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-18 14:18:56 +01:00
Ikey Doherty 02ece110b5 Release v31
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:29:35 +01:00
Ikey Doherty 3a6bf4a167 Fix print statements when a dir cannot be created
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:20:31 +01:00
Ikey Doherty fc61591870 Fix distcheck
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:16:53 +01:00
Ikey Doherty 2f26fc55c7 Stick myself in authors
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:14:47 +01:00
Ikey Doherty dfff530803 Address unused return issues
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:13:41 +01:00
Ikey Doherty 9685fcb632 Fix some warnings
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:07:57 +01:00
Ikey Doherty 14495e064b Make use of autofree to simplify exit conditions
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 17:03:54 +01:00
Ikey Doherty 7b64fb601d server: Forcibly inline functions
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 16:41:17 +01:00
Ikey Doherty b24d93fe3d server: Atomically manage a maximum connection ceiling
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 16:32:01 +01:00
Ikey Doherty 5b7c484b64 Fix up some warnings
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-17 15:17:05 +01:00
Ikey Doherty d23b8e17bd fuse: Greatly simplify allocation and management of strings
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 17:08:00 +01:00
Ikey Doherty b120d1c38f Fix last of the mkdir users
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:54:09 +01:00
Ikey Doherty bbc34ad790 Get rid of mkdir calls
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:50:27 +01:00
Ikey Doherty caf84d9b5f Use consistent file headers
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:43:09 +01:00
Ikey Doherty bc276b4920 Incorporate the nica files component
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:42:41 +01:00
Ikey Doherty 78485dc583 Take care of unused variables
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:37:52 +01:00
Ikey Doherty ce02c64c0c Drop glib entirely
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:34:21 +01:00
Ikey Doherty f272c08d4f Use a hashmap to maintain memory
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:30:17 +01:00
Ikey Doherty 3babbe4b7b Incorporate portions of libnica
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 16:18:38 +01:00
Ikey Doherty aad87bbdc1 Use curly braces everywhere
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:48:34 +01:00
Ikey Doherty 4505d32c68 Fix last commit
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:35:01 +01:00
Ikey Doherty 67a9735acb Swap GMutex for pthread_mutex_t
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:33:37 +01:00
Ikey Doherty 1c043d5bd7 Add strict flags
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:22:21 +01:00
Ikey Doherty 8c01f4012f Apply clang-format to codebase (switching to spaces for maintainence)
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:20:29 +01:00
Ikey Doherty 955c1b7e99 Incorporate clang-format helpers (3.8)
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:20:12 +01:00
Ikey Doherty a92e6f386a Add an autogen script
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2016-05-11 15:17:46 +01:00
Arjan van de Ven 670069d659 version 30 2016-02-01 14:34:27 -05:00
Patrick McCarty be74f893e6 Bump version for release 2015-10-20 10:24:49 -07:00
Patrick McCarty 7a5e4a9acd Do not log for non-fatal curl errors 2015-10-20 10:17:40 -07:00
Patrick McCarty 07a4c3d305 Bump version for release
This release fixes the handling of a server HTTP 304 response, now
treating it as non-fatal.
2015-10-02 12:15:07 -07:00
Patrick McCarty ab0e59628f Treat HTTP 304 codes as non-fatal
We set the If-Modified-Since header field for each GET request, and if
the condition fails, the server will respond with HTTP 304.

The 304 means the cached debuginfo is up-to-date, not requiring a fresh
download. Since this condition is non-fatal, avoid swapping the download
URLs in this case.
2015-10-02 12:15:02 -07:00
Patrick McCarty af4a3f64f0 Update gitignore 2015-10-02 10:36:39 -07:00
Patrick McCarty 8c65e96479 Bump version for release
This release fixes some compiler warnings and the distcheck target, and
addresses a recurrent 404 issue by avoiding a download attempt for
".tar" from the root debuginfo directory.
2015-09-29 12:32:23 -07:00
Patrick McCarty 7ea9a50b19 Avoid attempt to download {lib,src}/debug
The associated cache directories in /var/cache/debuginfo/{lib,src}
already exist, and the tar for / is not created on the server, so avoid
downloading it altogether.
2015-09-29 12:25:13 -07:00
Patrick McCarty e16ab76926 Refresh version in configure.ac
The configure.ac version was falling behind the tagged version...
2015-09-28 14:02:03 -07:00
Patrick McCarty 090cb27a34 Fix distcheck target
Adding two --with-* configure options for the systemd pkgconfig-defined
variables gives opportunity for distcheck to succeed; the variables use
hardcoded paths to /usr/lib/systemd/... without a PREFIX, so the paths
need to be relocated to $dc_install_base, a variable distcheck uses for
the normal variables that begin with PREFIX.
2015-09-28 13:52:24 -07:00
Patrick McCarty cbaf943fca Convert server to use modern Glib Thread API
Using a GStaticMutex and associated functions are deprecated, since
a GMutex can be statically allocated now (as of glib 2.32).
2015-09-25 14:20:22 -07:00
Patrick McCarty 32ae400e7d Update gitignore 2015-09-25 14:19:56 -07:00
Patrick McCarty 82f03f8e22 Fix build warning for implicitly included header 2015-09-25 14:00:33 -07:00
Arjan van de Ven c1f45d9f3c https 2015-07-20 23:02:09 -04:00
Arjan van de Ven bcff8fae39 default to the other server for now 2015-04-24 17:55:53 -04:00
Arjan van de Ven 66bab25bcd implement multi-url setup 2015-03-23 14:45:00 -04:00
Arjan van de Ven 49ac392f01 move logging up 2015-03-23 14:37:26 -04:00
Arjan van de Ven 71f4e789c4 better logging 2015-03-23 14:17:30 -04:00
Arjan van de Ven f776162882 few more checks 2015-03-23 14:15:42 -04:00
Arjan van de Ven a2abd8d2a9 put memory optimizations back 2015-03-23 14:12:31 -04:00
Arjan van de Ven 8844b0b0e1 Revert "Wait with curl init until the first connection comes in"
This reverts commit 6fb3c6f193.
2015-03-05 18:00:06 -05:00
Arjan van de Ven ca63e1be25 Wait with curl init until the first connection comes in
this should save a bunch of memory for the common case
2015-02-16 18:12:45 -05:00
Dimitri John Ledkov 078e9987d5 clr-debug-info: Use well known install paths.
systemd pkgconfig module is already required, which will have system
paths defined as pkg-config variables. This simplifies autofoo, and
installs tmpfile.d into a system location under /usr, rather than
admin location under /etc.
2015-02-02 10:30:49 +00:00
Arjan van de Ven 97431314b5 use the non-staging URL 2015-01-31 13:48:44 -05:00
Arjan van de Ven 07e1f033e1 version 11 2014-12-26 11:28:04 -05:00
Arjan van de Ven 45f22398ca don't "system" unless it's really really needed 2014-12-26 11:26:36 -05:00
Arjan van de Ven b538b457ce one more typo 2014-12-26 10:48:24 -05:00
Arjan van de Ven 09affa9bf5 make systemd not complain 2014-12-26 10:32:42 -05:00
Arjan van de Ven 2a217476f7 move the cache dir to /var/cache
for proper stateless operation
2014-12-26 10:32:05 -05:00
Arjan van de Ven 4d6639148f release 8 2014-12-25 19:29:02 -05:00
Arjan van de Ven 83509e34c7 XFS does nto support DT_DIR 2014-12-26 00:15:36 -05:00
Arjan van de Ven 43d54d03e6 Initial Import from Fenrus Linux 2014-12-25 23:53:41 -05:00
12 changed files with 346 additions and 307 deletions
+1 -1
View File
@@ -2,12 +2,12 @@
*.o
clr_debug_fuse
clr_debug_daemon
clr_debug_prepare
.deps/
.dirstamp
Makefile
Makefile.in
aclocal.m4
ar-lib
autom4te.cache/
compile
config.h
+8 -8
View File
@@ -1,4 +1,4 @@
EXTRA_DIST = COPYING clr_debug_fuse.service clr_debug_daemon.service debuginfo.conf
EXTRA_DIST = COPYING clr_debug_fuse.service clr_debug_daemon.service clr_debug_daemon.socket debuginfo.conf
DISTCHECK_CONFIGURE_FLAGS = \
--with-systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir) \
@@ -14,7 +14,9 @@ AM_CFLAGS = \
AM_CPPFLAGS = $(AM_CFLAGS) ${curl_CFLAGS} ${fuse_CFLAGS}
bin_PROGRAMS = clr_debug_fuse clr_debug_daemon clr_debug_prepare
bin_PROGRAMS = clr_debug_fuse clr_debug_daemon
dist_bin_SCRIPTS = scripts/clr_debug_prepare
noinst_LTLIBRARIES = \
libnica.la
@@ -33,15 +35,13 @@ clr_debug_fuse_SOURCES = src/fuse.c src/client.c
clr_debug_daemon_SOURCES = src/server.c
clr_debug_daemon_CFLAGS = \
-pthread \
$(AM_CFLAGS)
clr_debug_prepare_SOURCES = src/prepare.c
$(AM_CFLAGS) \
$(LIBSYSTEMD_CFLAGS)
clr_debug_fuse_LDADD = ${fuse_LIBS} libnica.la
clr_debug_daemon_LDADD = ${curl_LIBS} libnica.la
clr_debug_prepare_LDADD = libnica.la
clr_debug_daemon_LDADD = ${curl_LIBS} libnica.la ${LIBSYSTEMD_LIBS}
systemdsystemunit_DATA = clr_debug_fuse.service clr_debug_daemon.service
systemdsystemunit_DATA = clr_debug_fuse.service clr_debug_daemon.service clr_debug_daemon.socket
tmpfiles_DATA = debuginfo.conf
+1 -2
View File
@@ -1,11 +1,10 @@
[Unit]
Description=Clear Linux debuginfo daemon
DefaultDependencies=no
[Service]
Type=simple
ExecStart=/usr/bin/clr_debug_daemon
DefaultDependencies=no
Nice=10
[Install]
WantedBy=multi-user.target
+9
View File
@@ -0,0 +1,9 @@
[Unit]
Description=Clear Linux OS debuginfo daemon
[Socket]
ListenStream=/run/clr-debug-info
SocketMode=0600
[Install]
WantedBy=sockets.target
+3 -3
View File
@@ -1,12 +1,12 @@
[Unit]
Description=Clear Linux debuginfo fuse monitor
After=clr_debug_daemon.service
After=clr_debug_daemon.socket
Requires=clr_debug_daemon.socket
DefaultDependencies=no
[Service]
Type=simple
ExecStart=/usr/bin/clr_debug_fuse
DefaultDependencies=no
Nice=10
[Install]
WantedBy=multi-user.target
+5 -1
View File
@@ -2,17 +2,21 @@
# Process this file with autoconf to produce a configure script.
AC_PREREQ([2.66])
AC_INIT(clr-debug-info, 38, arjan@linux.intel.com)
AC_INIT(clr-debug-info, 50, arjan@linux.intel.com)
AM_INIT_AUTOMAKE([foreign -Wall -W subdir-objects])
AM_SILENT_RULES([yes])
AC_PROG_CC
AM_PROG_AR
AC_LANG(C)
AC_CONFIG_HEADERS([config.h])
PKG_CHECK_MODULES([curl], [libcurl])
PKG_CHECK_MODULES([fuse], [fuse])
PKG_CHECK_MODULES([SYSTEMD], [systemd])
PKG_CHECK_MODULES([LIBSYSTEMD], [libsystemd])
LT_INIT
AC_DEFINE([SOCKET_PATH], ["/run/clr-debug-info"], [path to communication socket])
dir=""
AC_ARG_WITH([systemdsystemunitdir], AS_HELP_STRING([--with-systemdsystemunitdir=DIR],
[path to systemd system service dir @<:@default=/usr/lib/systemd/system@:>@]), [dir=${withval}],
+4 -2
View File
@@ -8,5 +8,7 @@
# See tmpfiles.d(5) for details
# Clear tmp directories separately, to make them easier to override
d /var/cache/debuginfo/lib 755 root root 10d
d /var/cache/debuginfo/src 755 root root 1d
# Unfortunatly tmpfiles doesn't change the ownership for things if they
# are not listed.
d /var/cache/debuginfo/lib 755 dbginfo dbginfo 10d
d /var/cache/debuginfo/src 755 dbginfo dbginfo 1d
+146
View File
@@ -0,0 +1,146 @@
#!/bin/bash
#
# Clear Linux -- automatic debuginfo preparation
#
# Copyright 2019 Intel Corporation
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
# Foundation, version 3 or later of the License.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
# details.
#
# You should have received a copy of the GNU General Public License along with
# this program. If not, see <http://www.gnu.org/licenses/>.
if [ $# -eq 1 ] || [ $# -gt 2 ]; then
echo "Usage: $0 [SOURCEDIR] [DESTDIR]"
echo "Generates automatic debuginfo tarballs in DESTDIR using content from SOURCEDIR."
echo "The default SOURCEDIR and DESTDIR are /var/www/html/debuginfo.raw and"
echo "/var/www/html/debuginfo, respectively."
exit 0
fi >&2
if [ $# -eq 2 ]; then
if [ ! -d "$1" ]; then
echo "ERROR: SOURCEDIR argument must be an existing directory"
exit 1
fi
fi >&2
export SRC="${1:-/var/www/html/debuginfo.raw}"
export DEST="${2:-/var/www/html/debuginfo}"
srclist=$(mktemp -p .)
destlist=$(mktemp -p .)
trap "rm $srclist $destlist" EXIT
set -o pipefail
# First, scan the state of the SRC tree, containing all content for debuginfo RPMs.
if ! find "$SRC" -mindepth 4 -printf "%p %T@ %y\n" | sed "s|$SRC||" | sort -k 1 > "$srclist"; then
echo "ERROR: expected content in $SRC"
exit 1
fi
# Next, scan the state of the DEST tree, containing previously generated
# automatic debuginfo tarballs. Will be empty on first run.
mkdir -p "$DEST"
find "$DEST" -mindepth 2 -name '*.tar' -printf "%p %T@ %y\n" | sed "s|$DEST||;s|\.tar | |" | sort -k 1 > "$destlist"
process_one() {
srcname="$1"
destname="$2"
filetype="$3"
srcloc="$SRC/$srcname"
srcdir="$SRC/$(echo "$srcname" | sed -r 's@^(/usr/(lib|share|src)/debug(/src)?).*@\1@')"
tarcontent="$(echo "$srcname" | sed -r 's@^/usr/(lib|share|src)/debug(/src)?@.@')"
destdir="$DEST/$(dirname "$destname")"
dest="$DEST/$destname.tar"
echo "Creating $destname.tar ..."
mkdir -p "$destdir"
if [ "$filetype" = "f" ]; then
tar --no-recursion -C "$srcdir" -Jcf "$dest" "$tarcontent"
elif [ "$filetype" = "l" ]; then
# We "unsymlink" non-broken symlinks as an optimization by adding the
# symlink target to the dest tarball, transforming the name as appropriate.
if target=$(realpath -e "$srcloc" --relative-to="$srcdir" 2> /dev/null); then
tar --no-recursion -C "$srcdir" -Jcf "$dest" --transform='s|.*|'"$tarcontent"'|' "$target"
else
tar --no-recursion -C "$srcdir" -Jcf "$dest" "$tarcontent"
fi
elif [ "$filetype" = "d" ]; then
tar --no-recursion -C "$srcdir" -Jcf "$dest" "$tarcontent"
fi
}
export -f process_one
gawk '
BEGIN { OFS = "\t" }
LIST == "src" {
name = $1
srcname = name
ret = sub(/^\/usr\/lib\/debug/, "/lib", name)
if (ret == 0) {
ret = sub(/^\/usr\/src\/debug/, "/src", name)
if (ret == 0) {
ret = sub(/^\/usr\/share\/debug\/src/, "/src", name)
if (ret == 0) {
ret = sub(/^\/usr\/share\/debug/, "/lib", name)
if (ret == 0) {
print "Error: unexpected debuginfo file:", name
exit 1
}
}
}
}
src[name]["srcname"] = srcname
src[name]["time"] = $2
src[name]["type"] = $3
src[name]["done"] = 0
}
LIST == "dest" {
dest_content = 1
name = $1
time = $2
type = $3
if (name in src) {
# No need to recreate directory tars if they exist
if (src[name]["type"] == "d") {
src[name]["done"] = 1
next
} else {
# Only regenerate content if timestamp is newer
if (time < src[name]["time"]) {
print src[name]["srcname"], name, src[name]["type"]
}
src[name]["done"] = 1
}
}
}
END {
PROCINFO["sorted_in"] = "@ind_str_asc"
if (dest_content == 0) {
for (s in src) {
print src[s]["srcname"], s, src[s]["type"]
}
} else {
for (s in src) {
if (src[s]["done"] == 0) {
print src[s]["srcname"], s, src[s]["type"]
}
}
}
}
' LIST="src" "$srclist" LIST="dest" "$destlist" \
| parallel --colsep '\t' process_one
# vi: ft=sh et sw=2 sts=2
+8 -7
View File
@@ -36,9 +36,11 @@
#include <time.h>
#include <unistd.h>
#include "config.h"
/* 0.75 seconds timeout */
#define TIMEOUT 750000
#define TIMEOUT2 15000
#define TIMEOUT 75000
#define TIMEOUT2 1500
#define TIMEOUT3 500
char *prefix = "src";
@@ -58,7 +60,7 @@ void try_to_get(const char *path, int pid, time_t timestamp)
int shorttime = 0;
__nc_unused__ size_t wr = -1;
// printf("Trying to aquire %s\n", path);
// printf("Trying to aquire %s\n", path);
sockfd = socket(AF_UNIX, SOCK_STREAM, 0);
if (sockfd < 0) {
@@ -66,14 +68,13 @@ void try_to_get(const char *path, int pid, time_t timestamp)
}
sun.sun_family = AF_UNIX;
strcpy(sun.sun_path, ":clr-debug-info");
sun.sun_path[0] = 0; /* anonymous unix socket */
strcpy(sun.sun_path, SOCKET_PATH);
ret = connect(sockfd,
(struct sockaddr *)&sun,
offsetof(struct sockaddr_un, sun_path) + strlen(":clr-debug-info") + 1);
offsetof(struct sockaddr_un, sun_path) + strlen(SOCKET_PATH) + 1);
if (ret < 0) {
printf("Cannot connect %s\n", strerror(errno));
printf("Cannot connect to %s: %s\n", SOCKET_PATH, strerror(errno));
close(sockfd);
return;
}
+10
View File
@@ -75,6 +75,16 @@ static int xmp_getattr(const char *path, struct stat *stbuf)
memset(stbuf, 0, sizeof(struct stat));
res = lstat(newpath, stbuf);
/*
* filter out things that never should get fetched
* this prevents us from asking curl to fetch us useless things
*/
if (strncmp(path, "/.Trash", strlen("/.Trash")) == 0) {
return -ENOENT;
} else if (strcmp(path, "/") == 0) {
return 0;
}
/*
* get the file. if the st_mtime is set, this is just an async refresh, otherwise it's
* a synchronous request.
-243
View File
@@ -1,243 +0,0 @@
/*
* Fenrus Linux -- automatic debug information installation
* Clear Linux -- automatic debug information installation
* Server side preparation logic
*
* Copyright (C) 2013 Arjan van de Ven
* Copyright (C) 2014 Intel Corporation
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, version 2 or later of the License.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* Authors:
* Arjan van de Ven <arjanvandeven@gmail.com>
* Ikey Doherty <michael.i.doherty@intel.com>
*
*/
#define _GNU_SOURCE
#include <dirent.h>
#include <errno.h>
#include <libgen.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/types.h>
#include <sys/un.h>
#include <unistd.h>
#include "nica/files.h"
/*
this program takes a directory /var/www/html/debuginfo.raw,
in which all debuginfo rpms have previously been rpm2cpio'd, and
creates the content for /var/www/html/debuginfo
which contains tar'd up/compressed versions of all the files.
As part of this symlinks will be resolved server side
as much as possible.
*/
static void unsymlink(char *filename)
{
char *target;
__nc_unused__ int r;
target = canonicalize_file_name(filename);
// sleep(1);
// printf("Symlink %s points to %s\n", filename, target);
unlink(filename);
r = link(target, filename);
}
static void wait_for_loadavg(void)
{
FILE *file;
char line[4096];
__nc_unused__ char *ret = NULL;
while (1) {
double d;
file = fopen("/proc/loadavg", "r");
if (!file) {
return;
}
line[0] = 0;
ret = fgets(line, 4096, file);
fclose(file);
d = strtod(line, NULL);
if (d < 50) {
return;
}
sleep(5);
}
}
int tarcount;
static void do_one_file(char *base1, char *base2, char *path, int isdir)
{
char *dir2 = NULL;
autofree(char) * fullpath1, *fullpath2, *dir = NULL;
struct stat buf1, buf2;
int ret;
int sret;
if (asprintf(&fullpath1, "%s%s", base1, path) < 0) {
return;
}
if (asprintf(&fullpath2, "%s%s.tar", base2, path) < 0) {
return;
}
ret = lstat(fullpath1, &buf1);
if (ret) {
return;
}
if (S_ISLNK(buf1.st_mode)) {
// printf("%s is a symlink\n", fullpath1);
ret = stat(fullpath1, &buf2);
if (!ret) {
unsymlink(fullpath1);
}
ret = lstat(fullpath1, &buf1);
}
ret = lstat(fullpath2, &buf2);
if (ret || buf1.st_mtime > buf2.st_mtime) {
char *command = NULL;
unlink(fullpath2);
dir = strdup(fullpath2);
dir2 = dirname(dir);
if (dir2) {
(void)nc_mkdir_p(dir2, 00755);
}
if (chdir(base1) != 0) {
fprintf(stderr, "Failed to chdir: %s\n", strerror(errno));
return;
}
if (!isdir &&
asprintf(&command,
"tar --no-recursion -C %s -Jcf %s %s &",
base1,
fullpath2,
path) >= 0) {
// printf("Running -%s-\n", command);
printf("Processing %s\n", fullpath2);
tarcount++;
if (tarcount > 50) {
wait_for_loadavg();
tarcount = 0;
}
if ((sret = system(command)) != 0) {
fprintf(stderr, "Failure in command: [%d] %s\n", sret, command);
}
free(command);
}
if (isdir &&
asprintf(&command,
"tar --no-recursion -C %s -Jcf %s `find %s -type d` `find %s "
"-maxdepth 1 -type l` &",
base1,
fullpath2,
path,
path) >= 0) {
// printf("Running -%s-\n", command);
printf("Processing %s\n", fullpath2);
tarcount++;
if (tarcount > 50) {
wait_for_loadavg();
tarcount = 0;
}
if ((sret = system(command)) != 0) {
fprintf(stderr, "Failure in command: [%d] %s\n", sret, command);
}
free(command);
}
}
}
static void recurse_dir(char *base1, char *base2, char *path)
{
DIR *dir;
char *fullpath1 = NULL;
struct dirent *entry;
if (asprintf(&fullpath1, "%s%s", base1, path) < 0) {
return;
}
dir = opendir(fullpath1);
if (!dir) {
return;
}
while (1) {
autofree(char) *fullpath2 = NULL;
autofree(char) *newpath = NULL;
entry = readdir(dir);
if (!entry) {
break;
}
if (strcmp(entry->d_name, ".") == 0) {
continue;
}
if (strcmp(entry->d_name, "..") == 0) {
continue;
}
newpath = NULL;
fullpath2 = NULL;
if (asprintf(&fullpath2, "%s/%s", fullpath1, entry->d_name) < 0) {
return;
}
if (asprintf(&newpath, "%s/%s", path, entry->d_name) >= 0) {
struct stat sb;
stat(fullpath2, &sb);
if (entry->d_type == DT_DIR || S_ISDIR(sb.st_mode)) {
do_one_file(base1, base2, newpath, 1);
recurse_dir(base1, base2, newpath);
} else {
do_one_file(base1, base2, newpath, 0);
}
}
}
closedir(dir);
}
int main(int argc, char **argv)
{
if (argc < 0)
(void)argv;
recurse_dir("/var/www/html/debuginfo.raw/usr/lib/debug/",
"/var/www/html/debuginfo/lib/",
".");
recurse_dir("/var/www/html/debuginfo.raw/usr/src/debug/",
"/var/www/html/debuginfo/src/",
".");
return EXIT_SUCCESS;
}
+151 -40
View File
@@ -2,7 +2,7 @@
* Clear Linux -- automatic debug information installation
*
* Copyright (C) 2013 Arjan van de Ven
* Curl portions borrowed from the Fenrus Update code
* Curl portions borrowed from the Fenrus Update code
* which in part is (C) 2012 Intel Corporation
* Copyright (C) 2014 Intel Corporation
*
@@ -27,9 +27,12 @@
#define _GNU_SOURCE
#include <errno.h>
#include <grp.h>
#include <linux/capability.h>
#include <malloc.h>
#include <pthread.h>
#include <pwd.h>
#include <libgen.h>
#include <signal.h>
#include <stddef.h>
#include <stdio.h>
@@ -47,16 +50,20 @@
#include <curl/curl.h>
#include "systemd/sd-daemon.h"
#include "config.h"
#ifdef HAVE_ATOMIC_SUPPORT
#include <stdatomic.h>
#endif
#define TIMEOUT 600 /* 10 minutes */
static pthread_mutex_t dupes_mutex = PTHREAD_MUTEX_INITIALIZER;
char *urls[2] = { "https://cdn.download.clearlinux.org/debuginfo/",
"https://cdn.download.clearlinux.org/debuginfo/" };
"https://cdn-alt.download.clearlinux.org/debuginfo/" };
int urlcounter = 1;
static NcHashmap *hash = NULL;
@@ -162,11 +169,11 @@ static int curl_get_file(const char *url, const char *prefix, time_t timestamp)
{
CURLcode code;
long ret;
long changed;
int fd;
char filename[PATH_MAX];
autofree(char) *filename = NULL;
CURL *curl = NULL;
FILE *file;
struct stat statbuf;
if (avoid_dupes(url)) {
return 300;
@@ -177,12 +184,15 @@ static int curl_get_file(const char *url, const char *prefix, time_t timestamp)
return 301;
}
strcpy(filename, "/tmp/clr-debug-info-XXXXXX");
// fprintf(stderr, "Fetching %s, prefix %s, path %s\n", url, prefix, path);
if (asprintf(&filename, "/tmp/clr-debug-info-XXXXXX") < 0) {
curl_easy_cleanup(curl);
return 418;
}
fd = mkstemp(filename);
if (fd < 0) {
curl_easy_cleanup(curl);
return 500;
return 418;
}
file = fdopen(fd, "w");
@@ -203,6 +213,9 @@ static int curl_get_file(const char *url, const char *prefix, time_t timestamp)
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 30);
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1024);
/* request timestamp of files from server */
curl_easy_setopt(curl, CURLOPT_FILETIME, 1);
if (timestamp) {
curl_easy_setopt(curl, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE);
curl_easy_setopt(curl, CURLOPT_TIMEVALUE, timestamp);
@@ -217,7 +230,7 @@ static int curl_get_file(const char *url, const char *prefix, time_t timestamp)
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &ret);
fflush(file);
// printf("HTTP return code is %i\n", ret);
// printf("HTTP return code is %i\n", ret);
/* HTTP 304 is returned if (a) the cached debuginfo has the same
* timestamp or is newer than that on the server and (b) we haven't
@@ -231,20 +244,58 @@ static int curl_get_file(const char *url, const char *prefix, time_t timestamp)
if (ret == 200) {
autofree(char) *command = NULL;
// printf("Filename is %s\n", filename);
struct stat statbuf;
/* get timestamp, if any */
curl_easy_getinfo(curl, CURLINFO_FILETIME, &changed);
if (changed >= 0) {
struct timespec times[2];
times[0].tv_sec = (time_t)changed;
times[0].tv_nsec = 0;
times[1].tv_sec = (time_t)changed;
times[1].tv_nsec = 0;
futimens(fd, times);
}
memset(&statbuf, 0, sizeof(statbuf));
stat(filename, &statbuf);
if (statbuf.st_size > 0 &&
asprintf(&command,
"tar -C /var/cache/debuginfo/%s --no-same-owner --no-same-permissions -xf %s",
if (statbuf.st_size <= 0) {
ret = 418;
goto out;
}
/* test extraction first */
if (asprintf(&command, "tar -C /var/cache/debuginfo/%s --no-same-owner "
"--no-same-permissions -tf %s",
prefix,
filename) >= 0) {
if (system(command) != 0) {
fputs("Warning: tar extraction failed\n", stderr);
}
filename) < 0) {
ret = 418;
goto out;
}
if (system(command) != 0) {
ret = 418;
fprintf(stderr, "Error: tar validation failed\n");
goto out;
}
free(command); /* reuse */
if (asprintf(&command, "tar -C /var/cache/debuginfo/%s --no-same-owner "
"--no-same-permissions -xf %s",
prefix,
filename) < 0) {
ret = 418;
goto out;
}
if (system(command) != 0) {
ret = 418;
fprintf(stderr, "Error: tar extraction failed\n");
goto out;
}
}
out:
unlink(filename);
curl_easy_cleanup(curl);
fclose(file);
@@ -318,7 +369,7 @@ static void *server_thread(void *arg)
goto thread_end;
}
// printf("Getting url %s %i:%06i\n", url, before.tv_sec, before.tv_usec);
// printf("Getting url %s %i:%06i\n", url, before.tv_sec, before.tv_usec);
ret = curl_get_file(url, prefix, timestamp);
switch (ret) {
@@ -329,12 +380,12 @@ static void *server_thread(void *arg)
// ignore these error codes
break;
default:
printf("Request for %s resulted in error %i\n", url, ret);
fprintf(stderr, "Request for %s resulted in error %i\n", url, ret);
break;
}
gettimeofday(&after, NULL);
#if 0
#if 0
if (timedelta(before, after) > 0.6)
printf("Request for %s took %5.2f seconds (%i - %i)\n",
url,
@@ -360,8 +411,19 @@ int main(__nc_unused__ int argc, __nc_unused__ char **argv)
struct sockaddr_un sun;
int ret;
int curl_done = 0;
uid_t dbg_user = 0;
gid_t dbg_group = 0;
struct passwd *passwdentry;
const char *required_paths[] = { "/var/cache/debuginfo/lib", "/var/cache/debuginfo/src" };
umask(0);
passwdentry = getpwnam("dbginfo");
if (passwdentry) {
dbg_user = passwdentry->pw_uid;
dbg_group = passwdentry->pw_gid;
}
endpwent();
if (prctl(PR_SET_DUMPABLE, 0) != 0) {
fprintf(stderr,
"Failed to disable PR_SET_DUMPABLE. Do NOT gdb attach this process: %s\n",
@@ -376,44 +438,95 @@ int main(__nc_unused__ int argc, __nc_unused__ char **argv)
for (size_t i = 0; i < ARRAY_SIZE(required_paths); i++) {
const char *req_path = required_paths[i];
if (nc_file_exists(req_path)) {
continue;
struct stat st = { .st_ino = 0 };
if (lstat(req_path, &st) == 0) {
/* If the file already exists, check ownership
* and delete tree if incorrect. Essentially a
* one-off operation to transition from root owned to
* dbginfo owned */
if (st.st_uid == dbg_user) {
continue;
}
fprintf(stderr, "Removing old debug information %s\n", req_path);
nc_rm_rf(req_path);
}
if (!nc_mkdir_p(req_path, 00755)) {
fprintf(stderr, "Failed to mkdir: %s %s\n", strerror(errno), req_path);
return EXIT_FAILURE;
}
if (chown(req_path, dbg_user, dbg_group) != 0) {
fprintf(stderr, "Failed to chown: %s %s\n", strerror(errno), req_path);
return EXIT_FAILURE;
}
}
signal(SIGPIPE, SIG_IGN);
sockfd = socket(AF_UNIX, SOCK_STREAM, 0);
if (sockfd < 0) {
return EXIT_FAILURE;
if (sd_listen_fds(0) == 1) {
/* systemd socket activation */
sockfd = SD_LISTEN_FDS_START + 0;
} else if (sd_listen_fds(0) > 1) {
fprintf(stderr, "Too many file descriptors received.\n");
exit(EXIT_FAILURE);
} else {
sockfd = socket(AF_UNIX, SOCK_STREAM, 0);
if (sockfd < 0) {
exit(EXIT_FAILURE);
}
sun.sun_family = AF_UNIX;
strcpy(sun.sun_path, SOCKET_PATH);
ret = bind(sockfd,
(struct sockaddr *)&sun,
offsetof(struct sockaddr_un, sun_path) + strlen(SOCKET_PATH) + 1);
if (ret < 0) {
fprintf(stderr, "Failed to bind:%s \n", strerror(errno));
exit(EXIT_FAILURE);
}
if (listen(sockfd, 16) < 0) {
fprintf(stderr, "Failed to listen:%s \n", strerror(errno));
exit(EXIT_FAILURE);
}
}
sun.sun_family = AF_UNIX;
strcpy(sun.sun_path, ":clr-debug-info");
sun.sun_path[0] = 0; /* anonymous unix socket */
ret = bind(sockfd,
(struct sockaddr *)&sun,
offsetof(struct sockaddr_un, sun_path) + strlen(":clr-debug-info") + 1);
if (ret < 0) {
printf("Failed to bind:%s \n", strerror(errno));
return EXIT_FAILURE;
if (setgid(dbg_group)) {
fprintf(stderr, "Unable to drop privileges setgid %s\n", strerror(errno));
exit(EXIT_FAILURE);
}
if (listen(sockfd, 16) < 0) {
printf("Failed to listen:%s \n", strerror(errno));
return EXIT_FAILURE;
if (setgroups(1, &dbg_group)) {
fprintf(stderr, "Unable to drop privileges setgroups %s\n", strerror(errno));
exit(EXIT_FAILURE);
}
if (setuid(dbg_user)) {
fprintf(stderr, "Unable to drop privileges setuid %s\n", strerror(errno));
exit(EXIT_FAILURE);
}
while (1) {
fd_set rfds;
struct timeval tv;
int ret;
int clientsock;
pthread_t thread;
malloc_trim(0);
/* use select() to timeout and exit gracefully */
FD_ZERO(&rfds);
FD_SET(sockfd, &rfds);
tv.tv_sec = TIMEOUT;
tv.tv_usec = 0;
ret = select(sockfd + 1, &rfds, NULL, NULL, &tv);
if (ret == -1) {
perror("select()");
exit(EXIT_FAILURE);
} else if (ret == 0) {
break;
}
clientsock = accept(sockfd, NULL, NULL);
/* Too many connections, wait for the next loop/retry */
@@ -438,6 +551,4 @@ int main(__nc_unused__ int argc, __nc_unused__ char **argv)
if (hash) {
nc_hashmap_free(hash);
}
close(sockfd);
}