Release v2.1.0

journalprobe: don't process entire journal on restart
This adds an optional argument to journalprobe that allows seeking to the end of the journal. This is then used in a new service file that is called by telemctl on restart. This allows us to scan the entire journal on boot, and only scan new journal entries when restarting, or opting into telemetry manually. This reduces spam from repeated messages and fixes a possible privacy concern, as previously running telemctl start could include journal errors that occurred before telemetrics was opted into. Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
2026-09-01 11:15:51 +00:00 · 2019-02-20 18:09:47 +00:00 · 2019-02-18 09:09:39 -08:00 · 2019-02-18 08:59:05 -08:00 · 2019-02-07 17:11:51 -08:00 · 2019-01-28 11:37:53 -08:00
52 changed files with 889 additions and 681 deletions
@@ -42,9 +42,9 @@ hprobe
 bertprobe
 crashprobe
 journalprobe
+pythonprobe
 telem-record-gen
 klogscanner
-oopsprobe
 pstoreclean
 pstoreprobe
 telem_journal
@@ -56,7 +56,11 @@ src/gcc.out
 tests/demo
 tests/check_config
 tests/check_daemon
+tests/check_journal
 tests/check_libtelemetry
+tests/check_ncb64
+tests/check_postd
+tests/check_probd
 tests/check_probes
 src/data/*.path
 src/data/*.service
@@ -1,3 +1,5 @@
+[![Build Status](https://travis-ci.org/clearlinux/telemetrics-client.svg?branch=master)](https://travis-ci.org/clearlinux/telemetrics-client)
+
 Telemetrics-client
 ==================

@@ -321,3 +323,8 @@ $ sudo telemctl journal --record_id a19a0d41ba16788881e274b19b8a1be4 --include_r
 $ org.clearlinux/hello/world     Mon 2018-04-02 17:48:01 UTC a19a0d41ba16788881e274b19b8a1be4 5de9de8d5f3c6a7d445d75ba01cc3322 60c014cd-4693-40f1-b334-548cd932949b
 $ hello
 ```
+
+## Security Disclosures
+
+To report a security issue or receive security advisories please follow procedures
+in this [link](https://01.org/security).
@@ -24,7 +24,6 @@ AM_CPPFLAGS = \
 	-DABSTOPSRCDIR=\"$(abs_top_srcdir)\" \
 	-DDATADIR=\"$(datadir)\" \
 	-DLOCALSTATEDIR=\"$(localstatedir)\" \
-	-DKERNELOOPSDIR=\"$(localstatedir)/cache/telemetry/oops\" \
 	-DPSTOREDIR=\"$(localstatedir)/cache/telemetry/pstore\" \
 	-DTESTOOPSDIR=\"$(top_srcdir)/tests/oops_test_files\"

@@ -2,7 +2,7 @@
 # Process this file with autoconf to produce a configure script.

 AC_PREREQ([2.69])
-AC_INIT([telemetrics-client], [2.0.0], [https://clearlinux.org/])
+AC_INIT([telemetrics-client], [2.1.0], [https://clearlinux.org/])
 AC_CONFIG_AUX_DIR([build-aux])
 AM_INIT_AUTOMAKE([1.14 -Wall -Werror -Wno-extra-portability foreign subdir-objects])
 AM_SILENT_RULES([yes])
@@ -1,9 +1,9 @@

 ## Notes on manual pages

-The manual pages are generated using `ronn(1)`. To recreate them,
-run `make manpages` in the toplevel  folder. If you want to edit
-the documentation for upstream changes, make sure to edit the `*.md`
+The manual pages are generated using `rst2man.py`. To recreate them,
+run `make manpages` in the toplevel folder. If you want to edit
+the documentation for upstream changes, make sure to edit the `*.rst`
 files and not the shipped nroff output files.


@@ -35,10 +35,10 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 \fBtelem\-record\-gen\fP <options>
 .SH DESCRIPTION
 .sp
-Generate and attempt to send a telemetry record to \fBtelemd\fP(1). This
+Generate and attempt to send a telemetry record to \fBtelemprobd\fP(1). This
 program can be used to create a telemetry record from a shell script
 or from another program using e.g. \fBsystem()\fP\&. The program connects
-to a local \fBtelemd\fP(1) daemon to spool the record for delivery to
+to a local \fBtelemprobd\fP(1) daemon to spool the record for delivery to
 the actual collection service.
 .sp
 The payload can be provided as standard input, as a file with the
@@ -79,7 +79,7 @@ File to read payload from.
 Version number for format of payload (default 1).
 .IP \(bu 2
 \fB\-e\fP, \fB\-\-event\-id\fP <version>:
-Event id to use in the record. If not provided a ramdonly generated id will be assigned to record.
+Event id to use in the record. If not provided a randomly generated id will be assigned to record.
 .UNINDENT
 .UNINDENT
 .UNINDENT
@@ -19,10 +19,10 @@ SYNOPSIS
 DESCRIPTION
 ===========

-Generate and attempt to send a telemetry record to ``telemd``\(1). This
+Generate and attempt to send a telemetry record to ``telemprobd``\(1). This
 program can be used to create a telemetry record from a shell script
 or from another program using e.g. ``system()``. The program connects
-to a local ``telemd``\(1) daemon to spool the record for delivery to
+to a local ``telemprobd``\(1) daemon to spool the record for delivery to
 the actual collection service.

 The payload can be provided as standard input, as a file with the
@@ -79,4 +79,3 @@ SEE ALSO
 * ``telemd``\(1)
 * https://github.com/clearlinux/telemetrics-client
 * https://clearlinux.org/documentation/
-
@@ -38,8 +38,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .SH DESCRIPTION
 .sp
 Control actions for telemetry services. The command can be used to start,
-restart, or stop \fBtelemd\fP(1), or to opt\-in or opt\-out of telemetry delivery
-of records to a central telemetry service.
+restart, or stop \fBtelemprobd\fP(1) and \fBtelempostd\fP(1), or to opt\-in or opt\-out of telemetry delivery of records to a central telemetry service.
 .SH OPTIONS
 .INDENT 0.0
 .INDENT 3.5
@@ -22,8 +22,7 @@ DESCRIPTION
 ===========

 Control actions for telemetry services. The command can be used to start,
-restart, or stop ``telemd``\(1), or to opt-in or opt-out of telemetry delivery
-of records to a central telemetry service.
+restart, or stop ``telemprobd``\(1) and ``telempostd``\(1), or to opt-in or opt-out of telemetry delivery of records to a central telemetry service.


 OPTIONS
@@ -32,11 +31,11 @@ OPTIONS
 * ``start``|``stop``|``restart``:
   Starts, stops or restarts all running telemetry services.

- * ``opt-in``: 
+ * ``opt-in``:
   Opts in to telemetry, and starts telemetry services. The opt-out file
   ``/etc/telemetrics/opt-out`` is removed.

- * ``opt-out``: 
+ * ``opt-out``:
   Opts out of telemetry, and stops telemetry services. The opt-out file
   ``/etc/telemetrics/opt-out`` is created.

@@ -56,4 +55,3 @@ SEE ALSO
 * ``telemd``\(1)
 * https://github.com/clearlinux/telemetrics-client
 * https://clearlinux.org/documentation/
-
@@ -37,8 +37,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 \fB/usr/share/defaults/telemetrics/telemetrics.conf\fP
 .SH DESCRIPTION
 .sp
-This file contains configuration parameters for the \fBtelemd\fP(1) telemetry
-service daemon. The daemon reads this file at startup if it exists.
+This file contains configuration parameters for the \fBtelemprobd\fP(1) and \fBtelempostd\fP(1) telemetry service daemons. The daemon reads this file at startup if it exists.
 .SH SYNTAX
 .sp
 The configuration file contains \fBkey=value\fP pairs, formatted as plain
@@ -54,7 +53,7 @@ Server URL including protocol designator.
 .IP \(bu 2
 \fBsocket_path=<path>\fP
 .sp
-Path to the socket that \fItelemd\fP will listen on.
+Path to the socket that \fItelemprobd\fP will listen on.
 .IP \(bu 2
 \fBcainfo=<path>\fP
 .sp
@@ -82,7 +81,7 @@ and not the actual size of the record itself.
 .IP \(bu 2
 \fBspool_process_time=<seconds>\fP
 .sp
-Time in seconds for processing spool. Valid range: 120..3600. Values
+Time in seconds for processing spool. Valid range: 120..300. Values
 outside this range are clamped.
 .IP \(bu 2
 \fBrate_limit_enabled=<true|false>\fP
@@ -114,7 +113,9 @@ delivery over network. Valid stategies: \fBspool\fP, \fBdrop\fP\&.
 .SH SEE ALSO
 .INDENT 0.0
 .IP \(bu 2
-\fBtelemd\fP(1)
+\fBtelemprobd\fP(1)
+.IP \(bu 2
+\fBtelempostd\fP(1)
 .IP \(bu 2
 \fI\%https://github.com/clearlinux/telemetrics\-client\fP
 .IP \(bu 2
@@ -21,8 +21,7 @@ SYNOPSIS
 DESCRIPTION
 ===========

-This file contains configuration parameters for the ``telemd``\(1) telemetry
-service daemon. The daemon reads this file at startup if it exists.
+This file contains configuration parameters for the ``telemprobd``\(1) and ``telempostd``\(1) telemetry service daemons. The daemon reads this file at startup if it exists.


 SYNTAX
@@ -43,7 +42,7 @@ OPTIONS

 -  ``socket_path=<path>``

-   Path to the socket that `telemd` will listen on.
+   Path to the socket that `telemprobd` will listen on.

 -  ``cainfo=<path>``

@@ -71,7 +70,7 @@ OPTIONS

 -  ``spool_process_time=<seconds>``

-   Time in seconds for processing spool. Valid range: 120..3600. Values
+   Time in seconds for processing spool. Valid range: 120..300. Values
   outside this range are clamped.

 -  ``rate_limit_enabled=<true|false>``
@@ -104,7 +103,7 @@ OPTIONS
 SEE ALSO
 ========

-* ``telemd``\(1)
+* ``telemprobd``\(1)
+* ``telempostd``\(1)
 * https://github.com/clearlinux/telemetrics-client
 * https://clearlinux.org/documentation/
-
@@ -44,11 +44,11 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .sp
 \fBvoid tm_free_record(struct telem_ref *t_ref)\fP
 .sp
-\fBvoid tm_set_config_file(char *c_file)\fP
+\fBint tm_set_config_file(const char *c_file)\fP
 .SH DESCRIPTION
 .sp
 The functions in the telemetry library facilitate the delivery of
-telemetry data to the \fBtelemd\fP(1) service.
+telemetry data to the \fBtelemprobd\fP(1) service.
 .sp
 The function \fBtm_create_record()\fP initializes a telemetry record and
 sets the severity and classification of that record, as well as the
@@ -67,12 +67,12 @@ configuration path to the telemetry library.
 .SH RETURN VALUES
 .sp
 All these functions return \fB0\fP on success, or a non\-zero return value
-if an error occurred. The functions \fBtm_free_record()\fP and \fBtm_set_config_file()\fP
-do not return any values.
+if an error occurred. The function \fBtm_free_record()\fP does not return
+any value.
 .SH SEE ALSO
 .INDENT 0.0
 .IP \(bu 2
-\fBtelemd\fP(1)
+\fBtelemprobd\fP(1)
 .IP \(bu 2
 \fI\%https://github.com/clearlinux/telemetrics\-client\fP
 .IP \(bu 2
@@ -14,9 +14,9 @@ SYNOPSIS
 ========

 ``#include "telemetry.h"``
-  
+
 ``struct telem_ref { struct telem_record *record; };``
-  
+
 ``int tm_create_record(struct telem_ref **t_ref, uint32_t severity, char *classification, uint32_t payload_version)``

 ``int tm_set_payload(struct telem_ref *t_ref, char *payload)``
@@ -25,14 +25,14 @@ SYNOPSIS

 ``void tm_free_record(struct telem_ref *t_ref)``

-``void tm_set_config_file(char *c_file)``
+``int tm_set_config_file(const char *c_file)``


 DESCRIPTION
 ===========

 The functions in the telemetry library facilitate the delivery of
-telemetry data to the ``telemd``\(1) service.
+telemetry data to the ``telemprobd``\(1) service.

 The function ``tm_create_record()`` initializes a telemetry record and
 sets the severity and classification of that record, as well as the
@@ -54,14 +54,13 @@ RETURN VALUES
 =============

 All these functions return ``0`` on success, or a non-zero return value
-if an error occurred. The functions ``tm_free_record()`` and ``tm_set_config_file()``
-do not return any values.
+if an error occurred. The function ``tm_free_record()`` does not return
+any value.


 SEE ALSO
 ========

-* ``telemd``\(1)
+* ``telemprobd``\(1)
 * https://github.com/clearlinux/telemetrics-client
 * https://clearlinux.org/documentation/
-
@@ -49,7 +49,7 @@ telemetry service. Telemetry data can be in any format, and is relayed as\-is.
 .INDENT 0.0
 .IP \(bu 2
 \fB\-f\fP, \fB\-\-config_file\fP [<file>]:
-Configuration file. This overides the other parameters.
+Configuration file. This overrides the other parameters.
 .IP \(bu 2
 \fB\-h\fP, \fB\-\-help\fP:
 Display this help message.
@@ -1,8 +1,8 @@
 .\" Man page generated from reStructuredText.
 .
-.TH TELEMD 1 "" "" ""
+.TH TELEMPROBD 1 "" "" ""
 .SH NAME
-telemd \- Telemetry client service
+telemprobd \- Telemetry probes service
 .
 .nr rst2man-indent-level 0
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-\fBtelemd\fP <flags>
+\fBtelemprobd\fP <flags>
 .sp
 \fB/etc/telemetrics/telemetrics.conf\fP
 .sp
@@ -41,7 +41,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 \fB/etc/telemetrics/opt\-in\-static\-machine\-id\fP
 .SH DESCRIPTION
 .sp
-The \fBtelemd\fP program handles communication between telemetry client and telemetry
+The \fBtelemprobd\fP program handles communication between telemetry client and telemetry
 probes.
 .SH OPTIONS
 .INDENT 0.0
@@ -65,7 +65,7 @@ Print the program version.
 \fB/usr/share/defaults/telemetrics/telemetrics.conf\fP
 .INDENT 2.0
 .INDENT 3.5
-If no custom configuration file is found, \fBtelemd\fP uses the
+If no custom configuration file is found, \fBtelemprobd\fP uses the
 settings in this file.
 .UNINDENT
 .UNINDENT
@@ -73,7 +73,7 @@ settings in this file.
 \fB/etc/telemetrics/telemetrics.conf\fP
 .INDENT 2.0
 .INDENT 3.5
-Custom configuration file that \fBtelemd\fP reads. See \fBtelemetrics.conf\fP(5).
+Custom configuration file that \fBtelemprobd\fP reads. See \fBtelemetrics.conf\fP(5).
 .UNINDENT
 .UNINDENT
 .IP \(bu 2
@@ -4,16 +4,17 @@ declare -a SPECIAL_UNITS=(
  hprobe.timer
  telemprobd.socket
  telempostd.path
+  klogscanner.service
+  journal-probe-tail.service
+  python-probe.path
 )

 declare -a SERVICES=(
  hprobe.service
-  journal-probe.service
-  oops-probe.service
  pstore-probe.service
-  klogscanner.service
  telemprobd.service
  telempostd.service
+  journal-probe.service
 )

 SCRIPT="$0"
@@ -78,7 +78,7 @@ static const uint32_t RECORD_FORMAT_VERSION = 4;
 #define TM_OPT_OUT_FILE "/etc/telemetrics/opt-out"

 /* Currently max supported payload size is 8kb */
-#define MAX_PAYLOAD_SIZE 8192
+#define MAX_PAYLOAD_LENGTH 8192

 /* Maximum size for a category string is 122 bytes */
 #define MAX_CLASS_LENGTH 122
@@ -86,12 +86,12 @@ static const uint32_t RECORD_FORMAT_VERSION = 4;
 /* Maximum size for a sub category 40 bytes */
 #define MAX_SUBCAT_LENGTH 40

-/* Max Payload size is 8 kibabytes (8192) */
-#define MAX_PAYLOAD_LENGTH 8192
+/* Exit daemon after idle for this duration in seconds */
+#define TM_DAEMON_EXIT_TIME  (5 * 60)

 /* SPOOL DEFINES */
-/* Spooling should run every TM_SPOOL_RUN_MAX atleast */
-#define TM_SPOOL_RUN_MAX (60 /*min*/ * 60 /*sec*/)
+/* Spooling should run at least every TM_SPOOL_RUN_MAX */
+#define TM_SPOOL_RUN_MAX TM_DAEMON_EXIT_TIME

 /* Spooling should not run more often than TM_SPOOL_RUN_MIN */
 #define TM_SPOOL_RUN_MIN (2 /*min*/ * 60 /*sec*/)
@@ -102,9 +102,6 @@ static const uint32_t RECORD_FORMAT_VERSION = 4;
 /* Maximum records that can be processed in a single spool run loop*/
 #define TM_SPOOL_MAX_PROCESS_RECORDS 60

-/* Time to check to exit from  the daemon */
-#define TM_DAEMON_EXIT_TIME  (2 /*h*/ * 60 /*m*/ * 60 /*sec*/)
-
 /* Very simple structure. Array of header strings and a payload. Calling
 * program is reponsible for passing in the payload as a simple string.
 */
@@ -52,7 +52,7 @@ const char *config_key_bool[] = { NULL, "rate_limit_enabled", "daemon_recycling_

 static struct configuration config = { { 0 }, { 0 }, { 0 }, false, NULL };

-int validate_config_file(char *f)
+static int validate_config_file(const char *f)
 {
        struct stat sbuf;

@@ -60,26 +60,34 @@ int validate_config_file(char *f)
                return -EINVAL;
        }

+        /* check if the file exists and is a regular file */
        if (stat(f, &sbuf) != 0) {
                return -errno;
        }

+        if (!S_ISREG(sbuf.st_mode)) {
+                return -EINVAL;
+        }
+
        return 0;

 }

-void set_config_file(char *filename)
+int set_config_file(const char *filename)
 {
+        int ret;

-        /* TODO: This should return a status, but that will require coordinating
-         * an API change.  For now, config-entries referencing a null config
-         * file will fail, so the end result still works. */
-
-        if (validate_config_file(filename) == 0) {
-                config_file = strdup(filename);
-                cmd_line_cfg = true;
+        if ((ret = validate_config_file(filename)) == 0) {
+                char *filename1 = strdup(filename);
+                if (filename1 == NULL) {
+                    ret = -errno;
+                } else {
+                    config_file = filename1;
+                    cmd_line_cfg = true;
+                }
        }

+        return ret;
 }

 bool read_config_from_file(char *config_file, struct configuration *config)
@@ -63,7 +63,7 @@ typedef struct configuration {
 } configuration;

 /* Sets the configuration file to be used later */
-void set_config_file(char *filename);
+int set_config_file(const char *filename);

 /* Parses the ini format config file */
 bool read_config_from_file(char *filename, struct configuration *config);
@@ -16,7 +16,7 @@ spool_dir=/tmp/spool
 spool_max_size=1024

 #time in seconds for processing spool
-spool_process_time=900
+spool_process_time=180

 #rate limiting record burst limit
 record_burst_limit=100
@@ -5,7 +5,7 @@ After=telemprobd.socket
 ConditionPathExists=!/etc/telemetrics/opt-out

 [Service]
-ExecStart=@bindir@/hprobe -H
+ExecStart=@bindir@/hprobe -Hlu
 User=telemetry

 [Install]
@@ -0,0 +1,12 @@
+[Unit]
+Description=Telemetrics Systemd Journal Probe
+Requires=telemprobd.socket
+After=telemprobd.socket
+ConditionPathExists=!/etc/telemetrics/opt-out
+
+[Service]
+ExecStart=@bindir@/journalprobe -t
+User=telemetry
+
+[Install]
+WantedBy=multi-user.target
@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics Kernel Log Scanner
 Requires=telemprobd.socket
-After=telemprobd.socket oops-probe.service
+After=telemprobd.socket

 [Service]
 ExecStart=@bindir@/klogscanner
@@ -16,7 +16,9 @@ EXTRA_DIST += \
 	%D%/hprobe.timer \
 	%D%/bert-probe.service.in \
 	%D%/journal-probe.service.in \
-	%D%/oops-probe.service.in \
+	%D%/journal-probe-tail.service.in \
+	%D%/python-probe.service.in \
+	%D%/python-probe.path.in \
 	%D%/pstore-probe.service.in \
 	%D%/klogscanner.service.in \
 	%D%/pstore-clean.service.in \
@@ -53,7 +55,9 @@ systemdunit_DATA = \
 	%D%/hprobe.timer \
 	%D%/bert-probe.service \
 	%D%/journal-probe.service \
-	%D%/oops-probe.service \
+	%D%/journal-probe-tail.service \
+	%D%/python-probe.service \
+	%D%/python-probe.path \
 	%D%/pstore-probe.service \
 	%D%/klogscanner.service \
 	%D%/pstore-clean.service \
@@ -72,7 +76,7 @@ systemdunit_DATA = \
 %D%/journal-probe.service: %D%/journal-probe.service.in
 	$(pathfix) < $< > $@

-%D%/oops-probe.service: %D%/oops-probe.service.in
+%D%/journal-probe-tail.service: %D%/journal-probe-tail.service.in
 	$(pathfix) < $< > $@

 %D%/pstore-probe.service: %D%/pstore-probe.service.in
@@ -90,6 +94,12 @@ systemdunit_DATA = \
 %D%/telempostd.service: %D%/telempostd.service.in
 	$(pathfix) < $< > $@

+%D%/python-probe.service: %D%/python-probe.service.in
+	$(pathfix) < $< > $@
+
+%D%/python-probe.path: %D%/python-probe.path.in
+	$(pathfix) < $< > $@
+
 %D%/telemprobd.service: %D%/telemprobd.service.in
 	$(pathfix) < $< > $@

@@ -113,13 +123,15 @@ clean-local:
 		%D%/telemprobd.socket \
 		%D%/telempostd.service \
 		%D%/telempostd.path \
+		%D%/python-probe.service \
+		%D%/python-probe.path \
 		%D%/telemprobd-update-trigger.service \
 		%D%/telemetrics.conf \
 		%D%/telemetrics-dirs.conf \
 		%D%/libtelemetry.pc \
 		%D%/40-crash-probe.conf \
 		%D%/journal-probe.service \
-		%D%/oops-probe.service \
+		%D%/journal-probe-tail.service \
 		%D%/pstore-probe.service \
 		%D%/klogscanner.service \
 		%D%/pstore-clean.service \
@@ -0,0 +1,9 @@
+[Unit]
+Description=Python Probe Daemon Staging
+ConditionPathExists=!/etc/telemetrics/opt-out
+
+[Path]
+DirectoryNotEmpty=@localstatedir@/lib/telemetry/python
+
+[Install]
+WantedBy=multi-user.target
@@ -1,11 +1,13 @@
 [Unit]
-Description=Telemetrics Kernel Oops Probe
+Description=Telemetrics Python Probe
 Requires=telemprobd.socket
 After=telemprobd.socket
+ConditionPathExists=!/etc/telemetrics/opt-out

 [Service]
-ExecStart=@bindir@/oopsprobe
 User=telemetry
+ExecStart=@bindir@/pythonprobe

 [Install]
 WantedBy=multi-user.target
+
@@ -1,7 +1,7 @@
 d @localstatedir@/lib/telemetry 0755 telemetry telemetry -
+d @localstatedir@/lib/telemetry/python 01777 telemetry telemetry -
 d @localstatedir@/spool/telemetry 0750 telemetry telemetry -
 d @localstatedir@/log/telemetry 0750 telemetry telemetry -
 d @localstatedir@/log/telemetry/records 0750 telemetry telemetry -
 d @localstatedir@/cache/telemetry 0750 telemetry telemetry -
-d @localstatedir@/cache/telemetry/oops 0750 telemetry telemetry -
 d @localstatedir@/cache/telemetry/pstore 0750 telemetry telemetry -
@@ -23,8 +23,8 @@ spool_dir=@localstatedir@/spool/telemetry
 spool_max_size=5120

 # time in seconds for processing spool
-# Valid range: 120..3600. Values outside this range are clamped.
-spool_process_time=900
+# Valid range: 120..300. Values outside this range are clamped.
+spool_process_time=120

 # rate limit enabled - if this is set to false then all rate-limiting disabled.
 # It is possible to disable each rate-limit individually below.
@@ -11,7 +11,8 @@ bin_PROGRAMS = \
 	%D%/journal/journal.h

 %C%_telemprobd_LDADD = $(CURL_LIBS) \
-	%D%/libtelem-shared.la
+	%D%/libtelem-shared.la \
+	%D%/libtelemetry.la

 %C%_telemprobd_CFLAGS = \
 	$(AM_CFLAGS)
@@ -51,7 +52,8 @@ endif
 	%D%/iorecord.h

 %C%_telempostd_LDADD = $(CURL_LIBS) \
-	%D%/libtelem-shared.la
+	%D%/libtelem-shared.la \
+	%D%/libtelemetry.la

 %C%_telempostd_CFLAGS = \
 	$(AM_CFLAGS)
@@ -21,6 +21,7 @@
 #include <stdbool.h>
 #include <sys/stat.h>

+#include "telemetry.h"
 #include "log.h"
 #include "spool.h"
 #include "configuration.h"
@@ -40,9 +41,7 @@ int main(int argc, char **argv)
 {

        int c;
-        int ret = 0;
        int opt_index = 0;
-        char *config_file = NULL;
        TelemPostDaemon daemon;

        struct option opts[] = {
@@ -55,17 +54,11 @@ int main(int argc, char **argv)
        while ((c = getopt_long(argc, argv, "f:hV", opts, &opt_index)) != -1) {
                switch (c) {
                        case 'f':
-                                config_file = optarg;
-                                struct stat buf;
-                                ret = stat(config_file, &buf);
-
-                                /* check if the file exists and is a regular file */
-                                if (ret == -1 || !S_ISREG(buf.st_mode)) {
-                                        telem_log(LOG_ERR, "Configuration file"
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
                                                  " path not valid\n");
-                                        exit(EXIT_FAILURE);
+                                    exit(EXIT_FAILURE);
                                }
-                                set_config_file(config_file);
                                break;
                        case 'V':
                                printf(PACKAGE_VERSION "\n");
@@ -31,6 +31,7 @@
 #include <signal.h>
 #include <malloc.h>

+#include "telemetry.h"
 #include "config.h"
 #include "common.h"
 #ifdef  HAVE_SYSTEMD_SD_DAEMON_H
@@ -74,7 +75,6 @@ int main(int argc, char **argv)
        client *cl = NULL;
        client *current_client = NULL;
        int c;
-        char *config_file = NULL;
        int opt_index = 0;
        sigset_t mask;
        //bool interrupted = false;
@@ -92,18 +92,12 @@ int main(int argc, char **argv)

        while ((c = getopt_long(argc, argv, "f:hV", opts, &opt_index)) != -1) {
                switch (c) {
-                        case 'f':
-                                config_file = optarg;
-                                struct stat buf;
-                                ret = stat(config_file, &buf);
-
-                                /* check if the file exists and is a regular file */
-                                if (ret == -1 || !S_ISREG(buf.st_mode)) {
-                                        telem_log(LOG_ERR, "Configuration file"
+                         case 'f':
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
                                                  " path not valid\n");
-                                        exit(EXIT_FAILURE);
+                                    exit(EXIT_FAILURE);
                                }
-                                set_config_file(config_file);
                                break;
                        case 'h':
                                print_usage(argv[0]);
@@ -220,7 +214,7 @@ int main(int argc, char **argv)

        bool daemon_recycling_enabled = daemon_recycling_enabled_config();
        int spool_process_time = spool_process_time_config();
-        time_t last_daemon_start_time = time(NULL);
+        time_t last_record_received = time(NULL);

        ret = update_machine_id();
        if (ret == -1) {
@@ -315,13 +309,14 @@ int main(int argc, char **argv)
                                        }
                                        assert(current_client);
                                        handle_client(&daemon, i, current_client);
+                                        last_record_received = time(NULL);
                                }
                        }
                } else {
                        time_t now = time(NULL);
                        /* time to recycle the daemon has elapsed*/
                        if (daemon_recycling_enabled &&
-                            difftime(now, last_daemon_start_time) >= TM_DAEMON_EXIT_TIME) {
+                            difftime(now, last_record_received) >= TM_DAEMON_EXIT_TIME) {
                                /* Exit */
                                telem_log(LOG_INFO, "Daemon exiting for recycling\n");
                                goto clean_exit;
@@ -14,11 +14,13 @@ The default probes provided along the telemetry client code are:
 * journalprobe: a probe that monitors the systemd journal for log messages
  from failed services.

-* oopsprobe: a probe to collect 'oops messages' when the kernel detects a
+* klogscanner: a probe to collect 'oops messages' when the kernel detects a
  problem.

 * pstoreprobe: probe to collect messages left on pstore filesystem.

+* pythonprobe: a probe that monitors Python generated traceback files.
+
 * telem-record-gen: this is a "general-purpose probe" for sending custom
  records on-the-fly. This tool can be used stand alone or as a part of a
  script to implement a probe.
@@ -25,6 +25,7 @@
 #include "nica/b64enc.h"

 #include "config.h"
+#include "log.h"

 static const char bert_record_file[] = "/sys/firmware/acpi/tables/data/BERT";
 static const char telem_record_class[] = "org.clearlinux/bert/debug";
@@ -39,35 +40,16 @@ void print_usage(char *prog)
        printf("  -V,  --version        Print the program version\n");
 }

-static int allocate_payload_buffer(char **payload)
-{
-        int ret = 0;
-
-        *payload = (char *)malloc(MAX_PAYLOAD_SIZE);
-
-        if (*payload == NULL) {
-                goto out;
-        }
-
-        *payload = memset(*payload, 0, MAX_PAYLOAD_SIZE);
-        ret = 1;
-
-out:
-        return ret;
-}
-
 int main(int argc, char **argv)
 {
        struct telem_ref *tm_handle = NULL;
        char *classification = (char *)telem_record_class;
-        char *payload = NULL;
+        char *payload;
+        int ret;

        // Following vars are for arg parsing.
        int c;
-        char *cfile = NULL;
        int opt_index = 0;
-        int ret = 0;
-
        struct option opts[] = {
                { "config_file", 1, NULL, 'f' },
                { "help", 0, NULL, 'h' },
@@ -75,22 +57,14 @@ int main(int argc, char **argv)
                { NULL, 0, NULL, 0 }
        };

-        while ((c = getopt_long(argc, argv, "f:hHV", opts, &opt_index)) != -1) {
+        while ((c = getopt_long(argc, argv, "f:hV", opts, &opt_index)) != -1) {
                switch (c) {
                        case 'f':
-                                if (asprintf(&cfile, "%s", (char *)optarg) < 0) {
-                                        exit(EXIT_FAILURE);
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
+                                                  " path not valid\n");
+                                    exit(EXIT_FAILURE);
                                }
-
-                                struct stat buf;
-                                ret = stat(cfile, &buf);
-
-                                /* check if the file exists and is a regular file */
-                                if (ret == -1 || !S_ISREG(buf.st_mode)) {
-                                        exit(EXIT_FAILURE);
-                                }
-
-                                tm_set_config_file(cfile);
                                break;
                        case 'h':
                                print_usage(argv[0]);
@@ -103,36 +77,37 @@ int main(int argc, char **argv)
                }
        }

-        if (!(ret = allocate_payload_buffer(&payload))) {
+        payload = calloc(sizeof(char), MAX_PAYLOAD_LENGTH);
+        if (!payload) {
                printf("Unable to allocate more memory.\n");
-                goto fail;
+                return -ENOMEM;
        }

-        if (!(ret = nc_b64enc_filename(bert_record_file, payload, MAX_PAYLOAD_SIZE))) {
+        if (!(ret = nc_b64enc_filename(bert_record_file, payload, MAX_PAYLOAD_LENGTH))) {
                printf("Failed to read payload from: %s\n", bert_record_file);
-                goto fail;
+                ret = EXIT_FAILURE;
+                goto done;
        }

        if ((ret = tm_create_record(&tm_handle, severity, classification,
                                    payload_version)) < 0) {
                printf("Failed to create record: %s\n", strerror(-ret));
-                goto fail;
+                goto done;
        }

        if ((ret = tm_set_payload(tm_handle, payload)) < 0) {
                printf("Failed to set record payload: %s\n", strerror(-ret));
-                goto fail;
+                goto done;
        }

-        free(payload);
-
        if ((ret = tm_send_record(tm_handle)) < 0) {
                printf("Failed to send record to daemon: %s\n", strerror(-ret));
-                goto fail;
+                goto done;
        }

-fail:
-        free(cfile);
+        ret = EXIT_SUCCESS;
+done:
+        free(payload);
        tm_free_record(tm_handle);
        tm_handle = NULL;

@@ -69,7 +69,6 @@ static char error_class[30] = "org.clearlinux/crash/error";
 static char unknown_class[30] = "org.clearlinux/crash/unknown";

 static char temp_core[] = "/tmp/corefile-XXXXXX";
-static bool keep_core = false;

 static const Dwfl_Callbacks cb =
 {
@@ -463,7 +462,6 @@ static bool is_banned_path(char *fullpath)
        return false;
 }

-static char *config_file = NULL;
 static char *core_file = NULL;
 static char *proc_path = NULL;
 static long int signal_num = -1;
@@ -490,7 +488,7 @@ static void print_help(void)
        printf("  -h, --help            Show help options\n");
        printf("\n");
        printf("Application Options:\n");
-        printf("  -f, --config-file     Path to configuration file (not implemented yet)\n");
+        printf("  -f, --config_file     Specify a configuration file other than default\n");
        printf("  -c, --core-file       Path to core file to process\n");
        printf("  -p, --process-name    Name of process for crash report (required)\n");
        printf("  -E, --process-path    Absolute path of crashed process, with ! or / delimiters\n");
@@ -532,7 +530,11 @@ int main(int argc, char **argv)
                                printf(PACKAGE_VERSION "\n");
                                goto success;
                        case 'f':
-                                config_file = strdup(optarg);
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
+                                                  " path not valid\n");
+                                    exit(EXIT_FAILURE);
+                                }
                                break;
                        case 'c':
                                core_file = strdup(optarg);
@@ -606,8 +608,6 @@ int main(int argc, char **argv)

                backtrace = nc_string_dup("Crash from Clear package build\n");

-                keep_core = true;
-
                if (!send_data(&backtrace, unknown_severity, clr_build_class)) {
                        goto fail;
                }
@@ -619,8 +619,6 @@ int main(int argc, char **argv)

                backtrace = nc_string_dup("Crash from third party\n");

-                keep_core = true;
-
                if (!send_data(&backtrace, unknown_severity, unknown_class)) {
                        goto fail;
                }
@@ -673,11 +671,6 @@ success:

        ret = EXIT_SUCCESS;
 fail:
-        // Do not remove the core file if any errors occur
-        if (ret == EXIT_FAILURE) {
-                keep_core = true;
-        }
-
        free(core_file);
        free(proc_name);
        free(proc_path);
@@ -707,8 +700,8 @@ fail:
        }

        // Remove the core file by default, except when the --core-file option
-        // is specified, or when keep_core is overridden to true.
-        if (!core_file && !keep_core) {
+        // is specified.
+        if (!core_file) {
                unlink(temp_core);
        }

@@ -1,7 +1,7 @@
 /*
 * This program is part of the Clear Linux Project
 *
- * Copyright 2015 Intel Corporation
+ * Copyright 2015-2019 Intel Corporation
 *
 * This program is free software; you can redistribute it and/or modify it under
 * the terms and conditions of the GNU Lesser General Public License, as
@@ -20,63 +20,156 @@
 #include <string.h>
 #include <sys/stat.h>
 #include <getopt.h>
+#include <locale.h>
+#include <sys/sysinfo.h>
+#include <dirent.h>
+#include "common.h"
 #include "telemetry.h"
-
 #include "config.h"
+#include "log.h"

-void print_usage(char *prog)
+#define PAYLOAD_LOCALE  0x01
+#define PAYLOAD_UPTIME  0x02
+#define PAYLOAD_BUNDLES 0x04
+
+static void print_usage(char *prog)
 {
        printf("%s: Usage\n", prog);
        printf("  -f,  --config_file    Configuration file. This overides the other parameters\n");
        printf("  -H,  --heartbeat      Create a \"heartbeat\" record instead\n");
        printf("  -h,  --help           Display this help message\n");
+        printf("  -l,  --locale         Include locale in the record\n");
+        printf("  -u,  --uptime         Include uptime in the record\n");
+        printf("  -b,  --bundles        Include installed bundles in the record\n");
        printf("  -V,  --version        Print the program version\n");
 }

+static long get_uptime(void)
+{
+        struct sysinfo s_info;
+        if (sysinfo(&s_info) == 0) {
+                return s_info.uptime;
+        }
+        return 0;
+}
+
+static int nodots(const struct dirent *dp)
+{
+        return (dp->d_name[0] != '.');
+}
+
+static char *create_payload(unsigned options)
+{
+        int n, buflen;
+        char *ppos, *payload;
+
+        payload = calloc(sizeof(char), MAX_PAYLOAD_LENGTH);
+        if (payload == NULL) {
+                return NULL;
+        }
+
+        ppos = payload;
+        buflen = MAX_PAYLOAD_LENGTH;
+
+        n = snprintf(ppos, (size_t)buflen, "hello\n");
+        buflen -= n, ppos += n;
+
+        if (options & PAYLOAD_LOCALE) {
+                n = snprintf(ppos, (size_t)buflen, "LC_ALL: %s\n", setlocale(LC_ALL, NULL));
+                buflen -= n, ppos += n;
+        }
+
+        if (options & PAYLOAD_UPTIME) {
+                n = snprintf(ppos, (size_t)buflen, "uptime: %ld\n", get_uptime());
+                buflen -= n, ppos += n;
+        }
+
+        if (options & PAYLOAD_BUNDLES) {
+                int num_bundles, i;
+                struct dirent **entries;
+
+                num_bundles = scandir("/usr/share/clear/bundles", &entries, nodots, alphasort);
+                if (num_bundles < 0) {
+                        telem_log(LOG_ERR, "scandir failed: %s\n", strerror(errno));
+                        free(payload);
+                        return NULL;
+                }
+                n = snprintf(ppos, (size_t)buflen, "\nBundles (%d):\n", num_bundles);
+                buflen -= n, ppos += n;
+
+                for (i = 0; i < num_bundles; i++) {
+                        n = snprintf(ppos, (size_t)buflen, "%s\n", entries[i]->d_name);
+                        /* Test if truncated output */
+                        if (n >= buflen) {
+                                int lastix = MAX_PAYLOAD_LENGTH-1;
+                                payload[lastix] = '\0';
+                                payload[lastix-1] = '.';
+                                payload[lastix-2] = '.';
+                                payload[lastix-3] = '.';
+                                payload[lastix-4] = '\n';
+                                break;
+                        }
+                        buflen -= n, ppos += n;
+                }
+
+                for (i= 0; i < num_bundles; i++) {
+                        free(entries[i]);
+                }
+
+                free(entries);
+        }
+
+        return payload;
+}
+
 int main(int argc, char **argv)
 {
        uint32_t severity = 1;
        uint32_t payload_version = 1;
        char classification[30] = "org.clearlinux/hello/world";
        struct telem_ref *tm_handle = NULL;
-        char *payload2;
+        char *payload;
        char *str;
+        int ret;
+        unsigned payload_options = 0;

        // Following vars are for arg parsing.
        int c;
-        char *cfile = NULL;
        int opt_index = 0;
-        int ret = 0;

        struct option opts[] = {
                { "config_file", 1, NULL, 'f' },
                { "heartbeat", 0, NULL, 'H' },
                { "help", 0, NULL, 'h' },
                { "version", 0, NULL, 'V' },
+                { "locale", 0, NULL, 'l' },
+                { "uptime", 0, NULL, 'u' },
+                { "bundles", 0, NULL, 'b' },
                { NULL, 0, NULL, 0 }
        };

-        while ((c = getopt_long(argc, argv, "f:hHV", opts, &opt_index)) != -1) {
+        while ((c = getopt_long(argc, argv, "f:hHVlub", opts, &opt_index)) != -1) {
                switch (c) {
                        case 'f':
-                                if (asprintf(&cfile, "%s", (char *)optarg) < 0) {
-                                        exit(EXIT_FAILURE);
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
+                                                  " path not valid\n");
+                                    exit(EXIT_FAILURE);
                                }
-
-                                struct stat buf;
-                                ret = stat(cfile, &buf);
-
-                                /* check if the file exists and is a regular file */
-                                if (ret == -1 || !S_ISREG(buf.st_mode)) {
-                                        exit(EXIT_FAILURE);
-                                }
-
-                                tm_set_config_file(cfile);
                                break;
                        case 'H':
                                str = "org.clearlinux/heartbeat/ping";
                                memcpy(classification, str, strlen(str));
                                break;
+                        case 'l':
+                                payload_options |= PAYLOAD_LOCALE;
+                                break;
+                        case 'u':
+                                payload_options |= PAYLOAD_UPTIME;
+                                break;
+                        case 'b':
+                                payload_options |= PAYLOAD_BUNDLES;
+                                break;
                        case 'h':
                                print_usage(argv[0]);
                                exit(EXIT_SUCCESS);
@@ -88,34 +181,39 @@ int main(int argc, char **argv)
                }
        }

-        if (asprintf(&payload2, "hello\n") < 0) {
+        setlocale(LC_ALL, "");
+        payload = create_payload(payload_options);
+        if (payload == NULL) {
                exit(EXIT_FAILURE);
        }

        if ((ret = tm_create_record(&tm_handle, severity, classification,
                                    payload_version)) < 0) {
-                printf("Failed to create record: %s\n", strerror(-ret));
-                ret = 1;
-                goto fail;
+                telem_log(LOG_ERR, "Failed to create record: %s\n",
+                       strerror(-ret));
+                ret = EXIT_FAILURE;
+                goto done;
        }

-        if ((ret = tm_set_payload(tm_handle, payload2)) < 0) {
-                printf("Failed to set record payload: %s\n", strerror(-ret));
-                ret = 1;
-                goto fail;
+        if ((ret = tm_set_payload(tm_handle, payload)) < 0) {
+                telem_log(LOG_ERR, "Failed to set record payload: %s\n",
+                        strerror(-ret));
+                ret = EXIT_FAILURE;
+                goto done;
        }

-        free(payload2);
+        free(payload);

        if ((ret = tm_send_record(tm_handle)) < 0) {
-                printf("Failed to send record to daemon: %s\n", strerror(-ret));
-                ret = 1;
-                goto fail;
+                telem_log(LOG_ERR, "Failed to send record to daemon: %s\n",
+                        strerror(-ret));
+                ret = EXIT_FAILURE;
+                goto done;
        } else {
                printf("Successfully sent record to daemon.\n");
-                ret = 0;
+                ret = EXIT_SUCCESS;
        }
-fail:
+done:
        tm_free_record(tm_handle);
        tm_handle = NULL;

@@ -265,7 +265,7 @@ static bool add_filters(sd_journal *journal)
        return true;
 }

-static bool process_journal(void)
+static bool process_journal(bool process_existing)
 {
        int r;

@@ -279,8 +279,14 @@ static bool process_journal(void)
                return false;
        }

-        if (!process_existing_entries(journal)) {
-                return false;
+        if (process_existing) {
+                if (!process_existing_entries(journal)) {
+                        return false;
+                }
+        } else {
+                if ((sd_journal_seek_tail(journal) < 0) || (sd_journal_previous(journal) < 0)) {
+                        return false;
+                }
        }

        struct pollfd pfd;
@@ -312,12 +318,11 @@ static bool process_journal(void)
        }
 }

-static char *config_file = NULL;
-
 static const struct option prog_opts[] = {
        { "help", no_argument, 0, 'h' },
        { "config-file", required_argument, 0, 'f' },
        { "version", no_argument, 0, 'V' },
+        { "tail", no_argument, 0, 't' },
        { 0, 0, 0, 0 }
 };

@@ -330,22 +335,19 @@ static void print_help(void)
        printf("  -h, --help            Show help options\n");
        printf("\n");
        printf("Application Options:\n");
-        printf("  -f, --config-file     Path to configuration file (not implemented yet)\n");
+        printf("  -f, --config_file     Specify a configuration file other than default\n");
+        printf("  -t, --tail            Don't process entries already in the journal\n");
        printf("  -V, --version         Print the program version\n");
        printf("\n");
 }

-static void free_strings(void)
-{
-        free(config_file);
-}
-
 int main(int argc, char **argv)
 {
        int ret = EXIT_FAILURE;
        int opt;
+        bool process_existing = true;

-        while ((opt = getopt_long(argc, argv, "hf:V", prog_opts, NULL)) != -1) {
+        while ((opt = getopt_long(argc, argv, "hf:Vt", prog_opts, NULL)) != -1) {
                switch (opt) {
                        case 'h':
                                print_help();
@@ -354,20 +356,25 @@ int main(int argc, char **argv)
                                printf(PACKAGE_VERSION "\n");
                                goto success;
                        case 'f':
-                                config_file = strdup(optarg);
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
+                                                  " path not valid\n");
+                                    exit(EXIT_FAILURE);
+                                }
+                                break;
+                        case 't':
+                                process_existing = false;
                                break;
                }
        }

-        if (!process_journal()) {
+        if (!process_journal(process_existing)) {
                goto fail;
        }

 success:
        ret = EXIT_SUCCESS;
 fail:
-        free_strings();
-
        if (journal) {
                sd_journal_close(journal);
        }
@@ -1,9 +1,24 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright © 2015-2019 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
 #define _GNU_SOURCE

 #include <stdlib.h>
 #include <stdio.h>
 #include <sys/klog.h>
-#include <fcntl.h>
 #include <errno.h>
 #include <unistd.h>
 #include <malloc.h>
@@ -22,54 +37,48 @@ int main(void)
        int log_size = 0;
        char *bufp = NULL;
        size_t buflen = 0;
-        int bytes = 0;

-        oops_parser_init(write_oops_to_file);
+        oops_parser_init(klog_process_oops_msgs);

-        // Allocate space to write the oops file to
-        if (allocate_filespace() < 0) {
-                telem_log(LOG_ERR, "%s\n", strerror(errno));
-                return 1;
-        }
-
-        // Signal Handling to close and unlink empty files
-        if (signal (SIGINT, signal_handler) == SIG_ERR) {
+        // Signal Handling to terminate the probe.
+        if (signal (SIGINT, signal_handler_fail) == SIG_ERR) {
                telem_log(LOG_ERR, "Error handling interrupt signal\n");
        }
-        if (signal (SIGTERM, signal_handler) == SIG_ERR) {
+        if (signal (SIGTERM, signal_handler_success) == SIG_ERR) {
                telem_log(LOG_ERR, "Error handling terminating signal\n");
        }

+        // Gets the size of the kernel ring buffer
+        log_size = klogctl(SYSLOG_ACTION_SIZE_BUFFER, NULL, 0);
+        if (log_size < 0) {
+                telem_log(LOG_ERR, "Cannot read size of kernel ring buffer: %s\n", strerror(errno));
+                return 1;
+        }
+
+        buflen = (size_t)log_size;
+        if (buflen > MAX_BUF) {
+                buflen = MAX_BUF;
+        }
+
+        // Gets the contents of the kernel ring buffer
+        bufp = (char *)malloc(buflen);
+
        while (1) {
-                // Gets the size of the kernel ring buffer
-                log_size = klogctl(SYSLOG_ACTION_SIZE_BUFFER, NULL, 0);
-                if (log_size < 0) {
-                        telem_log(LOG_ERR, "Cannot read size of kernel ring buffer: %s\n", strerror(errno));
-                        return 1;
-                }
-
-                buflen = (size_t)log_size;
-
-                if (buflen > MAX_BUF) {
-                        buflen = MAX_BUF;
-                }
-
-                // Gets the contents of the kernel ring buffer
-                bufp = (char *)calloc(buflen, sizeof(char));
-
+                int bytes_read;
+ 
                malloc_trim(0);
-                bytes = klogctl(SYSLOG_ACTION_READ, bufp, (int)buflen);
-                if (bytes < 0) {
+                memset(bufp, 0, buflen);
+                bytes_read = klogctl(SYSLOG_ACTION_READ, bufp, (int)buflen);
+                if (bytes_read < 0) {
                        telem_log(LOG_ERR, "Cannot read contents of kernel ring buffer: %s\n", strerror(errno));
                        return 1;
                }

-                // Splits the buffer into separate lines with /0 terminating
-                split_buf_by_line(bufp, (size_t)bytes);
-
-                free(bufp);
+                klog_process_buffer(bufp, (size_t)bytes_read);
        }

+        // Not reached
+        free(bufp);
        return 0;
 }

@@ -1,7 +1,7 @@
 /*
 * This program is part of the Clear Linux Project
 *
- * Copyright 2015 Intel Corporation
+ * Copyright © 2015-2019 Intel Corporation
 *
 * This program is free software; you can redistribute it and/or modify it under
 * the terms and conditions of the GNU Lesser General Public License, as
@@ -15,11 +15,10 @@
 */

 #define _GNU_SOURCE
-
 #include <stdlib.h>
 #include <stdio.h>
+#include <sys/types.h>
 #include <sys/klog.h>
-#include <fcntl.h>
 #include <errno.h>
 #include <unistd.h>
 #include <sys/stat.h>
@@ -27,20 +26,54 @@
 #include "log.h"
 #include "oops_parser.h"
 #include "klog_scanner.h"
+#include "telemetry.h"
+#include "nica/nc-string.h"

-#define MAX_OOPS_MSG_LENGTH 1024
-#define FILE_PATH KERNELOOPSDIR "/kerneloopsXXXXXX"
+static bool oops_processed = false;

-static int fd;
-char filename[] = FILE_PATH;
-int last_line_length;
+static uint32_t version = 1;
+
+static bool send_data(char *backtrace, char *class, uint32_t severity)
+{
+        struct telem_ref *handle = NULL;
+        int ret;
+
+        if ((ret = tm_create_record(&handle, severity, class, version)) < 0) {
+                telem_log(LOG_ERR, "Failed to create record: %s",
+                          strerror(-ret));
+                return false;
+        }
+
+        if ((ret = tm_set_payload(handle, backtrace)) < 0) {
+                telem_log(LOG_ERR, "Failed to set payload: %s", strerror(-ret));
+                tm_free_record(handle);
+                return false;
+        }
+
+        if ((ret = tm_send_record(handle)) < 0) {
+                telem_log(LOG_ERR, "Failed to send record: %s", strerror(-ret));
+                tm_free_record(handle);
+                return false;
+        }
+
+        tm_free_record(handle);
+
+        return true;
+}
+
+int klog_process_buffer(char *bufp, int bytes)
+{
+        // Splits the buffer into separate lines with /0 terminating
+        split_buf_by_line(bufp, (size_t)bytes);
+        return 0;
+}

-bool oops_written = false;
 void split_buf_by_line(char *bufp, int bytes)
 {
        char *start;
        char *eol;
        size_t linelength = 0;
+        oops_processed = false;

        while (bytes > 0) {
                eol = memchr(bufp, '\n', (size_t)bytes);
@@ -57,87 +90,80 @@ void split_buf_by_line(char *bufp, int bytes)
                if (bytes >= 0) {
                        start[linelength] = '\0';
                        parse_single_line(start, linelength);
+                        if (oops_processed == true) {
+                        #ifdef DEBUG
+                            printf("[%s] oops_processed detected !\n", __func__);
+                        #endif
+                            break;
+                        }
                        linelength = 0;
                }
        }
 }

-int allocate_filespace()
+void klog_process_oops_msgs(struct oops_log_msg *msg)
 {
-        int allocate_disk = 0;
-        strcpy(filename, FILE_PATH);
-
-        fd = mkstemp(filename);
-        if (fchmod(fd, 0644) == -1) {
-                telem_log(LOG_ERR, "Error while changing permissions for file %s: %s\n",
-                          filename, strerror(errno));
-        }
-
-        if (fd == -1) {
-                telem_log(LOG_ERR, "fd error: %s\n", strerror(errno));
-                return -1;
-        }
-
-        // Allocates disk space to write kernel oops to
-        allocate_disk = fallocate(fd, 0, 0, MAX_OOPS_MSG_LENGTH);
-        if (allocate_disk != 0) {
-                telem_log(LOG_ERR, "Error allocating disk space: %s\n", strerror(errno));
-                return -1;
-        }
-        return 0;
-}
-
-void write_oops_to_file(struct oops_log_msg *msg)
-{
-        ssize_t writer;
-        size_t linelength;
+        size_t linelength, size;
        char *line;
+        char *contents = NULL;
+        nc_string *payload;

+        // pass 1: calculate buffer size
+        size = 0;
+        for (int i = 0; i < msg->length; i++) {
+                size += strlen(msg->lines[i]) + 1;
+        }
+
+        contents = malloc(size);
+        if (!contents) {
+                telem_log(LOG_ERR, "Call to malloc failed");
+                oops_processed = true;
+                return;
+        }
+
+        // pass 2: copy strings into the buffer
+        size_t done = 0;
+        char *bp = contents;
        for (int i = 0; i < msg->length; i++) {
                //Add the newline character to the end of each line
                line = msg->lines[i];
                linelength = strlen(line);
-                line[linelength] = '\n';
-                linelength++;
+                strncpy(bp, line, linelength);
+                bp[linelength] = '\n';
+                done += linelength + 1;
+                bp = contents + done;
+        }

-                // Write each line of oops to file
-                writer = write(fd, line, linelength);
-                if (writer == -1) {
-                        telem_log(LOG_ERR, "Error writing buffer to file: %s\n", strerror(errno));
-                        exit(EXIT_FAILURE);
+        struct oops_log_msg oops_msg;
+        if (handle_entire_oops(contents, (long)size, &oops_msg)) {
+            #ifdef DEBUG
+                printf("Raw message:\n");
+                for (int i = 0; i < oops_msg.length; i++) {
+                        printf("%s\n", oops_msg.lines[i]);
                }
+            #endif
+                payload = parse_payload(&oops_msg);
+            #ifdef DEBUG
+                printf("Payload Parsed :%s\n", payload->str);
+            #endif
+                oops_msg_cleanup(&oops_msg);
+                send_data(payload->str, (char *)oops_msg.pattern->classification, (uint32_t)oops_msg.pattern->severity);
+                nc_string_free(payload);
+        } else {
+                /* File does no contain an oops! */
+                telem_log(LOG_ERR, "Did not find an oops in the oops directory\n");
        }

-        telem_log(LOG_INFO, "Oops written to file\n");
-        oops_written = true;
-        close(fd);
-
-        // Allocate more space to prepare for another oops
-        if (allocate_filespace() < 0) {
-                telem_log(LOG_ERR, "%s\n", strerror(errno));
-                exit(EXIT_FAILURE);
-        }
+        free(contents);
+        oops_processed = true;
 }

-bool get_oops_written(void)
+void signal_handler_success(int signum)
 {
-        return oops_written;
+        exit(EXIT_SUCCESS);
 }

-char *get_file_name(void)
-{
-        return filename;
-}
-
-__attribute__((destructor))
-void clean_up(void)
-{
-        // Unlink empty file and close any open file
-        unlink(filename);
-        close(fd);
-}
-
-void signal_handler(int signum)
+void signal_handler_fail(int signum)
 {
        exit(EXIT_FAILURE);
 }
@@ -1,7 +1,7 @@
 /*
 * This program is part of the Clear Linux Project
 *
- * Copyright 2015 Intel Corporation
+ * Copyright © 2015-2019 Intel Corporation
 *
 * This program is free software; you can redistribute it and/or modify it under
 * the terms and conditions of the GNU Lesser General Public License, as
@@ -17,45 +17,41 @@
 #pragma once
 #include "oops_parser.h"

+/**
+ * Process the buffer and send it to the backend
+ *
+ * @param *bufp A pointer to a buffer containing the bytes read
+ *		from the kernel ring buffer
+ * @param bytes Number of bytes read from ring buffer into bufp
+ * 
+ * @return 0 on succes, -1 on failure
+ *
+ */
+int klog_process_buffer(char *bufp, int bytes);
 /**
 * Splits the buffer into individual lines
 *
 * @param *bufp A pointer to a buffer containing the bytes read
 *		from the kernel ring buffer
- * @param bytes Bytes read from ring buffer into bufp
+ * @param bytes Number of bytes read from ring buffer into bufp
 *
 */
 void split_buf_by_line(char *bufp, int bytes);

 /**
- * Allocates memory for oops message to write to file
- *
- * @return 0 on succes, -1 on failure
- */
-int allocate_filespace(void);
-
-/**
- * Writes the oops message to tmp file
+ * Process the oops message and send it to the backend
 *
 * @param msg Struct that contains oops message lines and linecount
 *
 */
-void write_oops_to_file(struct oops_log_msg *msg);
-
+void klog_process_oops_msgs(struct oops_log_msg *msg);
 /**
 * Signal handler for cleanup purposes
 *
 * @param signum An integer for the signal type
 *
 */
-void signal_handler(int signum);
-
-/**
- * For Unit Tests Purposes
- * Determines whether an oops has been written to a file
- *
- * @return true if oops is written, false if not
- */
-bool get_oops_written(void);
+void signal_handler_fail(int signum);
+void signal_handler_success(int signum);

 /* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
@@ -4,10 +4,16 @@ bin_PROGRAMS += \
 	%D%/telem-record-gen \
 	%D%/klogscanner \
 	%D%/pstoreprobe \
-	%D%/oopsprobe \
+	%D%/pythonprobe \
 	%D%/pstoreclean \
 	%D%/bertprobe

+%C%_pythonprobe_SOURCES = %D%/python-probe.c
+%C%_pythonprobe_LDADD = $(top_builddir)/src/libtelemetry.la
+%C%_pythonprobe_LDFLAGS = \
+	$(AM_LDFLAGS) \
+	-pie
+
 %C%_hprobe_SOURCES = %D%/hello.c
 %C%_hprobe_LDADD = $(top_builddir)/src/libtelemetry.la
 %C%_hprobe_LDFLAGS = \
@@ -122,25 +128,6 @@ if HAVE_SYSTEMD_JOURNAL
 endif
 endif

-%C%_oopsprobe_SOURCES = \
-        %D%/oops_probe.c \
-	%D%/oops_parser.c \
-	src/nica/nc-string.c \
-	%D%/probe.h
-%C%_oopsprobe_CFLAGS = \
-        $(AM_CFLAGS)
-%C%_oopsprobe_LDADD = \
-        $(top_builddir)/src/libtelemetry.la \
-        $(top_builddir)/src/libtelem-shared.la
-
-if LOG_SYSTEMD
-if HAVE_SYSTEMD_JOURNAL
-%C%_oopsprobe_CFLAGS += \
-        $(SYSTEMD_JOURNAL_CFLAGS)
-%C%_oopsprobe_LDADD += \
-        $(SYSTEMD_JOURNAL_LIBS)
-endif
-endif

 if HAVE_SYSTEMD_JOURNAL
 if HAVE_SYSTEMD_ID128
@@ -1,7 +1,7 @@
 /*
 * This program is part of the Clear Linux Project
 *
- * Copyright 2015 Intel Corporation
+ * Copyright 2015-2019 Intel Corporation
 *
 * This program is free software; you can redistribute it and/or modify it under
 * the terms and conditions of the GNU Lesser General Public License, as
@@ -152,9 +152,9 @@ struct oops_pattern oops_patterns_arr[] = {
        },
 };

-int oops_patterns_cnt = sizeof(oops_patterns_arr) / sizeof(struct oops_pattern);
+static int oops_patterns_cnt = sizeof(oops_patterns_arr) / sizeof(struct oops_pattern);

-char *skip_log_level(char *line)
+static char *skip_log_level(char *line)
 {
        char *start = line;

@@ -169,7 +169,7 @@ char *skip_log_level(char *line)
        return start;
 }

-char *skip_timestamp(char *line)
+static char *skip_timestamp(char *line)
 {
        char *start = line;

@@ -184,7 +184,7 @@ char *skip_timestamp(char *line)
        return start;
 }

-char *skip_space(char *line)
+static char *skip_space(char *line)
 {
        char *start = line;

@@ -194,7 +194,7 @@ char *skip_space(char *line)
        return start;
 }

-char *skip_spaces(char *line)
+static char *skip_spaces(char *line)
 {
        char *start = line;

@@ -204,7 +204,7 @@ char *skip_spaces(char *line)
        return start;
 }

-bool starts_with(const char *line, const char *line_end, const char *substr)
+static bool starts_with(const char *line, const char *line_end, const char *substr)
 {
        size_t substrlen = strlen(substr);

@@ -214,7 +214,7 @@ bool starts_with(const char *line, const char *line_end, const char *substr)
        return false;
 }

-bool str_starts_with_casei(const char *line, const char *substr)
+static bool str_starts_with_casei(const char *line, const char *substr)
 {
        size_t len = strlen(substr);

@@ -224,7 +224,7 @@ bool str_starts_with_casei(const char *line, const char *substr)
        return false;
 }

-bool pattern_matches(char *line, char *line_end, struct oops_pattern *pattern)
+static bool pattern_matches(char *line, char *line_end, struct oops_pattern *pattern)
 {
        if (pattern->is_regex) {
                return (regexec(&(pattern->regex), line, 0, NULL, 0) == 0);
@@ -233,7 +233,7 @@ bool pattern_matches(char *line, char *line_end, struct oops_pattern *pattern)
        }
 }

-void init_pattern_regex(void)
+static void init_pattern_regex(void)
 {
        for (int i = 0; i < oops_patterns_cnt; i++) {
                struct oops_pattern *pattern = &oops_patterns_arr[i];
@@ -244,7 +244,7 @@ void init_pattern_regex(void)
        }
 }

-void free_pattern_regex(void)
+static void free_pattern_regex(void)
 {
        for (int i = 0; i < oops_patterns_cnt; i++) {
                struct oops_pattern *pattern = &oops_patterns_arr[i];
@@ -261,7 +261,6 @@ bool handle_entire_oops(char *buf, long size, struct oops_log_msg *msg)
        int i;

        msg->length = 0;
-        init_pattern_regex();

        line_end = memchr(buf, '\n', (size_t)size);
        if (line_end == NULL) {
@@ -318,7 +317,6 @@ void oops_msg_cleanup(struct oops_log_msg *msg)
                free(msg->lines[i]);
        }
        msg->length = 0;
-        free_pattern_regex();
 }

 static struct oops_log_msg oops_msg;
@@ -333,7 +331,7 @@ void oops_parser_init(oops_handler_t handler)
        init_pattern_regex();
 }

-void handle_msg_end(void)
+static void handle_msg_end(void)
 {
        for (int i = 0; i < oops_msg.length; i++) {
                free(oops_msg.lines[i]);
@@ -426,7 +424,6 @@ void parse_single_line(char *line, size_t size)
                        }
                }
        }
-
 }

 struct stack_frame {
@@ -437,7 +434,7 @@ struct stack_frame {
        struct stack_frame *next;
 };

-void stack_frame_append(struct stack_frame **head, struct stack_frame **tail, char *start)
+static void stack_frame_append(struct stack_frame **head, struct stack_frame **tail, char *start)
 {
        /*
         * Format is:
@@ -536,7 +533,7 @@ void stack_frame_append(struct stack_frame **head, struct stack_frame **tail, ch
        frame->module = "kernel";
 }

-void stack_frame_free(struct stack_frame **head)
+static void stack_frame_free(struct stack_frame **head)
 {
        struct stack_frame *frame;

@@ -553,7 +550,7 @@ void stack_frame_free(struct stack_frame **head)
 * CPU: 2 PID: 0 Comm: swapper/2 Not tainted  3.10.4-100.fc18.x86_64 #1
 * CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.0.5-300.fc22.x86_64 #1
 */
-void parse_kernel_cpu_line(char *line, char **kernel_version, char **tainted)
+static void parse_kernel_cpu_line(char *line, char **kernel_version, char **tainted)
 {
        char *end = NULL;

@@ -626,7 +623,7 @@ static struct reg_s *reg_head = NULL;
 *
 */

-void parse_registers(char *line)
+static void parse_registers(char *line)
 {
        uint64_t value;
        struct reg_s *reg_entry = NULL;
@@ -739,7 +736,7 @@ void parse_registers(char *line)

 }

-void append_registers_to_bt(nc_string **backtrace)
+static void append_registers_to_bt(nc_string **backtrace)
 {
        struct reg_s *reg_entry = reg_head;

@@ -762,7 +759,7 @@ void append_registers_to_bt(nc_string **backtrace)
        }
 }

-nc_string *parse_backtrace(struct oops_log_msg *msg)
+static nc_string *parse_backtrace(struct oops_log_msg *msg)
 {
        struct stack_frame *head = NULL, *tail = NULL, *elem = NULL;
        //int in_stack_dump = 0;
@@ -1,229 +0,0 @@
-/*
- * This program is part of the Clear Linux Project
- *
- * Copyright 2015 Intel Corporation
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms and conditions of the GNU Lesser General Public License, as
- * published by the Free Software Foundation; either version 2.1 of the License,
- * or (at your option) any later version.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT ANY
- * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
- * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
- * details.
- */
-
-#define _GNU_SOURCE
-#include <stdio.h>
-#include <sys/types.h>
-#include <stdlib.h>
-#include <unistd.h>
-#include <errno.h>
-#include <sys/inotify.h>
-#include <poll.h>
-
-#include "oops_parser.h"
-#include "log.h"
-#include "telemetry.h"
-
-#include "nica/nc-string.h"
-
-char *oops_dir_path = KERNELOOPSDIR;
-
-static uint32_t version = 1;
-
-static bool send_data(char *backtrace, char *class, uint32_t severity)
-{
-        struct telem_ref *handle = NULL;
-        int ret;
-
-        if ((ret = tm_create_record(&handle, severity, class, version)) < 0) {
-                telem_log(LOG_ERR, "Failed to create record: %s",
-                          strerror(-ret));
-                return false;
-        }
-
-        if ((ret = tm_set_payload(handle, backtrace)) < 0) {
-                telem_log(LOG_ERR, "Failed to set payload: %s", strerror(-ret));
-                tm_free_record(handle);
-                return false;
-        }
-
-        if ((ret = tm_send_record(handle)) < 0) {
-                telem_log(LOG_ERR, "Failed to send record: %s", strerror(-ret));
-                tm_free_record(handle);
-                return false;
-        }
-
-        tm_free_record(handle);
-
-        return true;
-}
-
-void handle_oops_file(const char *fname)
-{
-        int ret;
-        FILE *fp = NULL;
-        char *filename = NULL, *contents = NULL;
-        long sz;
-        size_t size, bytes_read;
-        nc_string *payload;
-
-        ret = asprintf(&filename, "%s/%s", oops_dir_path, fname);
-        if (ret == -1) {
-                exit(EXIT_FAILURE);
-        }
-
-        #ifdef DEBUG
-        printf("File to be handled: %s\n", filename);
-        #endif
-
-        fp = fopen(filename, "r");
-        if (fp == NULL) {
-                telem_log(LOG_ERR, "Failed to open oops log file %s: %s", fname, strerror(errno));
-                goto file_end;
-        }
-
-        if (fseek(fp, 0, SEEK_END) == -1) {
-                telem_log(LOG_ERR, "Failed to seek file %s: %s\n", fname, strerror(errno));
-                goto file_end;
-        }
-
-        sz = ftell(fp);
-        if (sz == -1L) {
-                telem_perror("ftell failed");
-                goto file_end;
-        }
-
-        /* If file is empty */
-        if (sz == 0L) {
-                telem_perror("File is empty\n");
-                goto file_end;
-        }
-
-        if (fseek(fp, 0, SEEK_SET) < 0) {
-                telem_log(LOG_ERR, "Fseek failed for file %s:%s\n", fname, strerror(errno));
-                goto  file_end;
-        }
-
-        size = (size_t)sz;
-        contents = malloc(size);
-        if (!contents) {
-                telem_log(LOG_ERR, "Call to malloc failed");
-                goto file_end;
-        }
-        memset(contents, 0, size);
-
-        bytes_read = fread(contents, sizeof(char), size, fp);
-
-        if (bytes_read < size) {
-                telem_log(LOG_ERR, "Error while reading file %s:%s\n", fname, strerror(errno));
-                goto file_end;
-        }
-
-        struct oops_log_msg oops_msg;
-        if (handle_entire_oops(contents, (long)size, &oops_msg)) {
-            #ifdef DEBUG
-                printf("Raw message:\n");
-                for (int i = 0; i < oops_msg.length; i++) {
-                        printf("%s\n", oops_msg.lines[i]);
-                }
-            #endif
-
-                payload = parse_payload(&oops_msg);
-            #ifdef DEBUG
-                printf("Payload Parsed :%s\n", payload->str);
-            #endif
-
-                oops_msg_cleanup(&oops_msg);
-                send_data(payload->str, (char *)oops_msg.pattern->classification, (uint32_t)oops_msg.pattern->severity);
-                nc_string_free(payload);
-        } else {
-                /* File does no contain an oops! */
-                telem_log(LOG_ERR, "Did not find an oops in the oops directort\n");
-        }
-file_end:
-        if (contents) {
-                free(contents);
-        }
-        if (fp) {
-                fclose(fp);
-        }
-        if (unlink(filename) == -1) {
-                telem_log(LOG_ERR, "Failed to unlink %s: %s\n", filename, strerror(errno));
-        }
-        free(filename);
-}
-
-void handle_inotify_event(const struct inotify_event *event)
-{
-
-        if (!event) {
-                telem_perror("Null event received");
-                return;
-        }
-
-        if (event->len == 0) {
-                telem_perror("inotify event received with no file.");
-                return;
-        }
-
-        if (event->mask & IN_CLOSE_WRITE) {
-                if (event->mask & IN_ISDIR) {
-                        return;
-                }
-
-                #ifdef DEBUG
-                telem_log(LOG_DEBUG, "New file written to the oops location: %s\n", event->name);
-                #endif
-                handle_oops_file(event->name);
-        }
-}
-
-int main(int argc, char **argv)
-{
-        int inotify_fd;
-        char buf[4096];
-        char *ptr = NULL;
-        ssize_t len;
-        const struct inotify_event *event;
-
-        inotify_fd = inotify_init();
-        if (inotify_fd == -1) {
-                telem_log(LOG_ERR, "Failed to create inotify instance: %s\n", strerror(errno));
-                exit(EXIT_FAILURE);
-        }
-
-        if (inotify_add_watch(inotify_fd, oops_dir_path, IN_CLOSE_WRITE) == -1) {
-                telem_perror("Error adding watch to the inotify instance");
-        }
-
-        while (1) {
-                len = read(inotify_fd, buf, sizeof buf);
-                if (len <= 0) {
-                        telem_perror("error reading event");
-                        continue;
-                }
-
-                if (len < sizeof(struct inotify_event)) {
-                        telem_perror("Incomplete event received");
-                        continue;
-                }
-
-                telem_log(LOG_INFO, "Got inotify event/s\n");
-
-                /* Loop over all events in the buffer */
-                for (ptr = buf; ptr < buf + len;
-                     ptr += sizeof(struct inotify_event) + event->len) {
-
-                        event = (const struct inotify_event *)ptr;
-                        /* handle the event */
-                        handle_inotify_event(event);
-                }
-        }
-
-        return 0;
-}
-
-/* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
@@ -0,0 +1,293 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2019 Intel Corporation
+ *
+ *   This program is free software: you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation, either version 3 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *   GNU General Public License for more details.
+ *
+ *   You should have received a copy of the GNU General Public License
+ *   along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#define _GNU_SOURCE
+#include <assert.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/inotify.h>
+#include <fcntl.h>
+#include <dirent.h>
+#include <pwd.h>
+#include <grp.h>
+#include <getopt.h>
+
+#include "log.h"
+#include "telemetry.h"
+
+#define PYTHON_TELEMETRY_DIR "/var/lib/telemetry/python"
+
+/* We expect something like: 1.python-exception.3.N2Qy24 */
+static bool validate_file_name(const char *filename, uint32_t *severity,
+                               char** class, uint32_t *version)
+{
+        unsigned int ver, sev;
+        char cls[PATH_MAX] = { 0 };
+        char rnd[PATH_MAX] = { 0 };
+        char *filename_base = basename(filename);
+        char *str_class = NULL;
+
+        if (!filename_base) {
+                telem_perror("basename failed");
+                return false;
+        }
+
+        if (sscanf(filename_base,"%u.%[^]0-9.].%u.%s", &ver, cls, &sev, rnd) == 4) {
+                if (asprintf(&str_class, "org.clearlinux/python/%s", cls) < 29) {
+                        telem_log(LOG_ERR, "%s: invalid telemetry classification\n",
+                                  str_class);
+                        return false;
+                }
+                *class = str_class;
+                *version = ver;
+                *severity = sev;
+                return true;
+        }
+
+        return false;
+}
+
+static char *read_file_into_buffer(const char *filename)
+{
+        FILE *fp = NULL;
+        char *contents = NULL;
+        long sz;
+        size_t size, bytes_read;
+
+        fp = fopen(filename, "r");
+        if (fp == NULL) {
+                telem_log(LOG_ERR, "Failed to open Python telemetry file %s: %s\n",
+                          filename, strerror(errno));
+                return NULL;
+        }
+
+        if (fseek(fp, 0, SEEK_END) == -1) {
+                telem_log(LOG_ERR, "Failed to seek file %s: %s\n", filename,
+                          strerror(errno));
+                goto error_end;
+        }
+
+        sz = ftell(fp);
+        if (sz == -1L) {
+                telem_perror("ftell failed");
+                goto error_end;
+        }
+
+        if (sz == 0L) {
+                telem_perror("File is empty\n");
+                goto error_end;
+        }
+
+        if (fseek(fp, 0, SEEK_SET) < 0) {
+                telem_log(LOG_ERR, "fseek failed for file %s:%s\n", filename,
+                          strerror(errno));
+                goto error_end;
+        }
+
+        size = (size_t)sz;
+        contents = calloc(sizeof(char), size);
+        if (!contents) {
+                telem_log(LOG_ERR, "Call to calloc failed\n");
+                goto error_end;
+        }
+
+        bytes_read = fread(contents, sizeof(char), size, fp);
+
+        if (bytes_read < size) {
+                telem_log(LOG_ERR, "Error while reading file %s:%s\n", filename,
+                          strerror(errno));
+                goto error_end;
+        }
+
+        fclose(fp);
+        return contents;
+
+error_end:
+        fclose(fp);
+        if (contents) {
+                free(contents);
+        }
+        return NULL;
+}
+
+static void send_data(char *contents, uint32_t severity, char *class,
+                      uint32_t version)
+{
+        struct telem_ref *handle = NULL;
+        int ret;
+
+        if ((ret = tm_create_record(&handle, severity, class, version)) < 0) {
+                telem_log(LOG_ERR, "Failed to create record: %s\n",
+                          strerror(-ret));
+                return;
+        }
+
+        if ((ret = tm_set_payload(handle, contents)) < 0) {
+                telem_log(LOG_ERR, "Failed to set payload: %s\n",
+                          strerror(-ret));
+                tm_free_record(handle);
+                return;
+        }
+
+        if ((ret = tm_send_record(handle)) < 0) {
+                telem_log(LOG_ERR, "Failed to send record: %s\n",
+                          strerror(-ret));
+        }
+
+        tm_free_record(handle);
+}
+
+/* Send a valid file to the backend. */
+static void deliver_payload(const char *filename)
+{
+        uint32_t severity;
+        uint32_t version;
+        char *class = NULL;
+        char *contents = NULL;
+
+        if (validate_file_name(filename, &severity, &class, &version)) {
+                if ((contents = read_file_into_buffer(filename)) != NULL) {
+                        send_data(contents, severity, class, version);
+                        free(contents);
+                }
+
+                free(class);
+        }
+
+        /* Keep PYTHON_TELEMETRY_DIR empty. Delete the file. */
+        if (unlink(filename) != 0) {
+                telem_log(LOG_ERR, "Failed to unlink %s: %s\n", filename,
+                          strerror(errno));
+        }
+}
+
+static void drop_privs(void)
+{
+        uid_t euid;
+
+        euid = geteuid();
+        if (euid != 0) {
+                telem_log(LOG_DEBUG, "Not root; skipping privilege drop\n");
+                return;
+        }
+
+        struct passwd *pw;
+
+        pw = getpwnam("telemetry");
+        if (!pw) {
+                telem_log(LOG_ERR, "telemetry user not found\n");
+                exit(EXIT_FAILURE);
+        }
+
+        // The order is important here:
+        // change supplemental groups, our gid, and then our uid
+        if (initgroups(pw->pw_name, pw->pw_gid) != 0) {
+                telem_perror("Failed to set supplemental group list");
+                exit(EXIT_FAILURE);
+        }
+        if (setgid(pw->pw_gid) != 0) {
+                telem_perror("Failed to set GID");
+                exit(EXIT_FAILURE);
+        }
+        if (setuid(pw->pw_uid) != 0) {
+                telem_perror("Failed to set UID");
+                exit(EXIT_FAILURE);
+        }
+
+        assert(getuid() == pw->pw_uid);
+        assert(geteuid() == pw->pw_uid);
+        assert(getgid() == pw->pw_gid);
+        assert(getegid() == pw->pw_gid);
+}
+
+static void print_usage(char *prog)
+{
+        printf("%s: Usage\n", prog);
+        printf("  -f,  --config_file    Specify a configuration file other than default\n");
+        printf("  -h,  --help           Display this help message\n");
+        printf("  -V,  --version        Print the program version\n");
+}
+
+int main(int argc, char **argv)
+{
+        DIR *dir;
+        struct dirent *entry;
+        int c;
+        int opt_index = 0;
+
+        struct option opts[] = {
+                { "config_file", 1, NULL, 'f' },
+                { "help", 0, NULL, 'h' },
+                { "version", 0, NULL, 'V' },
+                { NULL, 0, NULL, 0 }
+        };
+
+        drop_privs();
+
+        while ((c = getopt_long(argc, argv, "f:hV", opts, &opt_index)) != -1) {
+                switch (c) {
+                        case 'f':
+                                if (tm_set_config_file(optarg) != 0) {
+                                    telem_log(LOG_ERR, "Configuration file"
+                                                  " path not valid\n");
+                                    exit(EXIT_FAILURE);
+                                }
+                                break;
+                        case 'h':
+                                print_usage(argv[0]);
+                                exit(EXIT_SUCCESS);
+                        case 'V':
+                                printf(PACKAGE_VERSION "\n");
+                                exit(EXIT_SUCCESS);
+                        case '?':
+                                exit(EXIT_FAILURE);
+                }
+        }
+
+        if (chdir(PYTHON_TELEMETRY_DIR)) {
+                perror(PYTHON_TELEMETRY_DIR);
+                exit(EXIT_FAILURE);
+        }
+
+        /* Process all existing files */
+        dir = opendir(PYTHON_TELEMETRY_DIR);
+        if (!dir) {
+                perror(PYTHON_TELEMETRY_DIR);
+                exit(EXIT_FAILURE);
+        }
+
+        while (true) {
+                entry = readdir(dir);
+                if (!entry) {
+                        break;
+                }
+                if (entry->d_name[0] == '.') {
+                        continue;
+                }
+                deliver_payload(entry->d_name);
+        }
+
+        closedir(dir);
+        exit(EXIT_SUCCESS);
+}
@@ -228,13 +228,13 @@ int allocate_payload_buffer(char **payload)
 {
        int ret = 0;

-        *payload = (char *)malloc(MAX_PAYLOAD_SIZE);
+        *payload = (char *)malloc(MAX_PAYLOAD_LENGTH);

        if (*payload == NULL) {
                goto out1;
        }

-        *payload = memset(*payload, 0, MAX_PAYLOAD_SIZE);
+        *payload = memset(*payload, 0, MAX_PAYLOAD_LENGTH);
        ret = 1;

 out1:
@@ -256,7 +256,7 @@ int get_payload_from_file(char **payload)
        }

        bytes_in = fread(*payload, 1,
-                         MAX_PAYLOAD_SIZE - 1, fp);
+                         MAX_PAYLOAD_LENGTH - 1, fp);

        /* if fread fails */
        if (bytes_in == 0 && ferror(fp) != 0) {
@@ -277,8 +277,8 @@ void get_payload_from_opt(char **payload)
        size_t len = 0;

        len = strlen(opt_payload);
-        if (len >= MAX_PAYLOAD_SIZE) {
-                len = MAX_PAYLOAD_SIZE - 1;
+        if (len >= MAX_PAYLOAD_LENGTH) {
+                len = MAX_PAYLOAD_LENGTH - 1;
        }
        strncpy(*payload, opt_payload, len);
 }
@@ -288,9 +288,9 @@ void get_payload_from_stdin(char **payload)
        size_t bytes_in = 0;
        int c;

-        bytes_in = fread(*payload, 1, MAX_PAYLOAD_SIZE - 1, stdin);
+        bytes_in = fread(*payload, 1, MAX_PAYLOAD_LENGTH - 1, stdin);

-        if (bytes_in == MAX_PAYLOAD_SIZE - 1) {
+        if (bytes_in == MAX_PAYLOAD_LENGTH - 1) {
                /* Throw away the rest of stdin */
                while (EOF != (c = fgetc(stdin))) {
                        continue;
@@ -377,7 +377,7 @@ int print_record(char *payload)
        int ret = 0;
        int i = 0;

-        if ((ret = instanciate_record(&t_ref, payload)) != -1) {
+        if ((ret = instanciate_record(&t_ref, payload)) == 0) {
                for (i = 0; i < NUM_HEADERS; i++) {
                        fprintf(stdout, "%s", t_ref->record->headers[i]);
                }
@@ -777,9 +777,9 @@ static int set_payload_format_header(struct telem_ref *t_ref, uint32_t payload_v
        return status;
 }

-void tm_set_config_file(char *c_file)
+int tm_set_config_file(const char *c_file)
 {
-        set_config_file(c_file);
+        return set_config_file(c_file);
 }

 /**
@@ -58,11 +58,11 @@ struct telem_ref {
 /**
 * Set the configuration file name to use
 *
- * @param c_file Absolute file name of the configuration file. If the path name
- * provided is not an absolute path, or the path given cannot be successfuly
- * stat'd, then the effective configuration file is left unchanged.
+ * @param c_file Absolute file name of the configuration file.
+ * @return 0 on success, or a negative errno-style value on error. In case of 
+ *     an error return the effective configuration file is left unchanged.
 */
-void tm_set_config_file(char *c_file);
+int tm_set_config_file(const char *c_file);

 /**
 * Create a new telemetrics record
@@ -37,7 +37,7 @@
 /* spool window check */
 static bool inside_direct_spool_window(TelemPostDaemon *daemon, time_t current_time)
 {
-        return (current_time < daemon->bypass_http_post_ts + 1800) ? true : false;
+        return (current_time < daemon->bypass_http_post_ts + NETWORK_BYPASS_DURATION) ? true : false;
 }

 /* set http bypass */
@@ -275,10 +275,10 @@ bool post_record_http(char *headers[], char *body)
        if (res) {
                size_t len = strlen(errorbuf);
                if (len) {
-                        telem_log(LOG_ERR, "Failed sending record: %s%s", errorbuf,
+                        telem_log(LOG_DEBUG, "Failed sending record: %s%s", errorbuf,
                                  ((errorbuf[len - 1] != '\n') ? "\n" : ""));
                } else {
-                        telem_log(LOG_ERR, "Failed sending record: %s\n",
+                        telem_log(LOG_DEBUG, "Failed sending record: %s\n",
                                  curl_easy_strerror(res));
                }
        } else if (http_response != 201 && http_response != 200) {
@@ -396,12 +396,8 @@ static void rate_limit_checks(TelemPostDaemon *daemon, bool *record_check_passed
 }

 /* Wrapper for save local copy */
-static void apply_retention_policies(TelemPostDaemon *daemon, char *headers[], char *body)
+static void apply_retention_policies(TelemPostDaemon *daemon, char *body)
 {
-        time_t temp = time(NULL);
-
-        save_entry_to_journal(daemon, temp, headers);
-
        if (daemon->record_retention_enabled) {
                save_local_copy(daemon, body);
        }
@@ -467,7 +463,7 @@ static bool deliver_record(TelemPostDaemon *daemon, char *headers[], char *body)
        return ret;
 }

-bool process_staged_record(char *filename, TelemPostDaemon *daemon)
+bool process_staged_record(char *filename, bool is_retry, TelemPostDaemon *daemon)
 {
        int k;
        bool ret = false;
@@ -506,8 +502,13 @@ bool process_staged_record(char *filename, TelemPostDaemon *daemon)
                goto end_processing_file;
        }

-        /** Record retention **/
-        apply_retention_policies(daemon, headers, body);
+        /* Retries should not be recorded */
+        if (is_retry == false) {
+                /** Journal entry **/
+                save_entry_to_journal(daemon, current_time, headers);
+                /** Record retention **/
+                apply_retention_policies(daemon, body);
+        }

        /** Record delivery **/
        if (!daemon->record_server_delivery_enabled) {
@@ -570,20 +571,22 @@ static int directory_dot_filter(const struct dirent *entry)
        }
 }

-void staging_records_loop(TelemPostDaemon *daemon)
+int staging_records_loop(TelemPostDaemon *daemon)
 {
        int ret;
+        int processed;
        int numentries;
        struct dirent **namelist;

        numentries = scandir(spool_dir_config(), &namelist, directory_dot_filter, NULL);
+        processed = 0;

        if (numentries == 0) {
                telem_log(LOG_DEBUG, "No entries in staging\n");
-                return;
+                return numentries;
        } else if (numentries < 0) {
                telem_perror("Error while scanning staging");
-                return;
+                return numentries;
        }

        for (int i = 0; i < numentries; i++) {
@@ -595,8 +598,9 @@ void staging_records_loop(TelemPostDaemon *daemon)
                        telem_log(LOG_ERR, "Failed to allocate memory for staging record full path\n");
                        exit(EXIT_FAILURE);
                }
-                if (process_staged_record(record_path, daemon)) {
+                if (process_staged_record(record_path, true, daemon)) {
                        unlink(record_path);
+                        processed++;
                }
                free(record_path);
        }
@@ -605,24 +609,41 @@ void staging_records_loop(TelemPostDaemon *daemon)
                free(namelist[i]);
        }
        free(namelist);
+
+        return numentries - processed;
 }

 void run_daemon(TelemPostDaemon *daemon)
 {
        int ret;
+        int retry_attempt = MAX_RETRY_ATTEMPTS;
        int spool_process_time = spool_process_time_config();
        bool daemon_recycling_enabled = daemon_recycling_enabled_config();
        time_t last_spool_run_time = time(NULL);
-        time_t last_daemon_start_time = time(NULL);
+        time_t last_record_received = time(NULL);

        assert(daemon);
        assert(daemon->pollfds);
        assert(daemon->pollfds[signlfd].fd);
        assert(daemon->pollfds[watchfd].fd);

+        /* Post at boot failed initialize retries variable */
+        if (daemon->bypass_http_post_ts != 0) {
+                retry_attempt = 1;
+        }
+
        while (1) {
+                int retry_delay = spool_process_time;
                malloc_trim(0);
-                ret = poll(daemon->pollfds, NFDS, spool_process_time * 1000);
+
+                if (retry_attempt < MAX_RETRY_ATTEMPTS) {
+                        retry_delay = retry_attempt * retry_attempt;
+                        daemon->bypass_http_post_ts = 0;
+                        telem_log(LOG_INFO, "Record delivery failed will retry in %d seconds",
+                                  retry_delay);
+                }
+
+                ret = poll(daemon->pollfds, NFDS, retry_delay * 1000);
                if (ret == -1) {
                        telem_perror("Failed to poll daemon file descriptors");
                        break;
@@ -671,10 +692,11 @@ void run_daemon(TelemPostDaemon *daemon)
                                                                exit(EXIT_FAILURE);
                                                        }
                                                        /* Process inotify event */
-                                                        if (process_staged_record(record_name, daemon)) {
+                                                        if (process_staged_record(record_name, false, daemon)) {
                                                                unlink(record_name);
                                                        }
                                                        free(record_name);
+                                                        last_record_received = time(NULL);
                                                }
                                        }

@@ -685,19 +707,30 @@ void run_daemon(TelemPostDaemon *daemon)
                        time_t now = time(NULL);
                        /* time to recycle the daemon has elapsed*/
                        if (daemon_recycling_enabled &&
-                            difftime(now, last_daemon_start_time) >= TM_DAEMON_EXIT_TIME) {
+                            difftime(now, last_record_received) >= TM_DAEMON_EXIT_TIME) {
                                /* Exit */
                                telem_log(LOG_INFO, "Telemetry post daemon exiting for recycling\n");
                                break;
                        }
+                        /* Check if this is a retry */
+                        if (retry_attempt < MAX_RETRY_ATTEMPTS) {
+                                if (staging_records_loop(daemon) == 0) {
+                                        retry_attempt = MAX_RETRY_ATTEMPTS + 1;
+                                } else {
+                                        retry_attempt++;
+                                }
+                        } else if (retry_attempt == MAX_RETRY_ATTEMPTS) {
+                                retry_attempt = MAX_RETRY_ATTEMPTS + 1;
+                                telem_log(LOG_ERR, "Record deliver failed after %d attempts",
+                                          MAX_RETRY_ATTEMPTS);
+                        }
+                        /* Check spool  */
+                        if (difftime(now, last_spool_run_time) >= spool_process_time) {
+                                spool_records_loop(&(daemon->current_spool_size));
+                                last_spool_run_time = time(NULL);
+                        }
                }

-                /* Check spool  */
-                time_t now = time(NULL);
-                if (difftime(now, last_spool_run_time) >= spool_process_time) {
-                        spool_records_loop(&(daemon->current_spool_size));
-                        last_spool_run_time = time(NULL);
-                }
                /* Check journal records and prune if needed */
                ret = prune_journal(daemon->record_journal, JOURNAL_TMPDIR);
                if (ret != 0) {
@@ -19,6 +19,8 @@
 #define NFDS 2
 #define TM_RATE_LIMIT_SLOTS (1 /*h*/ * 60 /*m*/)
 #define TM_RECORD_COUNTER (1)
+#define MAX_RETRY_ATTEMPTS 8
+#define NETWORK_BYPASS_DURATION TM_DAEMON_EXIT_TIME

 #include <poll.h>
 #include <stdbool.h>
@@ -83,17 +85,20 @@ void close_daemon(TelemPostDaemon *daemon);
 * Processed record written on disk
 *
 * @param filename a pointor to record on disk
+ * @param is_retry a boolean value that indicates if
+ *        the record has been previously processed.
 * @param daemon post to telemetry post daemon
 */
-bool process_staged_record(char *filename, TelemPostDaemon *daemon);
+bool process_staged_record(char *filename, bool is_retry, TelemPostDaemon *daemon);

 /**
 * Scans staging directory to process files that were
 * missed by file watcher
 *
 * @param daemon a pointer to telemetry post daemon
+ * @return the number of records that were removed from spool
 */
-void staging_records_loop(TelemPostDaemon *daemon);
+int staging_records_loop(TelemPostDaemon *daemon);

 /**
 * Posts a record to backend
@@ -40,7 +40,7 @@ START_TEST(check_read_valid_config)
        ck_assert_str_eq(config.strValues[CONF_SPOOL_DIR], "/tmp/spool");
        ck_assert_int_eq(config.intValues[CONF_RECORD_EXPIRY], 1200);
        ck_assert_int_eq(config.intValues[CONF_SPOOL_MAX_SIZE], 1024);
-        ck_assert_int_eq(config.intValues[CONF_SPOOL_PROCESS_TIME], 900);
+        ck_assert_int_eq(config.intValues[CONF_SPOOL_PROCESS_TIME], 180);
        ck_assert_int_eq(config.intValues[CONF_RECORD_BURST_LIMIT], 100);
        ck_assert_int_eq(config.intValues[CONF_RECORD_WINDOW_LENGTH], 15);
        ck_assert_int_eq(config.intValues[CONF_BYTE_BURST_LIMIT], 1000);
@@ -94,7 +94,7 @@ START_TEST(check_config_initialised)
        ck_assert_str_eq(spool_dir_config(), "/tmp/spool");
        ck_assert_int_eq(record_expiry_config(), 1200);
        ck_assert_int_eq(spool_max_size_config(), 1024);
-        ck_assert_int_eq(spool_process_time_config(), 900);
+        ck_assert_int_eq(spool_process_time_config(), 180);
        ck_assert_int_eq(record_burst_limit_config(), 100);
        ck_assert_int_eq(record_window_length_config(), 15);
        ck_assert_int_eq(byte_burst_limit_config(), 1000);
@@ -62,7 +62,7 @@ START_TEST(check_handle_client_with_no_data)
        bool success;
        char *filename = ABSTOPSRCDIR "/tests/telempostd/empty_message";

-        success = process_staged_record(filename, &tdaemon);
+        success = process_staged_record(filename, false, &tdaemon);
        // Return true to remove corrupted record
        ck_assert(success == true);
 }
@@ -75,7 +75,7 @@ START_TEST(check_handle_client_with_incorrect_data)
        bool success;
        char *filename = ABSTOPSRCDIR "/tests/telempostd/incorrect_message";

-        success = process_staged_record(filename, &tdaemon);
+        success = process_staged_record(filename, false, &tdaemon);
        // Return true to remove corrupted record
        ck_assert(success == true);
 }
@@ -88,7 +88,7 @@ START_TEST(check_process_record_with_correct_size_and_data)
        bool success;
        char *filename = ABSTOPSRCDIR "/tests/telempostd/correct_message";

-        success = process_staged_record(filename, &tdaemon);
+        success = process_staged_record(filename, false, &tdaemon);
        ck_assert(success == true);
 }
 END_TEST
@@ -100,7 +100,7 @@ START_TEST(check_process_record_with_incorrect_headers)
        bool success;
        char *filename = ABSTOPSRCDIR "/tests/telempostd/incorrect_headers";

-        success = process_staged_record(filename, &tdaemon);
+        success = process_staged_record(filename, false, &tdaemon);
        // Return true to remove corrupted record
        ck_assert(success == true);
 }
@@ -40,27 +40,6 @@ void callback_func(struct oops_log_msg *msg)
        pl = parse_payload(msg);
 }

-void setup(char *oopsfile)
-{
-        char *buf = NULL;
-        int buflen = 0;
-
-        oops_parser_cleanup();
-
-        buf = readfile(oopsfile);
-        buflen = (int)getbuflen();
-
-        oops_parser_init(write_oops_to_file);
-
-        // Allocate space to write the oops file to
-        if (allocate_filespace() < 0) {
-                telem_log(LOG_ERR, "%s\n", strerror(errno));
-                exit(EXIT_FAILURE);
-        }
-
-        split_buf_by_line(buf, buflen);
-}
-
 void setup_payload(char *oopsfile)
 {
        char *buf = NULL;
@@ -136,6 +136,7 @@ EXTRA_DIST += \
        @CHECK_CFLAGS@
 %C%_check_probes_LDADD = \
        @CHECK_LIBS@ \
+        $(top_builddir)/src/libtelemetry.la \
        $(top_builddir)/src/libtelem-shared.la

 if LOG_SYSTEMD
Author	SHA1	Message	Date
Alex Jaramillo	191d91e8e3	Release v2.1.0	2019-02-20 18:09:47 +00:00
California Sullivan	4b777f71f7	journalprobe: don't process entire journal on restart This adds an optional argument to journalprobe that allows seeking to the end of the journal. This is then used in a new service file that is called by telemctl on restart. This allows us to scan the entire journal on boot, and only scan new journal entries when restarting, or opting into telemetry manually. This reduces spam from repeated messages and fixes a possible privacy concern, as previously running telemctl start could include journal errors that occurred before telemetrics was opted into. Signed-off-by: California Sullivan <california.l.sullivan@intel.com>	2019-02-18 09:09:39 -08:00
Juro Bystricky	e27051d1a2	heartbeat: modified payload Also report locale and uptime. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-02-18 08:59:05 -08:00
Juro Bystricky	b7cdf9255d	hello.c: additional payload content Optionally add locale, uptime and list of installed bundles to the payload. The default behavior is as before. Additional payload is can be enabled via cmd line arguments: -l : add locale info -b : add installed bundles info -u : add uptime info For consistency, always use telem_log(LOG_ERR,...) instead of fprintf(stderr,...). Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-02-07 17:11:51 -08:00
ajch	35b5d98c66	Security disclosures link Added required link for security disclosures.	2019-01-28 11:37:53 -08:00
Patrick McCarty	29917fb464	Add missing entries to gitignore Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>	2019-01-28 09:01:59 -08:00
Patrick McCarty	264f53dd47	crashprobe: always remove core files, except when passing --core-file In modern times, the default core_pattern for Clear Linux OS is set to \|/usr/lib/systemd/coredump-wrapper %E %P %u %g %s %t %c %h %e This wrapper script sends core files to both systemd-coredump and crashprobe (if installed). Because systemd-coredump will store core files in the systemd journal, there is no longer a strong reason to preserve the crashprobe core files in /tmp, aside from when crashprobe's --core-file option is used. If developers wish to access core files, they can use `coredumpctl` instead to extract them from the systemd journal. Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>	2019-01-28 09:01:59 -08:00
Juro Bystricky	0705f89533	bert_probe.c: don't return EXIT_SUCCESS in case of a failure The original code "allocate_payload_buffer" returned 0 if "malloc" failed and 1 if "malloc" succeeded. The "bertprobe" used this return value upon exit. However "bertprobe" return code 0 corresponds to EXIT_SUCCESS and return code 1 corresponds to EXIT_FALURE. Similarily, "nc_b64enc_filename" returns 1 upon success and 0 upon failure, so if the function fails, "bertprobe" exits with EXIT_SUCCESS. To fix this, simplify the code and return with -ENOMEM immediately if memory allocation fails and explicitly set return value to EXIT_FAILURE if "nc_b64enc_filename" returns 0. While at it, rename the label "fail" to "done" as this is common cleanup exit code for both EXIT_SUCCESS and EXIT_FAILURE. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-01-28 08:59:55 -08:00
Juro Bystricky	57db31a2c8	pythonprobe: don't use inotify Don't wait for new payload via inotify, let systemd path unit detect any new payloads in /usr/lib/telemetry/python and start the pythonprobe service. Also set the sticky bit for the above folder, preventing users to modify permissions. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-01-23 20:56:21 -08:00
California Sullivan	e20e572fad	telemprobd,telempostd: greatly reduce timeouts We don't need to stay resident for a static two hours. Instead, exit cleanly after five minutes idle. This also involves reducing the default and maximum values of spool_process_time, as it was previously set to 30 minutes. telempostd required several changes with timers, whereas telemprobd only required changing the values of spool_process_time and TM_DAEMON_EXIT_TIME, and refreshing the timeout when handling a client. Signed-off-by: California Sullivan <california.l.sullivan@intel.com>	2019-01-20 21:21:50 -08:00
Juro Bystricky	6b44aa21c7	klogscanner: merge functionality with oopsprobe The previous sequence of operations was: 1. klogscanner waits for/monitors kernel buffer for messages. 2. If they come, it creates a "raw" file in /var/cache/telemetry/oops. 3. Once this file is placed into /var/cache/telemetry/oops, it is detected by oopsprobe. 4. oopsprobe parses the "raw" file, creates new payload and and sends it to the backend. This commit merges the operations: 1. klogscanner waits for/monitors kernel buffer for messages. 2. If they come, parses the klog buffer, creates new payload and and sends it to the backend. While at it, also declare routines as "static" when possible. Fixed a few missplaced memory freeing. Simplify the main loop operations, move allocate/free buffer outside of the loop. Return SUCCESS if terminated by the signal SIGTERM. This is a clean way to stop the service. This prevents spamming the journal (and sending journal/error telemetry data via journalprobe) each time telemetry is restarted. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-01-20 21:18:58 -08:00
Juro Bystricky	dce6e0ee1d	python-probe: Initial commit The purpose of the python-probe is to monitor a specific folder (as defined by PYTHON_TELEMETRY_DIR) for any Python generated reports. If any such file is detected, it is sent to the telemetry backend for analysis. The reports are typically tracebacks of unhandled Python exceptions caused by Python programs residing in /usr/bin. It is the responsibility of Python to generate the traceback files and place them in PYTHON_TELEMTRY_DIR. The probe will only detect the presence of the files and send the content of the files as a payload. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-01-15 16:05:04 -07:00
Juro Bystricky	d7c1603d12	tm_set_config_file: return status and call from all probes This change was on a TODO list for a few years. The function tm_set_config_file could silently fail as it did not return any status. This forced implementing of all error checks by the callers of this function. The current implementation does its own error checks and string copying. The function returns an error code if errors were encountered. Since this is a library function, all callers needed to be modified as well. However, the modifications are code simplifications so this is not too painful. In addition, the change also affected library documentation as well (void -> int). As a direct consequence, this change allows simple implementation of the command line argument "--config_file" uniformly across all probes. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2019-01-09 11:55:39 -07:00
Alex Jaramillo	bb6a08378c	Fix for missing restart units This change fixes a bug where the oops probe and journal probe are not restarted when telemctl restart is executed. Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2019-01-04 13:47:06 -07:00
Juro Bystricky	196721b0a1	hello.c: various tweaks 1. Use stderr to print error messages 2. Rename the label 'fail' to 'done' as even non-failing code falls through the label 3. Consistently use EXIT_FAILURE/EXIT_SUCCESS, code sometimes used 1/0 instead. 4. free memory allocated via asprintf 5. Declare print_usage as 'static' Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2018-12-21 11:51:04 -07:00
Juro Bystricky	fbc52d01bf	bert_probe.c: remove unsupported cmdline option There is no entry for 'H' command line option, yet the getopt_long accepts it as a valid option. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2018-12-21 11:17:18 -07:00
Juro Bystricky	452c4d6ef5	Update docs/README.md We switched from .md files to .rst files a while ago in v1.11.1. Since then we genarate man pages using 'rst2man.py', hence the README.md file needs to be modified to reflect this change. (/usr/bin/ronn is actually not even availabel in CLR anymore) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2018-12-21 11:16:53 -07:00
Juro Bystricky	5488d4f75a	man pages: update The .rst were updated. All man pages need to be rebuilt/updated as well. (Mostly removing/replacing "telemd" references) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2018-12-19 14:33:24 -07:00
ajch	30fe6b9de1	Adding build status badge from travis	2018-12-19 09:25:59 -08:00
Juro Bystricky	a7db38d500	Remove redundant definition MAX_PAYLOAD_SIZE The file common.h contained two definitions to specify the maximum allowable payload: define MAX_PAYLOAD_SIZE 8192 define MAX_PAYLOAD_LENGTH 8192 This patch removes the definition for MAX_PAYLOAD_SIZE and replaces all occurences of MAX_PAYLOAD_SIZE with MAX_PAYLOAD_LENGTH. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>	2018-12-19 10:20:37 -07:00
California Sullivan	4524205238	probes/telem_record_gen.c: fix possible segfault in print_record When opt-out is enabled, the instanciate_record() function will return -111 and the headers will not be allocated. Since print_record only checks that instanciate_record does not return -1 before trying to go through the headers, a segfault will occur. This can be seen via: $ sudo touch /etc/telemetrics/opt-out $ telem-record-gen -c a/b/c -o -n -p 'payload goes here' Signed-off-by: California Sullivan <california.l.sullivan@intel.com>	2018-11-30 14:33:01 -08:00
Beth Dean	a6ab9fab7e	Docs: update .rst files to remove telemd references	2018-11-15 09:03:40 -08:00
Alex Jaramillo	d8d3cdee49	Release v2.0.1 Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-09-25 15:57:03 +00:00
Alex Jaramillo	839ee5c126	Retry logic for post failure on boot This change alters the logic of post telemetry message to backend to handle cases when the network or name resolution service is not available when the daemon starts. Currently if a message is not delivered it will be spooled and it should have to wait 900 seconds (default value) for a future loop to process this message. With this change when the daemon is started and is unable to deliver telemetry messages it will atempt to re-send those messages using a t*t delay in seconds up to t = 7 (or 1, 4, 9, 16, 25, 36, 49 seconds). Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-09-24 15:18:12 -07:00