Removing crash dump configuration

This change removes the crash probe exclusive use of the core dump
reported by the kernel. Other component above telemetry will handle
this to properly report core dumps to telemetry (when available).

Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>

README.md

@@ -6,7 +6,7 @@ solution for Linux-based operating systems. Specifically, the front end
 component includes:
 
 - telemetrics probes that collect specific types of data from the operating
-  system.
+  system. For more info on probes go [here](./src/probes).
 
 - a library, libtelemetry, that telemetrics probes use to create telemetrics
   records and send them to the daemon for further processing.
@@ -166,6 +166,56 @@ The daemon uses the following configuration options from the configuration file:
   threshold has been reached. Currently the options are 'drop' or 'spool', with
   spool being the default. If spool is chosen, records will be spooled and sent
   at a later time.
+* record_retention_enabled: When this key is enabled (true) the daemon saves a
+  copy of the payload on disk from all valid records. To avoid the excessive use
+  of disk space only the latest 100 records are kept. The default value for this
+  configuration key is false.
+* record_server_delivery_enabled: This key controls the delivery of records to
+  ```server```, when enabled (default value) the record will be posted to the
+  address in the configuration file. If this configuration key is disabled (false)
+  records will not be spooled or posted to backend. This configuration key can
+  be used in combination with ```record_retention_enabled``` to keep copies of
+  telemetry records locally only.
+
+
+Data reported
+---------------------
+
+The data reported by the telemetry client could be understood as two main sets:
+metadata and a payload.
+
+The metadata is used to report details of a machine's architecture. The
+following are the metadata values currently collected (Record Format Version 4):
+
+* record_format_version: version of the record, currently is 'Version 4'. This
+  value changes when new metadata is added.
+* classification: this field is used to identify the type of record sent by a
+  specific client probe; classifications use the format DOMAIN/PROBE/REST, where
+  DOMAIN is the vendor of the probe, PROBE is the probe name, and REST is a
+  probe-defined field to classify what is contained in the payload.
+* severity: this is an integer value between 1 and 4 where 1 is "low" and 4 is
+  "critical"
+* machine_id: a machine identifier that is rotate every 3 days for privacy reasons.
+* creation_timestamp: timestamp when the record was collected.
+* arch: a string describing machine architecture i.e. 'x86_64'.
+* host_type: a string with the combination of 'System Vendor', 'Product Name', and
+  'Product Version' read from dmi file system.
+* build: OS build number.
+* kernel_version: Kernel version.
+* payload_format_version: version of the payload, currently is 'Version 1'.
+* system_name: the value after 'ID=' from '/etc/os-release' (or distribution
+  provided folder)
+* board_name: a string read from dmi file system that combines 'Board Name' and
+  'Board Vendor'.
+* cpu_model: cpu model name extracted from '/proc/cpuinfo'.
+* bios_version: BIOS version.
+* event_id: an id to group multiple records if these were generated by a single
+  event occurrence.
+
+The payload as mentioned above is reported by probes. The telemetry library adds
+the metadata to the payload (done programatically when using library API) for
+more information about probes go [here](./src/probes).
+
 
 Machine id
 ---------------------
@@ -194,3 +244,69 @@ You can switch back to the rotating machine id by deleting the override file
 and restarting the daemon.  You can do a quick test to check that your
 machine-id has changed by running "hprobe" and verifying that a record has
 landed on your backend telemetrics server, with the specified machine id.
+
+Event Id
+----------------------
+
+This is a 32 character lowercase hexadecimal string i.e. '5de9de8d5f3c6a7d445d75ba01cc3322'.
+This header is used to group multiple records by an event id. Before this header
+every single record could have been thought of an event, however this is not always
+the case. There are "events" that trigger the creation of multiple records (i.e.
+updates). The ```event_id``` header was added for probes with the capability to
+detect events and group records based on such events. This header was added to
+```telem-record-gen``` and can be specified using the ```-e``` (```--event-id```
+long form) switch.
+
+```{r, engine=bash, count_lines}
+  -e, --event-id        Event id to use in the record
+```
+
+Debugging locally with telemetrics-client
+-----------------------------------------------
+
+The function of the telemetrics-client is to handle the transport of information
+reported by a probe to a backend (see ```server``` key in configuration). This
+information is helpful for developers to debug and fix reported crashes, however
+developers not always have access to the backend in these case users can leverage
+features added for local debugging. The following is a list of steps to enable
+local debug:
+
+* *Enabling record retention*: this step configures telemd to keep
+copies of telemetry records locally. To enable record retention set the value of
+```record_retention_enabled``` from ```false``` to ```true```. Optionally set
+```record_server_delivery_enabled`` to ```false``` to keep records local only.
+Remember to restart the daemon after configuration values are updated
+(```telemctl restart```).
+
+* *Creating a record*: run ```hprobe``` command to create a record for the purposes
+of this step by step guide. Once we have the record or records that you need to
+capture locally you can display the data.
+
+* *Displaying record metadata*: telemd keeps metadata of any valid record,
+to display this data a new option to telemctl was added ```telemctl journal```.
+Assuming that the last record created was the record from previous step `hprobe` we
+can use `tail -n 1` to print the last created record only, i.e.
+
+```
+$ sudo telemctl journal | tail -n 1
+$ org.clearlinux/hello/world     Mon 2018-04-02 17:48:01 UTC a19a0d41ba16788881e274b19b8a1be4 5de9de8d5f3c6a7d445d75ba01cc3322 60c014cd-4693-40f1-b334-548cd932949b
+```
+
+The headers for the metadata (along with other information) can be printed using the
+```-V``` switch with ```telemctl journal``` command, i.e.
+
+```
+$ sudo telemctl journal -V | head -n 1
+$ Classification             Time stamp              Record ID                    Event ID                     Boot ID
+```
+
+* *Displaying record payload*: to print the content of a record payload you can use
+the ```-i``` (```--include_record``` long format) option to ```telemctl journal```
+command. To print the specific record you created you can use the option ```-r```
+(```--record_id``` long format) with the Record Id of the generated record, i.e.
+
+```
+$ sudo telemctl journal --record_id a19a0d41ba16788881e274b19b8a1be4 --include_record
+$ org.clearlinux/hello/world     Mon 2018-04-02 17:48:01 UTC a19a0d41ba16788881e274b19b8a1be4 5de9de8d5f3c6a7d445d75ba01cc3322 60c014cd-4693-40f1-b334-548cd932949b
+$ hello
+```

configure.ac

@@ -2,7 +2,7 @@
 # Process this file with autoconf to produce a configure script.
 
 AC_PREREQ([2.69])
-AC_INIT([telemetrics-client], [1.16.0], [https://clearlinux.org/])
+AC_INIT([telemetrics-client], [1.16.3], [https://clearlinux.org/])
 AC_CONFIG_AUX_DIR([build-aux])
 AM_INIT_AUTOMAKE([1.14 -Wall -Werror -Wno-extra-portability foreign subdir-objects])
 AM_SILENT_RULES([yes])
@@ -23,7 +23,7 @@ AC_PROG_MAKE_SET
 # Checks for libraries.
 
 # check >= 0.9.12 is required for TAP output
-PKG_CHECK_MODULES([CHECK], [check >= 0.9.12])
+PKG_CHECK_MODULES([CHECK], [check >= 0.12])
 PKG_CHECK_MODULES([CURL], [libcurl])
 AC_CHECK_LIB([elf], [elf_begin], [have_elflib=yes], [AC_MSG_ERROR([Unable to find libelf from elfutils])])
 AC_CHECK_LIB([dw], [dwfl_begin], [have_dwlib=yes], [AC_MSG_ERROR([Unable to find libdw from elfutils])])

src/data/40-crash-probe.conf.in

@@ -1 +0,0 @@
-kernel.core_pattern = |@bindir@/crashprobe -p %e -E %E -s %s

src/data/local.mk

@@ -9,7 +9,6 @@ pathfix = @sed \
 
 EXTRA_DIST += \
 	%D%/40-core-ulimit.conf \
-	%D%/40-crash-probe.conf.in \
 	%D%/example.conf \
 	%D%/example.1.conf \
 	%D%/hprobe.service.in \
@@ -94,12 +93,6 @@ systemdunit_DATA = \
 %D%/telemd-update-trigger.service: %D%/telemd-update-trigger.service.in
 	$(pathfix) < $< > $@
 
-sysctldir = @SYSTEMD_SYSCTLDIR@
-sysctl_DATA = %D%/40-crash-probe.conf
-
-%D%/40-crash-probe.conf: %D%/40-crash-probe.conf.in
-	$(pathfix) < $< > $@
-
 systemconfdir = @SYSTEMD_SYSTEMCONFDIR@
 systemconf_DATA = %D%/40-core-ulimit.conf
 
@@ -111,7 +104,6 @@ clean-local:
 		%D%/telemetrics.conf \
 		%D%/telemetrics-dirs.conf \
 		%D%/libtelemetry.pc \
-		%D%/40-crash-probe.conf \
 		%D%/journal-probe.service \
 		%D%/oops-probe.service \
 		%D%/pstore-probe.service \

src/journal/journal.c

@@ -344,9 +344,11 @@ TelemJournal *open_journal(const char *journal_file)
         struct TelemJournal *telem_journal;
 
         // Use default location if journal_file parameter is NULL
-        fptr = (journal_file == NULL) ? fopen(JOURNAL_PATH, "a+") :
-               fopen(journal_file, "a+");
-
+        if (journal_file == NULL) {
+                fptr = fopen(JOURNAL_PATH, "a+");
+        } else {
+                fptr = fopen(journal_file, "a+");
+        }
         if (fptr == NULL) {
 #ifdef DEBUG
                 perror("Error while opening journal file: ");

src/probes/README.md

@@ -0,0 +1,53 @@
+# Probes
+
+The default probes provided along the telemetry client code are:
+
+* bertprobe: this probe reports on the Boot Error Region Table if such
+  entry is found in ```/sys/firmware/acpi/tables/BERT```.
+
+* crashprobe: This probe processes core dump files. It can be registered as
+  the kernel core file handler in /proc/sys/kernel/core_pattern.
+
+* hprobe: a simple probe that sends a keep alive message. This probe can also
+  be useful for testing
+
+* journalprobe: a probe that monitors the systemd journal for log messages
+  from failed services.
+
+* oopsprobe: a probe to collect 'oops messages' when the kernel detects a
+  problem.
+
+* pstoreprobe: probe to collect messages left on pstore filesystem.
+
+* telem-record-gen: this is a "general-purpose probe" for sending custom
+  records on-the-fly. This tool can be used stand alone or as a part of a
+  script to implement a probe.
+
+For example to generate an hprobe like message we can execute the following
+command:
+
+```
+$ telem-record-gen --class org.example/hello/world --payload hello
+```
+
+for more options use the ```--help``` switch i.e.:
+
+```
+$ telem-record-gen --help
+Usage:
+  telem-record-gen [OPTIONS] - create and send a custom telemetry record
+
+Help Options:
+  -h, --help            Show help options
+
+Application Options:
+  -f, --config-file     Path to configuration file (not implemented yet)
+  -V, --version         Print the program version
+  -s, --severity        Severity level (1-4) - (default 1)
+  -c, --class           Classification level_1/level_2/level_3
+  -p, --payload         Record body (max size = 8k)
+  -P, --payload-file    File to read payload from
+  -R, --record-version  Version number for format of payload (default 1)
+  -e, --event-id        Event id to use in the record
+
+```

tests/check_journal.c

@@ -20,11 +20,16 @@
 #include "common.h"
 #include "journal/journal.h"
 
+static char *journal_file = "journal.txt";
+static char *journal_print_file = "journal.print.txt";
 static struct TelemJournal *journal = NULL;
-static char *journal_file = TESTOOPSDIR "/journal.txt";
 static char *eid = "00007766547776eb7fc478eb0eb43e43";
 static int K = 20;
 
+void teardown(void) {
+        remove(journal_file);
+}
+
 START_TEST(check_open_journal)
 {
         struct TelemJournal *j = open_journal(journal_file);
@@ -108,6 +113,7 @@ void new_entry_teardown(void)
                 close_journal(journal);
                 journal = NULL;
         }
+        remove(journal_file);
 }
 
 void insert_n_records(int n, struct TelemJournal *j)
@@ -128,7 +134,7 @@ START_TEST(check_journal_file_prune)
 
         insert_n_records(j->record_count_limit * 2, j);
         ck_assert_int_gt(j->record_count, j->record_count_limit);
-        rc = prune_journal(j, TESTOOPSDIR);
+        rc = prune_journal(j, "./");
         ck_assert(rc == 0);
         // Record count should always be below the limit + hysteresis
         ck_assert_int_lt(j->record_count, j->record_count_limit + DEVIATION);
@@ -139,13 +145,12 @@ END_TEST
 void journal_entry_setup(void)
 {
         int result = 0;
-        const char *journal_print = TESTOOPSDIR "/journal.print.txt";
 
         if (journal) {
                 close_journal(journal);
         }
 
-        journal = open_journal(journal_print);
+        journal = open_journal(journal_print_file);
 
         insert_n_records(K, journal);
         /* Insert recors with specific values */
@@ -188,6 +193,7 @@ END_TEST
 void journal_entry_teardown(void)
 {
         close_journal(journal);
+        remove(journal_print_file);
 }
 
 Suite *config_suite(void)
@@ -197,6 +203,7 @@ Suite *config_suite(void)
 
         // Individual unit tests are added to "test cases"
         TCase *t = tcase_create("journal");
+        tcase_add_unchecked_fixture(t, NULL, teardown);
         tcase_add_test(t, check_open_journal);
         tcase_add_test(t, check_unsuccessful_open_journal);
         suite_add_tcase(s, t);
@@ -211,6 +218,7 @@ Suite *config_suite(void)
         suite_add_tcase(s, t);
 
         t = tcase_create("prunning journal");
+        tcase_add_unchecked_fixture(t, NULL, teardown);
         tcase_add_test(t, check_journal_file_prune);
         suite_add_tcase(s, t);
 

tests/local.mk

@@ -131,7 +131,6 @@ EXTRA_DIST += \
 
 %C%_check_journal_SOURCES = \
 	%D%/check_journal.c \
-	src/journal/journal.h \
 	src/journal/journal.c \
 	src/util.h \
 	src/util.c
@@ -145,7 +144,10 @@ EXTRA_DIST += \
 
 %C%_check_libtelemetry_SOURCES = \
 	%D%/check_libtelemetry.c \
-	src/telemetry.c
+	src/configuration.h \
+	src/util.h \
+	src/nica/hashmap.h \
+	src/nica/inifile.h
 
 %C%_check_libtelemetry_CFLAGS = \
 	$(AM_CFLAGS) \