Release v2.0.1

Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>
Retry logic for post failure on boot
2026-09-01 19:21:34 +00:00 · 2018-09-25 15:57:03 +00:00 · 2018-09-24 15:18:12 -07:00 · 2018-09-06 13:33:39 -07:00 · 2018-09-06 09:57:01 -07:00 · 2018-08-27 10:40:12 -07:00
50 changed files with 2149 additions and 1041 deletions
@@ -33,7 +33,8 @@ build-aux/m4/ltversion.m4
 *.trs
 /cscope.*
 /tags
-telemd
+telemprobd
+telempostd
 autoscan.log
 configure.scan
 tprobe
@@ -11,7 +11,10 @@ component includes:
 - a library, libtelemetry, that telemetrics probes use to create telemetrics
  records and send them to the daemon for further processing.

- a daemon, telemd, that prepares the records to send to a telemetrics server
+- telemprobd, a daemon that handles communication with probes and forwards records
+  to a second daemon.
+
+- telempostd, this daemon sends telemetry records to a telemetrics server
  (not included in this source tree), or spools the records on disk in case
  it's unable to successfully deliver them.

@@ -30,7 +33,7 @@ Build dependencies
 - elfutils, which provides libelf and libdwfl libraries..

 - (optional) libsystemd, for syslog-style logging to the systemd journal, and
-  socket/path activation of telemd by systemd.
+  socket/path activation of telemprobd and telempostd by systemd.


 Build and installation
@@ -60,31 +63,38 @@ and modify the /etc version.

 Descriptions of config options are listed below in the Usage section.

-Starting the daemon
+Starting the client
 ---------------------

+If the client was compiled with systemd support the respective activation units
+should be already in place (after a ```make install``` invocation). In this case
+the client wil start automatically when data is made available to it. i.e. when
+executing an ```/usr/bin/hprobe``` command.
+
 Method 1 (recommended):

 ```{r, engine='bash', count_lines}
- systemctl start telemd.socket telemd.path
+ telemctl start
 ```

 Note: the above invocation technically readies the service for both socket and
-path activation, so you may not see the daemon start right away.
+path activation, so you may not see an "active" status.

 Method 2:

 ```{r, engine='bash', count_lines}
-systemctl start telemd.service
+systemctl start telemprobd.service
+systemctl start telempostd.service
 ```

 Method 3:

 ```{r, engine='bash', count_lines}
-telemd &
+telemprobd &
+telempostd &
 ```

-Configure the daemon to autostart at boot
+Configure the client to autostart at boot
 ---------------------

 Method 1 (recommended):
@@ -92,22 +102,23 @@ Method 1 (recommended):
 Enable the socket-activated service and path unit:

 ```{r, engine='bash', count_lines}
-systemctl enable telemd.socket telemd.path
+systemctl enable telemprobd.socket telempostd.path
 ```

 Method 2:

-Enable the service itself, which automatically enables the socket and path
+Enable services, which automatically enables the socket and path
 units as well:

 ```{r, engine='bash', count_lines}
-systemctl enable telemd.service
+systemctl enable telemprobd.service
+systemctl enable telempostd.service
 ```

 Usage
 ---------------------

-Once the daemon is running, the telemetrics probes will be ready to use.
+Once the client is running, the telemetrics probes will be ready to use.

 Available probes:

@@ -117,11 +128,11 @@ that telemetrics-client works. It sends a "hello world" record to the server.
 * crash probe: A handler for core files that sends the corresponding backtraces
 to the server.

-The daemon uses the following configuration options from the configuration file:
+The client uses the following configuration options from the configuration file:

-* server: This specifies the web server that the daemon sends the telemetry records to
+* server: This specifies the web server that the telempostd sends the telemetry records to
 * socket_path: This specifies the path of the unix domain socket that the
-  daemon listens on for connections from the probes
+  telemprobd listens on for connections from the probes
 * spool_dir: This config option is related to spooling. If the daemon is not
  able to send the telemetry records to the backend server due to reasons such
  as the network availability, then it stores the records in a spool directory.
@@ -131,7 +142,7 @@ The daemon uses the following configuration options from the configuration file:
 ```{r, engine='bash', count_lines}
    mkdir -p /var/spool/telemetry
    chown -R telemetry:telemetry /var/spool/telemetry
-    systemctl restart telemd.service
+    systemctl restart telemprobd.service
 ```

 * record_expiry: This is the time in minutes after which the records in the
@@ -231,17 +242,17 @@ you can opt in by creating a static machine id file named
 # echo "unique machine ID" > /etc/telemetrics/opt-in-static-machine-id
 ```

-The telemetry daemon reads, at most, the first 32 characters from this file
+The telemetry client reads, at most, the first 32 characters from this file
 uses it for the machine id. You can put a string like 'my-machine-name' in this
-file to easily identify your machine.  Restart telemd for the machine id
+file to easily identify your machine.  Restart telemprobd for the machine id
 changes to take effect by running:

 ```{r, engine='bash', count_lines}
-# systemctl restart telemd.service
+# systemctl restart telemprobd.service
 ```

 You can switch back to the rotating machine id by deleting the override file
-and restarting the daemon.  You can do a quick test to check that your
+and restarting the client.  You can do a quick test to check that your
 machine-id has changed by running "hprobe" and verifying that a record has
 landed on your backend telemetrics server, with the specified machine id.

@@ -271,18 +282,18 @@ developers not always have access to the backend in these case users can leverag
 features added for local debugging. The following is a list of steps to enable
 local debug:

-* *Enabling record retention*: this step configures telemd to keep
+* *Enabling record retention*: this step configures telempostd to keep
 copies of telemetry records locally. To enable record retention set the value of
 ```record_retention_enabled``` from ```false``` to ```true```. Optionally set
 ```record_server_delivery_enabled`` to ```false``` to keep records local only.
-Remember to restart the daemon after configuration values are updated
+Remember to restart the client after configuration values are updated
 (```telemctl restart```).

 * *Creating a record*: run ```hprobe``` command to create a record for the purposes
 of this step by step guide. Once we have the record or records that you need to
 capture locally you can display the data.

-* *Displaying record metadata*: telemd keeps metadata of any valid record,
+* *Displaying record metadata*: telempostd keeps metadata of any valid record,
 to display this data a new option to telemctl was added ```telemctl journal```.
 Assuming that the last record created was the record from previous step `hprobe` we
 can use `tail -n 1` to print the last created record only, i.e.
@@ -2,13 +2,13 @@
 # Process this file with autoconf to produce a configure script.

 AC_PREREQ([2.69])
-AC_INIT([telemetrics-client], [1.16.2], [https://clearlinux.org/])
+AC_INIT([telemetrics-client], [2.0.1], [https://clearlinux.org/])
 AC_CONFIG_AUX_DIR([build-aux])
 AM_INIT_AUTOMAKE([1.14 -Wall -Werror -Wno-extra-portability foreign subdir-objects])
 AM_SILENT_RULES([yes])
 LT_INIT([disable-static])
 AC_CONFIG_MACRO_DIRS([build-aux/m4])
-AC_CONFIG_SRCDIR([src/main.c])
+AC_CONFIG_SRCDIR([src/probe.c])
 AC_CONFIG_HEADERS([config.h])

 # Checks for programs.
@@ -1,6 +1,7 @@
 MANPAGES = \
 	docs/man/telemctl.1 \
-	docs/man/telemd.1 \
+	docs/man/telemprobd.1 \
+	docs/man/telempostd.1 \
 	docs/man/telem-record-gen.1 \
 	docs/man/telemetry.3 \
 	docs/man/telemetrics.conf.5
@@ -77,6 +77,9 @@ File to read payload from.
 .IP \(bu 2
 \fB\-R\fP, \fB\-\-record\-version\fP <version>:
 Version number for format of payload (default 1).
+.IP \(bu 2
+\fB\-e\fP, \fB\-\-event\-id\fP <version>:
+Event id to use in the record. If not provided a ramdonly generated id will be assigned to record.
 .UNINDENT
 .UNINDENT
 .UNINDENT
@@ -62,6 +62,10 @@ OPTIONS
 * ``-R``, ``--record-version`` <version>:
   Version number for format of payload (default 1).

+ * ``-e``, ``--event-id`` <version>:
+   Event id to use in the record. If not provided a randomly generated id will be assigned to record.
+
+

 RETURN VALUES
 =============
@@ -55,6 +55,9 @@ Opts in to telemetry, and starts telemetry services. The opt\-out file
 \fBopt\-out\fP:
 Opts out of telemetry, and stops telemetry services. The opt\-out file
 \fB/etc/telemetrics/opt\-out\fP is created.
+.IP \(bu 2
+\fBis\-active\fP:
+Checks if telemetry client daemons are active (telemprobd and telempostd).
 .UNINDENT
 .UNINDENT
 .UNINDENT
@@ -40,6 +40,9 @@ OPTIONS
   Opts out of telemetry, and stops telemetry services. The opt-out file
   ``/etc/telemetrics/opt-out`` is created.

+ * ``is-active``:
+   Checks if telemetry client daemons are active (telemprobd and telempostd).
+

 RETURN VALUES
 =============
@@ -0,0 +1,97 @@
+.\" Man page generated from reStructuredText.
+.
+.TH TELEMPOSTD 1 "" "" ""
+.SH NAME
+telempostd \- Telemetry client service
+.
+.nr rst2man-indent-level 0
+.
+.de1 rstReportMargin
+\\$1 \\n[an-margin]
+level \\n[rst2man-indent-level]
+level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
+-
+\\n[rst2man-indent0]
+\\n[rst2man-indent1]
+\\n[rst2man-indent2]
+..
+.de1 INDENT
+.\" .rstReportMargin pre:
+. RS \\$1
+. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
+. nr rst2man-indent-level +1
+.\" .rstReportMargin post:
+..
+.de UNINDENT
+. RE
+.\" indent \\n[an-margin]
+.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
+.nr rst2man-indent-level -1
+.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
+.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
+..
+.SH SYNOPSIS
+.sp
+\fBtelempostd\fP <flags>
+.sp
+\fB/etc/telemetrics/telemetrics.conf\fP
+.sp
+\fB/usr/share/defaults/telemetrics/telemetrics.conf\fP
+.sp
+\fB/etc/telemetrics/opt\-in\-static\-machine\-id\fP
+.SH DESCRIPTION
+.sp
+The \fBtelempostd\fP program delivers locally generated telemetry records to a remote
+telemetry service. Telemetry data can be in any format, and is relayed as\-is.
+.SH OPTIONS
+.INDENT 0.0
+.INDENT 3.5
+.INDENT 0.0
+.IP \(bu 2
+\fB\-f\fP, \fB\-\-config_file\fP [<file>]:
+Configuration file. This overides the other parameters.
+.IP \(bu 2
+\fB\-h\fP, \fB\-\-help\fP:
+Display this help message.
+.IP \(bu 2
+\fB\-V\fP, \fB\-\-version\fP:
+Print the program version.
+.UNINDENT
+.UNINDENT
+.UNINDENT
+.SH FILES
+.INDENT 0.0
+.IP \(bu 2
+\fB/usr/share/defaults/telemetrics/telemetrics.conf\fP
+.INDENT 2.0
+.INDENT 3.5
+If no custom configuration file is found, \fBtelempostd\fP uses the
+settings in this file.
+.UNINDENT
+.UNINDENT
+.IP \(bu 2
+\fB/etc/telemetrics/telemetrics.conf\fP
+.INDENT 2.0
+.INDENT 3.5
+Custom configuration file that \fBtelempostd\fP reads. See \fBtelemetrics.conf\fP(5).
+.UNINDENT
+.UNINDENT
+.UNINDENT
+.SH EXIT STATUS
+.sp
+0 when no errors occurred. A non\-zero exit status indicates a failure occurred.
+.SH SEE ALSO
+.INDENT 0.0
+.IP \(bu 2
+\fBtelemetry\fP(3)
+.IP \(bu 2
+\fBtelemetrics.conf\fP(5)
+.IP \(bu 2
+\fI\%https://github.com/clearlinux/telemetrics\-client\fP
+.IP \(bu 2
+\fI\%https://clearlinux.org/documentation/\fP
+.UNINDENT
+.SH COPYRIGHT
+(C) 2018 Intel Corporation, CC-BY-SA-3.0
+.\" Generated by docutils manpage writer.
+.
@@ -0,0 +1,71 @@
+==========
+telempostd
+==========
+
+------------------------
+Telemetry client service
+------------------------
+
+:Copyright: \(C) 2018 Intel Corporation, CC-BY-SA-3.0
+:Manual section: 1
+
+
+SYNOPSIS
+========
+
+``telempostd`` \<flags\>
+
+``/etc/telemetrics/telemetrics.conf``
+
+``/usr/share/defaults/telemetrics/telemetrics.conf``
+
+``/etc/telemetrics/opt-in-static-machine-id``
+
+
+DESCRIPTION
+===========
+
+The ``telempostd`` program delivers locally generated telemetry records to a remote
+telemetry service. Telemetry data can be in any format, and is relayed as-is.
+
+
+OPTIONS
+=======
+
+  * ``-f``, ``--config_file`` \[\<file\>\]:
+    Configuration file. This overrides the other parameters.
+
+  * ``-h``, ``--help``:
+    Display this help message.
+
+  * ``-V``, ``--version``:
+    Print the program version.
+
+
+FILES
+=====
+
+* ``/usr/share/defaults/telemetrics/telemetrics.conf``
+
+    If no custom configuration file is found, ``telempostd`` uses the
+    settings in this file.
+
+* ``/etc/telemetrics/telemetrics.conf``
+
+    Custom configuration file that ``telempostd`` reads. See ``telemetrics.conf``\(5).
+
+
+EXIT STATUS
+===========
+
+0 when no errors occurred. A non-zero exit status indicates a failure occurred.
+
+
+SEE ALSO
+========
+
+* ``telemetry``\(3)
+* ``telemetrics.conf``\(5)
+* https://github.com/clearlinux/telemetrics-client
+* https://clearlinux.org/documentation/
+
@@ -41,8 +41,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 \fB/etc/telemetrics/opt\-in\-static\-machine\-id\fP
 .SH DESCRIPTION
 .sp
-The \fBtelemd\fP program delivers locally generated telemetry records to a remote
-telemetry service. Telemetry data can be in any format, and is relayed as\-is.
+The \fBtelemd\fP program handles communication between telemetry client and telemetry
+probes.
 .SH OPTIONS
 .INDENT 0.0
 .INDENT 3.5
@@ -1,9 +1,9 @@
-======
-telemd
-======
+==========
+telemprobd
+==========

 ------------------------
-Telemetry client service
+Telemetry probes service
 ------------------------

 :Copyright: \(C) 2017 Intel Corporation, CC-BY-SA-3.0
@@ -13,7 +13,7 @@ Telemetry client service
 SYNOPSIS
 ========

-``telemd`` \<flags\>
+``telemprobd`` \<flags\>

 ``/etc/telemetrics/telemetrics.conf``

@@ -25,8 +25,8 @@ SYNOPSIS
 DESCRIPTION
 ===========

-The ``telemd`` program delivers locally generated telemetry records to a remote
-telemetry service. Telemetry data can be in any format, and is relayed as-is.
+The ``telemprobd`` program handles communication between telemetry client and telemetry
+probes.


 OPTIONS
@@ -47,12 +47,12 @@ FILES

 * ``/usr/share/defaults/telemetrics/telemetrics.conf``

-    If no custom configuration file is found, ``telemd`` uses the
+    If no custom configuration file is found, ``telemprobd`` uses the
    settings in this file.

 * ``/etc/telemetrics/telemetrics.conf``

-    Custom configuration file that ``telemd`` reads. See ``telemetrics.conf``\(5).
+    Custom configuration file that ``telemprobd`` reads. See ``telemetrics.conf``\(5).


 * ``/etc/telemetrics/opt-in-static-machine-id``
@@ -2,8 +2,8 @@

 declare -a SPECIAL_UNITS=(
  hprobe.timer
-  telemd.socket
-  telemd.path
+  telemprobd.socket
+  telempostd.path
 )

 declare -a SERVICES=(
@@ -12,7 +12,8 @@ declare -a SERVICES=(
  oops-probe.service
  pstore-probe.service
  klogscanner.service
-  telemd.service
+  telemprobd.service
+  telempostd.service
 )

 SCRIPT="$0"
@@ -69,7 +70,8 @@ telem_start() {

 telem_is_active() {
  # check only activation units
-  systemctl is-active telemd.socket
+  echo "telemprobd :" $(systemctl is-active telemprobd.socket)
+  echo "telempostd :"  $(systemctl is-active telempostd.path)
 }

 telem_opt_out() {
@@ -103,10 +105,10 @@ usage() {
  printf "$format" "stop" "Stops all running telemetry services"
  printf "$format" "start" "Starts all telemetry services"
  printf "$format" "restart" "Restarts all telemetry services"
-  printf "$format" "is-active" "Checks if telemd is active"
+  printf "$format" "is-active" "Checks if telemprobd and telempostd are active"
  printf "$format" "opt-in" "Opts in to telemetry, and starts telemetry services"
  printf "$format" "opt-out" "Opts out of telemetry, and stops telemetry services"
-  printf "$format" "journal" "Prints telemd journal contents. Use -h argument with"
+  printf "$format" "journal" "Prints telemtry journal contents. Use -h argument with"
  printf "$format" ""        "command for more options"
  printf "\n"
  exit 2
@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics BERT Probe
-Requires=telemd.socket
-After=telemd.socket
+Requires=telemprobd.socket
+After=telemprobd.socket
 ConditionPathExists=!/etc/telemetrics/opt-out
 ConditionPathExists=/sys/firmware/acpi/tables/data/BERT

@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics Heartbeat Probe
-Requires=telemd.socket
-After=telemd.socket
+Requires=telemprobd.socket
+After=telemprobd.socket
 ConditionPathExists=!/etc/telemetrics/opt-out

 [Service]
@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics Systemd Journal Probe
-Requires=telemd.socket
-After=telemd.socket
+Requires=telemprobd.socket
+After=telemprobd.socket
 ConditionPathExists=!/etc/telemetrics/opt-out

 [Service]
@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics Kernel Log Scanner
-Requires=telemd.socket
-After=telemd.socket oops-probe.service
+Requires=telemprobd.socket
+After=telemprobd.socket oops-probe.service

 [Service]
 ExecStart=@bindir@/klogscanner
@@ -21,10 +21,11 @@ EXTRA_DIST += \
 	%D%/klogscanner.service.in \
 	%D%/pstore-clean.service.in \
 	%D%/libtelemetry.pc.in \
-	%D%/telemd.path.in \
-	%D%/telemd.service.in \
-	%D%/telemd.socket.in \
-	%D%/telemd-update-trigger.service.in \
+	%D%/telempostd.service.in \
+	%D%/telempostd.path.in \
+	%D%/telemprobd.service.in \
+	%D%/telemprobd.socket.in \
+	%D%/telemprobd-update-trigger.service.in \
 	%D%/telemetrics-dirs.conf.in \
 	%D%/telemetrics.conf.in

@@ -56,10 +57,11 @@ systemdunit_DATA = \
 	%D%/pstore-probe.service \
 	%D%/klogscanner.service \
 	%D%/pstore-clean.service \
-	%D%/telemd.service \
-	%D%/telemd.socket \
-	%D%/telemd-update-trigger.service \
-	%D%/telemd.path
+	%D%/telemprobd.service \
+	%D%/telemprobd.socket \
+	%D%/telemprobd-update-trigger.service \
+	%D%/telempostd.service \
+	%D%/telempostd.path

 %D%/hprobe.service: %D%/hprobe.service.in
 	$(pathfix) < $< > $@
@@ -82,16 +84,19 @@ systemdunit_DATA = \
 %D%/pstore-clean.service: %D%/pstore-clean.service.in
 	$(pathfix) < $< > $@

-%D%/telemd.service: %D%/telemd.service.in
+%D%/telempostd.path: %D%/telempostd.path.in
 	$(pathfix) < $< > $@

-%D%/telemd.socket: %D%/telemd.socket.in
+%D%/telempostd.service: %D%/telempostd.service.in
 	$(pathfix) < $< > $@

-%D%/telemd.path: %D%/telemd.path.in
+%D%/telemprobd.service: %D%/telemprobd.service.in
 	$(pathfix) < $< > $@

-%D%/telemd-update-trigger.service: %D%/telemd-update-trigger.service.in
+%D%/telemprobd.socket: %D%/telemprobd.socket.in
+	$(pathfix) < $< > $@
+
+%D%/telemprobd-update-trigger.service: %D%/telemprobd-update-trigger.service.in
 	$(pathfix) < $< > $@

 sysctldir = @SYSTEMD_SYSCTLDIR@
@@ -104,10 +109,11 @@ systemconfdir = @SYSTEMD_SYSTEMCONFDIR@
 systemconf_DATA = %D%/40-core-ulimit.conf

 clean-local:
-	-rm -f  %D%/telemd.service \
-		%D%/telemd.socket \
-		%D%/telemd.path \
-		%D%/telemd-update-trigger.service \
+	-rm -f  %D%/telemprobd.service \
+		%D%/telemprobd.socket \
+		%D%/telempostd.service \
+		%D%/telempostd.path \
+		%D%/telemprobd-update-trigger.service \
 		%D%/telemetrics.conf \
 		%D%/telemetrics-dirs.conf \
 		%D%/libtelemetry.pc \
@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics Kernel Oops Probe
-Requires=telemd.socket
-After=telemd.socket
+Requires=telemprobd.socket
+After=telemprobd.socket

 [Service]
 ExecStart=@bindir@/oopsprobe
@@ -1,7 +1,7 @@
 [Unit]
 Description=Telemetrics Pstore Probe
-Requires=telemd.socket
-After=telemd.socket
+Requires=telemprobd.socket
+After=telemprobd.socket

 [Service]
 ExecStart=@bindir@/pstoreprobe
@@ -1,5 +1,5 @@
 [Unit]
-Description=Telemetry Daemon Spool
+Description=Telemetry Post Daemon staging
 ConditionPathExists=!/etc/telemetrics/opt-out

 [Path]
@@ -0,0 +1,12 @@
+[Unit]
+Description=Telemetrics Post Daemon
+ConditionPathExists=!/etc/telemetrics/opt-out
+
+[Service]
+ExecStart=@bindir@/telempostd
+WorkingDirectory=@localstatedir@/spool/telemetry
+User=telemetry
+
+[Install]
+Also=telempostd.path
+WantedBy=multi-user.target
@@ -4,4 +4,4 @@ BindsTo=update-triggers.target

 [Service]
 Type=oneshot
-ExecStart=@systemctldir@/systemctl try-restart telemd.service
+ExecStart=@systemctldir@/systemctl try-restart telemprobd.service
@@ -3,10 +3,10 @@ Description=Telemetrics Daemon
 ConditionPathExists=!/etc/telemetrics/opt-out

 [Service]
-ExecStart=@bindir@/telemd
+ExecStart=@bindir@/telemprobd
 WorkingDirectory=@localstatedir@/spool/telemetry
 User=telemetry

 [Install]
-Also=telemd.socket telemd.path
+Also=telemprobd.socket
 WantedBy=multi-user.target
@@ -0,0 +1,162 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2018 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <limits.h>
+#include <stdbool.h>
+#include <sys/stat.h>
+
+#include "log.h"
+#include "util.h"
+#include "common.h"
+#include "iorecord.h"
+
+bool _fgets(char *s, int n, FILE *stream)
+{
+        char *i;
+
+        if (!fgets(s, n, stream)) {
+                return false;
+        }
+        if ((i = strchr(s, '\n')) != NULL) {
+                *i = '\0';
+        }
+
+        return true;
+}
+
+void stage_record(char *filepath, char *headers[], char *body)
+{
+        int tmpfd;
+        FILE *tmpfile = NULL;
+
+        // Use default path if not provided
+        if (filepath == NULL) {
+                telem_log(LOG_ERR, "filepath value must be provided, aborting\n");
+                exit(EXIT_FAILURE);
+        }
+
+        tmpfd = mkstemp(filepath);
+        if (!tmpfd) {
+                telem_perror("Error opening staging file");
+                close(tmpfd);
+                if (unlink(filepath)) {
+                        telem_perror("Error deleting staging file");
+                }
+                goto clean_exit;
+        }
+
+        // open file
+        tmpfile = fdopen(tmpfd, "a");
+        if (!tmpfile) {
+                telem_perror("Error opening temp stage file");
+                close(tmpfd);
+                if (unlink(filepath)) {
+                        telem_perror("Error deleting temp stage file");
+                }
+                goto clean_exit;
+        }
+
+        // write headers
+        for (int i = 0; i < NUM_HEADERS; i++) {
+                fprintf(tmpfile, "%s\n", headers[i]);
+        }
+
+        //write body
+        fprintf(tmpfile, "%s\n", body);
+        fflush(tmpfile);
+        fclose(tmpfile);
+
+clean_exit:
+
+        return;
+}
+
+bool read_record(char *fullpath, char *headers[], char **body)
+{
+        int i, ret = 0;
+        bool result = false;
+        FILE *fp = NULL;
+        long offset;
+        char line[LINE_MAX] = { 0 };
+        struct stat buf;
+        long size;
+
+        ret = stat(fullpath, &buf);
+        if (ret == -1) {
+                telem_log(LOG_ERR, "Unable to stat record %s in staging\n", fullpath);
+                return false;
+        }
+        size = buf.st_size;
+
+        fp = fopen(fullpath, "r");
+        if (fp == NULL) {
+                telem_log(LOG_ERR, "Unable to open file %s in staging\n", fullpath);
+                return false;
+        }
+
+        for (i = 0; i < NUM_HEADERS; i++) {
+                const char *header_name = get_header_name(i);
+                if (!_fgets(line, LINE_MAX, fp)) {
+                        telem_log(LOG_ERR, "Error while parsing staged record\n");
+                        fclose(fp);
+                        return false;
+                }
+                //Get rid of trailing newline
+                result = get_header(line, header_name, &headers[i]);
+                if (!result) {
+                        telem_log(LOG_ERR, "read_record: Incorrect"
+                                  " headers in record\n");
+                        goto read_error;
+                }
+        }
+
+        offset = ftell(fp);
+        if (offset == -1) {
+                telem_perror("Failed to get staged file offset of record payload");
+                result = false;
+                goto read_error;
+        }
+
+        size = size - offset + 1;
+        *body = malloc((size_t)size);
+        if (*body == NULL) {
+                telem_log(LOG_ERR, "Could not allocate memory for payload from staged file\n");
+                result = false;
+                goto read_error;
+        }
+        memset(*body, 0, (size_t)size);
+
+        //read rest of file
+        size_t newlen = fread(*body, sizeof(char), (size_t)size, fp);
+        if (newlen == 0) {
+                telem_perror("Error reading staged file");
+                result = false;
+                goto read_error;
+        } else {
+                result = true;
+        }
+
+read_error:
+        if (fp) {
+                fclose(fp);
+        }
+
+        return result;
+}
@@ -0,0 +1,38 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2018 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
+#include <stdbool.h>
+
+/**
+ * Save a telemetry record to disk
+ *
+ * @param path pointer to a directory to save record
+ * @param headers pointer to array of headers and values
+ * @param body record message content
+ *
+ */
+void stage_record(char *path, char *headers[], char *body);
+
+/**
+ * Reads a telemetry record
+ *
+ * @param fullpath pointer to full path file name
+ * @param headers pointer to array of headers and values
+ * @param body record message content
+ *
+ * @return true if successful otherwise false
+ */
+bool read_record(char *fullpath, char *headers[], char **body);
@@ -1,39 +1,78 @@
-bin_PROGRAMS = %D%/telemd
+bin_PROGRAMS = \
+	%D%/telemprobd \
+	%D%/telempostd

-%C%_telemd_SOURCES = \
-	%D%/main.c \
+%C%_telemprobd_SOURCES = \
+	%D%/probe.c \
 	%D%/telemdaemon.c \
 	%D%/telemdaemon.h \
-	%D%/spool.c \
-	%D%/spool.h \
+	%D%/iorecord.c \
 	%D%/journal/journal.c \
-	%D%/journal/journal.h \
-	%D%/retention.c \
-	%D%/retention.h
+	%D%/journal/journal.h

-%C%_telemd_LDADD = $(CURL_LIBS) \
+%C%_telemprobd_LDADD = $(CURL_LIBS) \
 	%D%/libtelem-shared.la

-%C%_telemd_CFLAGS = \
+%C%_telemprobd_CFLAGS = \
 	$(AM_CFLAGS)

-%C%_telemd_LDFLAGS = \
+%C%_telemprobd_LDFLAGS = \
 	$(AM_LDFLAGS) \
 	-pie

 if HAVE_SYSTEMD_DAEMON
-%C%_telemd_CFLAGS += \
-        $(SYSTEMD_DAEMON_CFLAGS)
-%C%_telemd_LDADD += \
-        $(SYSTEMD_DAEMON_LIBS)
+%C%_telemprobd_CFLAGS += \
+	$(SYSTEMD_DAEMON_CFLAGS)
+%C%_telemprobd_LDADD += \
+	$(SYSTEMD_DAEMON_LIBS)
 endif

 if LOG_SYSTEMD
 if HAVE_SYSTEMD_JOURNAL
-%C%_telemd_CFLAGS += \
-        $(SYSTEMD_JOURNAL_CFLAGS)
-%C%_telemd_LDADD += \
-        $(SYSTEMD_JOURNAL_LIBS)
+%C%_telemprobd_CFLAGS += \
+	$(SYSTEMD_JOURNAL_CFLAGS)
+%C%_telemprobd_LDADD += \
+	$(SYSTEMD_JOURNAL_LIBS)
+endif
+endif
+
+# Telemetry post daemon
+%C%_telempostd_SOURCES = \
+	%D%/post.c \
+	%D%/telempostdaemon.c \
+	%D%/telempostdaemon.h \
+	%D%/journal/journal.c \
+	%D%/journal/journal.h \
+	%D%/spool.h \
+	%D%/spool.c \
+	%D%/retention.h \
+	%D%/retention.c \
+	%D%/iorecord.c \
+	%D%/iorecord.h
+
+%C%_telempostd_LDADD = $(CURL_LIBS) \
+	%D%/libtelem-shared.la
+
+%C%_telempostd_CFLAGS = \
+	$(AM_CFLAGS)
+
+%C%_telempostd_LDFLAGS = \
+	$(AM_LDFLAGS) \
+	-pie
+
+if HAVE_SYSTEMD_DAEMON
+%C%_telempostd_CFLAGS += \
+	$(SYSTEMD_DAEMON_CFLAGS)
+%C%_telempostd_LDADD += \
+	$(SYSTEMD_DAEMON_LIBS)
+endif
+
+if LOG_SYSTEMD
+if HAVE_SYSTEMD_JOURNAL
+%C%_telempostd_CFLAGS += \
+	$(SYSTEMD_JOURNAL_CFLAGS)
+%C%_telempostd_LDADD += \
+	$(SYSTEMD_JOURNAL_LIBS)
 endif
 endif

@@ -0,0 +1,98 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2018 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <getopt.h>
+#include <unistd.h>
+#include <stdbool.h>
+#include <sys/stat.h>
+
+#include "log.h"
+#include "spool.h"
+#include "configuration.h"
+#include "telempostdaemon.h"
+
+bool (*post_record_ptr)(char *[], char *) = post_record_http;
+
+void print_usage(char *prog)
+{
+        printf("%s: Usage\n", prog);
+        printf("  -f,  --config_file    Configuration file. This overides the other parameters\n");
+        printf("  -h,  --help           Display this help message\n");
+        printf("  -V,  --version        Print the program version\n");
+}
+
+int main(int argc, char **argv)
+{
+
+        int c;
+        int ret = 0;
+        int opt_index = 0;
+        char *config_file = NULL;
+        TelemPostDaemon daemon;
+
+        struct option opts[] = {
+                { "config_file", 1, NULL, 'f' },
+                { "help", 0, NULL, 'h' },
+                { "version", 0, NULL, 'V' },
+                { NULL, 0, NULL, 0 }
+        };
+
+        while ((c = getopt_long(argc, argv, "f:hV", opts, &opt_index)) != -1) {
+                switch (c) {
+                        case 'f':
+                                config_file = optarg;
+                                struct stat buf;
+                                ret = stat(config_file, &buf);
+
+                                /* check if the file exists and is a regular file */
+                                if (ret == -1 || !S_ISREG(buf.st_mode)) {
+                                        telem_log(LOG_ERR, "Configuration file"
+                                                  " path not valid\n");
+                                        exit(EXIT_FAILURE);
+                                }
+                                set_config_file(config_file);
+                                break;
+                        case 'V':
+                                printf(PACKAGE_VERSION "\n");
+                                exit(EXIT_SUCCESS);
+                        case 'h':
+                                print_usage(argv[0]);
+                                exit(EXIT_SUCCESS);
+                        case '?':
+                                print_usage(argv[0]);
+                                exit(EXIT_FAILURE);
+                }
+        }
+
+        initialize_daemon(&daemon);
+
+        daemon.current_spool_size = get_spool_dir_size();
+
+        /* When path activated this will process
+         * the activating message or previously
+         * spooled data */
+        staging_records_loop(&daemon);
+
+        /* This function is blocking, it will
+        * block until a signal is received */
+        run_daemon(&daemon);
+
+        close_daemon(&daemon);
+
+        return 0;
+}
@@ -49,15 +49,12 @@
 #include "log.h"
 #include "telemdaemon.h"
 #include "configuration.h"
-#include "retention.h"
-#include "spool.h"

 /*
 *  Using a function pointer for unit testing to isolate the call to actual post function.
 *  The call to post the record to the server is stubbed out in the unit tests
 *  using pointer to a fake function.
 */
-bool (*post_record_ptr)(char *[], char *, bool) = post_record_http;

 void print_usage(char *prog)
 {
@@ -79,7 +76,6 @@ int main(int argc, char **argv)
        int c;
        char *config_file = NULL;
        int opt_index = 0;
-        nfds_t initial_nfds = 0;
        sigset_t mask;
        //bool interrupted = false;

@@ -123,11 +119,6 @@ int main(int argc, char **argv)
        }
        initialize_daemon(&daemon);

-        /* Register record retention delete action as a callback to prune entry */
-        if (daemon.record_journal) {
-                daemon.record_journal->prune_entry_callback = &delete_record_by_id;
-        }
-
        sigemptyset(&mask);

        if (sigaddset(&mask, SIGHUP) != 0) {
@@ -227,19 +218,9 @@ int main(int argc, char **argv)

        telem_log(LOG_INFO, "Listening on socket...\n");

-        /* Calculate the spool directory size */
-        if (is_spool_valid()) {
-                daemon.is_spool_valid = true;
-                daemon.current_spool_size = get_spool_dir_size();
-        }
-
-        int spool_process_time = spool_process_time_config();
-
-        spool_records_loop(&(daemon.current_spool_size));
-        time_t last_spool_run_time = time(NULL);
-        time_t last_daemon_start_time = time(NULL);
-
        bool daemon_recycling_enabled = daemon_recycling_enabled_config();
+        int spool_process_time = spool_process_time_config();
+        time_t last_daemon_start_time = time(NULL);

        ret = update_machine_id();
        if (ret == -1) {
@@ -251,18 +232,9 @@ int main(int argc, char **argv)

        time_t last_refresh_time = time(NULL);

-        /* Save initial count for non connection fds */
-        initial_nfds = daemon.nfds;
-
        /* Loop to accept clients */
        while (1) {
                malloc_trim(0);
-                if (initial_nfds == daemon.nfds) {
-                        ret = prune_journal(daemon.record_journal, JOURNAL_TMPDIR);
-                        if (ret != 0) {
-                                telem_log(LOG_WARNING, "Unable to prune journal\n");
-                        }
-                }
                ret = poll(daemon.pollfds, daemon.nfds, spool_process_time * 1000);
                if (ret == -1) {
                        telem_perror("Failed to poll daemon file descriptors");
@@ -292,12 +264,6 @@ int main(int argc, char **argv)
                                                telem_log(LOG_INFO, "Received a SIGHUP signal\n");
                                                /* reload configuration file */
                                                reload_config();
-
-                                                if (is_spool_valid()) {
-                                                        daemon.is_spool_valid = true;
-                                                        //calc_spool_dir_size();
-                                                        daemon.current_spool_size = get_spool_dir_size();
-                                                }
                                        }
                                }

@@ -353,21 +319,15 @@ int main(int argc, char **argv)
                        }
                } else {
                        time_t now = time(NULL);
-                        /* If spool is empty and time to recycle the daemon has elapsed*/
-                        if (daemon_recycling_enabled && (daemon.current_spool_size == 0) &&
+                        /* time to recycle the daemon has elapsed*/
+                        if (daemon_recycling_enabled &&
                            difftime(now, last_daemon_start_time) >= TM_DAEMON_EXIT_TIME) {
                                /* Exit */
                                telem_log(LOG_INFO, "Daemon exiting for recycling\n");
                                goto clean_exit;
                        }
                }
-
                time_t now = time(NULL);
-                if (difftime(now, last_spool_run_time) >= spool_process_time) {
-                        spool_records_loop(&(daemon.current_spool_size));
-                        last_spool_run_time = time(NULL);
-                }
-
                if (difftime(now, last_refresh_time) >= TM_REFRESH_RATE) {
                        int ret = update_machine_id();
                        if (ret == -1) {
@@ -378,14 +338,13 @@ int main(int argc, char **argv)
        }

 clean_exit:
-        close_journal(daemon.record_journal);

        /* Free memory before exiting */
        while ((cl = LIST_FIRST(&(daemon.client_head))) != NULL) {
                remove_client(&(daemon.client_head), cl);
        }
        free(daemon.pollfds);
-
+        free(daemon.machine_id_override);
        if (LIST_EMPTY(&(daemon.client_head))) {
                telem_log(LOG_INFO, "Client list cleared\n");
        }
@@ -641,16 +641,12 @@ int main(int argc, char **argv)
                nc_string_append_printf(header, "Signal: %ld\n", signal_num);
        }

-        if (process_corefile(&backtrace) < 0) {
-                goto fail;
-        }
-
        /* On Clear Linux OS, missing symbols may appear if automatic debuginfo
         * downloads are still in flight. So if any missing symbols appear on
         * the first run (indicated by the presence of "??? - ["), wait 10
-         * seconds and try again.
+         * seconds and try again. Also retry if errors occur in the first run.
         */
-        if (strstr(backtrace->str, "??? - [")) {
+        if ((process_corefile(&backtrace) < 0) || (strstr(backtrace->str, "??? - ["))) {
                sleep(10);

                if (prepare_corefile(&e_core, core_fd) < 0) {
@@ -39,10 +39,13 @@ static char *opt_payload = NULL;
 static uint32_t payload_version = 1;
 static char *opt_payload_file = NULL;
 static char *opt_event_id = NULL;
+static bool opt_echo = false;
+static bool opt_nopost = false;

 static const struct option prog_opts[] = {
        { "help", no_argument, 0, 'h' },
-        { "config-file", required_argument, 0, 'f' },
+        { "echo", no_argument, 0, 'o' },
+        { "no-post", no_argument, 0, 'n' },
        { "version", no_argument, 0, 'V' },
        { "severity", required_argument, 0, 's' },
        { "class", required_argument, 0, 'c' },
@@ -62,14 +65,15 @@ static void print_help(void)
        printf("  -h, --help            Show help options\n");
        printf("\n");
        printf("Application Options:\n");
-        printf("  -f, --config-file     Path to configuration file (not implemented yet)\n");
        printf("  -V, --version         Print the program version\n");
        printf("  -s, --severity        Severity level (1-4) - (default 1)\n");
-        printf("  -c, --class           Classification level_1/level_2/level_3\n");
-        printf("  -p, --payload         Record body (max size = 8k)\n");
+        printf("  -c, --class           Classification level_1/level_2/level_3 (required)\n");
+        printf("  -p, --payload         Record body (max size = 8k) (required)\n");
        printf("  -P, --payload-file    File to read payload from\n");
        printf("  -R, --record-version  Version number for format of payload (default 1)\n");
        printf("  -e, --event-id        Event id to use in the record\n");
+        printf("  -o, --echo            Echo record to stdout\n");
+        printf("  -n, --no-post         Do not post record just print\n");
        printf("\n");
 }

@@ -94,7 +98,7 @@ int parse_options(int argc, char **argv)
        long unsigned int tmp = 0;

        int opt;
-        while ((opt = getopt_long(argc, argv, "hc:Vs:c:p:P:R:e:", prog_opts, NULL)) != -1) {
+        while ((opt = getopt_long(argc, argv, "hc:Vs:c:p:P:R:e:on", prog_opts, NULL)) != -1) {
                switch (opt) {
                        case 'h':
                                print_help();
@@ -149,6 +153,12 @@ int parse_options(int argc, char **argv)
                                        goto fail;
                                }
                                break;
+                        case 'o':
+                                opt_echo = true;
+                                break;
+                        case 'n':
+                                opt_nopost = true;
+                                break;
                }
        }

@@ -322,34 +332,62 @@ out1:
        return ret;
 }

-int send_record(char *payload)
+int instanciate_record(struct telem_ref **t_ref, char *payload)
 {
-        struct telem_ref *t_ref;
        int ret = 0;

-        if (tm_create_record(&t_ref, (uint32_t)severity,
-                             opt_class, payload_version) < 0) {
+        if ((ret = tm_create_record(t_ref, (uint32_t)severity,
+                                    opt_class, payload_version)) < 0) {
                goto out1;
        }

-        if (opt_event_id && tm_set_event_id(t_ref, opt_event_id) < 0) {
-                goto out2;
+        if ((ret = opt_event_id && tm_set_event_id(*t_ref, opt_event_id)) < 0) {
+                goto out1;
        }

-        if (tm_set_payload(t_ref, payload) < 0) {
-                goto out2;
+        if ((ret = tm_set_payload(*t_ref, payload)) < 0) {
+                goto out1;
        }
-
-        if (tm_send_record(t_ref) >= 0) {
-                ret = 1;
-        }
-
-out2:
-        tm_free_record(t_ref);
 out1:
        return ret;
 }

+int send_record(char *payload)
+{
+        struct telem_ref *t_ref = NULL;
+        int ret = 0;
+
+        if ((ret = instanciate_record(&t_ref, payload)) < 0) {
+                goto out;
+        }
+
+        if ((ret = tm_send_record(t_ref)) < 0) {
+                goto out;
+        }
+
+out:
+        tm_free_record(t_ref);
+
+        return ret;
+}
+
+int print_record(char *payload)
+{
+        struct telem_ref *t_ref = NULL;
+        int ret = 0;
+        int i = 0;
+
+        if ((ret = instanciate_record(&t_ref, payload)) != -1) {
+                for (i = 0; i < NUM_HEADERS; i++) {
+                        fprintf(stdout, "%s", t_ref->record->headers[i]);
+                }
+                fprintf(stdout, "%s\n", t_ref->record->payload);
+                tm_free_record(t_ref);
+        }
+
+        return ret;
+}
+
 int main(int argc, char **argv)
 {
        int ret = EXIT_FAILURE;
@@ -367,7 +405,11 @@ int main(int argc, char **argv)
                goto fail;
        }

-        if (!send_record(payload)) {
+        if (opt_echo == true) {
+                print_record(payload);
+        }
+
+        if (opt_nopost == false && send_record(payload) != 0) {
                goto fail;
        }

@@ -377,6 +419,7 @@ fail:
        free(opt_class);
        free(opt_payload);
        free(opt_event_id);
+        free(payload);

        return ret;
 }
@@ -27,7 +27,7 @@
 #include <errno.h>

 #include "spool.h"
-#include "telemdaemon.h"
+#include "telempostdaemon.h"
 #include "log.h"
 #include "configuration.h"
 #include "util.h"
@@ -221,7 +221,7 @@ void transmit_spooled_record(char *record_path, bool *post_succeeded, long size)
                }
                //Get rid of trailing newline
                strtok(line, "\n");
-                if (get_header(line, header_name, &headers[i], strlen(header_name))) {
+                if (get_header(line, header_name, &headers[i])) {
                        continue;
                } else {
                        telem_log(LOG_ERR, "transmit_spooled_record: Incorrect"
@@ -254,7 +254,7 @@ void transmit_spooled_record(char *record_path, bool *post_succeeded, long size)
                goto read_error;
        }

-        *post_succeeded = post_record_http(headers, payload, false);
+        *post_succeeded = post_record_http(headers, payload);
        if (*post_succeeded) {
                unlink(record_path);
        }
@@ -29,12 +29,12 @@
 #include <malloc.h>
 #include <sys/uio.h>

+#include "iorecord.h"
 #include "telemdaemon.h"
 #include "common.h"
 #include "util.h"
 #include "log.h"
 #include "configuration.h"
-#include "retention.h"

 void initialize_daemon(TelemDaemon *daemon)
 {
@@ -43,32 +43,7 @@ void initialize_daemon(TelemDaemon *daemon)
        daemon->nfds = 0;
        daemon->pollfds = NULL;
        daemon->client_head = head;
-        daemon->bypass_http_post_ts = 0;
-        daemon->is_spool_valid = false;
-        daemon->record_journal = open_journal(JOURNAL_PATH);
-        initialize_rate_limit(daemon);
-        initialize_record_delivery(daemon);
-        daemon->current_spool_size = 0;
-}
-
-void initialize_rate_limit(TelemDaemon *daemon)
-{
-        for (int i = 0; i < TM_RATE_LIMIT_SLOTS; i++) {
-                daemon->record_burst_array[i] = 0;
-                daemon->byte_burst_array[i] = 0;
-        }
-        daemon->rate_limit_enabled = rate_limit_enabled_config();
-        daemon->record_burst_limit = record_burst_limit_config();
-        daemon->record_window_length = record_window_length_config();
-        daemon->byte_burst_limit = byte_burst_limit_config();
-        daemon->byte_window_length = byte_window_length_config();
-        daemon->rate_limit_strategy = rate_limit_strategy_config();
-}
-
-void initialize_record_delivery(TelemDaemon *daemon)
-{
-        daemon->record_retention_enabled = record_retention_enabled_config();
-        daemon->record_server_delivery_enabled = record_server_delivery_enabled_config();
+        daemon->machine_id_override = NULL;
 }

 client *add_client(client_list_head *client_head, int fd)
@@ -272,42 +247,18 @@ void machine_id_replace(char **machine_header, char *machine_id_override)
        free(old_header);
 }

-void save_entry_to_journal(TelemDaemon *daemon, time_t t_stamp, char *headers[])
-{
-        char *classification_value;
-        char *event_id_value;
-
-        if (get_header_value(headers[TM_CLASSIFICATION], &classification_value) &&
-            get_header_value(headers[TM_EVENT_ID], &event_id_value)) {
-                if (new_journal_entry(daemon->record_journal, classification_value, t_stamp, event_id_value) != 0) {
-                        telem_log(LOG_INFO, "new_journal_entry in process_record: failed saving record entry\n");
-                }   
-        }   
-        free(classification_value);
-        free(event_id_value);
-}
-
 void process_record(TelemDaemon *daemon, client *cl)
 {
        int i = 0;
+        int ret = 0;
        char *headers[NUM_HEADERS];
-        bool do_spool = false;
-        bool record_sent = false;
        char *tok = NULL;
        size_t header_size = 0;
        size_t message_size = 0;
        char *temp_headers = NULL;
        char *msg;
        char *body;
-        int current_minute = 0;
-        bool record_check_passed = true;
-        bool byte_check_passed = true;
-        bool record_burst_enabled = true;
-        bool byte_burst_enabled = true;
-        /* Gets the current minute of time */
-        time_t temp = time(NULL);
-        struct tm *tm_s = localtime(&temp);
-        current_minute = tm_s->tm_min;
+        char *recordpath = NULL;

        header_size = *(uint32_t *)cl->buf;
        message_size = cl->size - header_size;
@@ -321,7 +272,7 @@ void process_record(TelemDaemon *daemon, client *cl)
        for (i = 0; i < NUM_HEADERS; i++) {
                const char *header_name = get_header_name(i);

-                if (get_header(tok, header_name, &headers[i], strlen(header_name))) {
+                if (get_header(tok, header_name, &headers[i])) {
                        if (strcmp(header_name, TM_MACHINE_ID_STR) == 0) {
                                machine_id_replace(&headers[i], daemon->machine_id_override);
                        }
@@ -335,83 +286,15 @@ void process_record(TelemDaemon *daemon, client *cl)
        /* TODO : check if the body is within the limits. */
        body = msg + header_size;

-        /* Save record in journal */
-        save_entry_to_journal(daemon, temp, headers);
-
-        /* Save local copy */
-        if (daemon->record_retention_enabled) {
-                save_local_copy(daemon, body);
-        }
-
-        /* Bail out if server delivery is not enabled */
-        if (!daemon->record_server_delivery_enabled) {
-#ifdef DEBUG
-                fprintf(stdout, "process_record: record server delivery disabled\n");
-#endif
-                goto end;
-        }
-
-        if (inside_direct_spool_window(daemon, time(NULL))) {
-                telem_log(LOG_INFO, "process_record: delivering directly to spool\n");
-                spool_record(daemon, headers, body);
-                goto end;
-        }
-        if (daemon->record_window_length == -1 ||
-            daemon->byte_window_length == -1) {
-                telem_log(LOG_ERR, "Invalid value for window length\n");
+        /* Save record to stage */
+        ret = asprintf(&recordpath, "%s/XXXXXX", spool_dir_config());
+        if (ret == -1) {
+                telem_log(LOG_ERR, "Failed to allocate memory for record name in staging folder, aborting\n");
                exit(EXIT_FAILURE);
        }
-        /* Checks if entirety of rate limiting is enabled */
-        if (daemon->rate_limit_enabled) {
-                /* Checks whether record and byte bursts are enabled individually */
-                record_burst_enabled = burst_limit_enabled(daemon->record_burst_limit);
-                byte_burst_enabled = burst_limit_enabled(daemon->byte_burst_limit);
-
-                if (record_burst_enabled) {
-                        record_check_passed = rate_limit_check(current_minute,
-                                                               daemon->record_burst_limit, daemon->record_window_length,
-                                                               daemon->record_burst_array, TM_RECORD_COUNTER);
-                }
-                if (byte_burst_enabled) {
-                        byte_check_passed = rate_limit_check(current_minute,
-                                                             daemon->byte_burst_limit, daemon->byte_window_length,
-                                                             daemon->byte_burst_array, cl->size);
-                }
-                /* If both record and byte burst disabled, rate limiting disabled */
-                if (!record_burst_enabled && !byte_burst_enabled) {
-                        daemon->rate_limit_enabled = false;
-                }
-        }
-
-        /* Sends record if rate limiting is disabled, or all checks passed */
-        if (!daemon->rate_limit_enabled || (record_check_passed && byte_check_passed)) {
-                /* Send the record as https post */
-                record_sent = post_record_ptr(headers, body, true);
-        }
-        // Get rate-limit strategy
-        do_spool = spool_strategy_selected(daemon);
-
-        // Drop record
-        if (!record_sent && !do_spool) {
-                goto end;
-        }
-        // Spool Record
-        else if (!record_sent && do_spool) {
-                start_network_bypass(daemon);
-                telem_log(LOG_INFO, "process_record: initializing direct-spool window\n");
-                spool_record(daemon, headers, body);
-        } else {
-                /* Updates rate limiting arrays if record sent */
-                if (record_burst_enabled) {
-                        rate_limit_update (current_minute, daemon->record_window_length,
-                                           daemon->record_burst_array, TM_RECORD_COUNTER);
-                }
-                if (byte_burst_enabled) {
-                        rate_limit_update (current_minute, daemon->byte_window_length,
-                                           daemon->byte_burst_array, cl->size);
-                }
-        }

+        stage_record(recordpath, headers, body);
+        free(recordpath);
 end:
        free(temp_headers);
        for (int k = 0; k < i; k++)
@@ -419,274 +302,6 @@ end:
        return;
 }

-bool burst_limit_enabled(int64_t burst_limit)
-{
-        return (burst_limit > -1) ? true : false;
-}
-
-bool rate_limit_check(int current_minute, int64_t burst_limit, int
-                      window_length, size_t *array, size_t incValue)
-{
-        size_t count = 0;
-
-        /* Variable to determine last minute allowed in burst window */
-        int window_start = (TM_RATE_LIMIT_SLOTS + (current_minute - window_length + 1))
-                           % TM_RATE_LIMIT_SLOTS;
-
-        /* The modulo is not placed in the for loop inself, because if
-         * the current minute is less than the window start (from wrapping)
-         * the for loop will never be entered.
-         */
-
-        /* Counts all elements within burst_window in array */
-        for (int i = window_start; i < (window_start + window_length); i++) {
-                count += array[i % TM_RATE_LIMIT_SLOTS];
-
-                if ((array[i % TM_RATE_LIMIT_SLOTS] + incValue) >= SIZE_MAX) {
-                        /* Exceeds maximum size array can hold */
-                        return false;
-                }
-        }
-        /* Include current record being processed to count */
-        count += incValue;
-
-        /* Determines whether the count has exceeded the limit or not */
-        return (count > burst_limit) ? false : true;
-}
-
-void rate_limit_update(int current_minute, int window_length, size_t *array,
-                       size_t incValue)
-{
-        /* Variable to determine the amount of zero'd out spots needed */
-        int blank_slots = (TM_RATE_LIMIT_SLOTS - window_length);
-
-        /* Update specified array depending on increment value */
-        array[current_minute] += incValue;
-
-        /* Zero out expired records for record array */
-        for (int i = current_minute + 1; i < (blank_slots + current_minute + 1); i++) {
-                array[i % TM_RATE_LIMIT_SLOTS] = 0;
-        }
-}
-
-bool spool_strategy_selected(TelemDaemon *daemon)
-{
-        /* Performs action depending on strategy choosen */
-        return (strcmp(daemon->rate_limit_strategy, "spool") == 0) ? true : false;
-}
-
-bool inside_direct_spool_window(TelemDaemon *daemon, time_t current_time)
-{
-        return (current_time < daemon->bypass_http_post_ts + 1800) ? true : false;
-}
-
-void start_network_bypass(TelemDaemon *daemon)
-{
-        daemon->bypass_http_post_ts = time(NULL);
-}
-
-size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
-{
-        telem_log(LOG_DEBUG, "Received data:\n%s\n", ptr);
-        return size * nmemb;
-}
-
-bool post_record_http(char *headers[], char *body, bool spool)
-{
-        CURL *curl;
-        int res = 0;
-        char *content = "Content-Type: application/text";
-        struct curl_slist *custom_headers = NULL;
-        char errorbuf[CURL_ERROR_SIZE];
-        long http_response = 0;
-        const char *cert_file = get_cainfo_config();
-        const char *tid_header = get_tidheader_config();
-
-        // Initialize the libcurl global environment once per POST. This lets us
-        // clean up the environment after each POST so that when the daemon is
-        // sitting idle, it will be consuming as little memory as possible.
-        curl_global_init(CURL_GLOBAL_ALL);
-
-        curl = curl_easy_init();
-        if (!curl) {
-                telem_log(LOG_ERR, "curl_easy_init(): Unable to start libcurl"
-                          " easy session, exiting\n");
-                exit(EXIT_FAILURE);
-                /* TODO: check if memory needs to be released */
-        }
-
-        // Errors for any curl_easy_* functions will store nice error messages
-        // in errorbuf, so send log messages with errorbuf contents
-        curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorbuf);
-
-        curl_easy_setopt(curl, CURLOPT_URL, server_addr_config());
-        curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 5L);
-        curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10L);
-        curl_easy_setopt(curl, CURLOPT_POST, 1);
-#ifdef DEBUG
-        curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
-#endif
-
-        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);
-
-        for (int i = 0; i < NUM_HEADERS; i++) {
-                custom_headers = curl_slist_append(custom_headers, headers[i]);
-        }
-        custom_headers = curl_slist_append(custom_headers, tid_header);
-        // This should be set by probes/libtelemetry in the future
-        custom_headers = curl_slist_append(custom_headers, content);
-
-        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, custom_headers);
-        curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body);
-        curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, strlen(body));
-        curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_TRY);
-
-        if (strlen(cert_file) > 0) {
-                if (access(cert_file, F_OK) != -1) {
-                        curl_easy_setopt(curl, CURLOPT_CAINFO, cert_file);
-                        telem_log(LOG_INFO, "cafile was set to %s\n", cert_file);
-                }
-        }
-
-        telem_log(LOG_DEBUG, "Executing curl operation...\n");
-        errorbuf[0] = 0;
-        res = curl_easy_perform(curl);
-        curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_response);
-
-        if (res) {
-                size_t len = strlen(errorbuf);
-                if (len) {
-                        telem_log(LOG_ERR, "Failed sending record: %s%s", errorbuf,
-                                  ((errorbuf[len - 1] != '\n') ? "\n" : ""));
-                } else {
-                        telem_log(LOG_ERR, "Failed sending record: %s\n",
-                                  curl_easy_strerror(res));
-                }
-        } else if (http_response != 201 && http_response != 200) {
-                /*  201 means the record was  successfully created
-                 *  200 is a generic "ok".
-                 *  TODO: Make list of white-listed response codes configurable?
-                 */
-                telem_log(LOG_ERR, "Encountered error %ld on the server\n",
-                          http_response);
-                // We treat HTTP error codes the same as libcurl errors
-                res = 1;
-        } else {
-                telem_log(LOG_INFO, "Record sent successfully\n");
-        }
-
-        curl_slist_free_all(custom_headers);
-        curl_easy_cleanup(curl);
-
-        curl_global_cleanup();
-
-        return res ? false : true;
-}
-
-void save_local_copy(TelemDaemon *daemon, char *body)
-{
-        int ret = 0;
-        char *tmpbuf = NULL;
-        FILE *tmpfile = NULL;
-
-        if (daemon == NULL || daemon->record_journal == NULL || 
-            daemon->record_journal->latest_record_id == NULL) {
-                return;
-        }
-
-        ret = asprintf(&tmpbuf, "%s/%s", RECORD_RETENTION_DIR,
-                       daemon->record_journal->latest_record_id);
-        if (ret == -1) {
-                telem_log(LOG_ERR, "Failed to allocate memory for record full path, aborting\n");
-                return;
-        }
-
-        tmpfile = fopen(tmpbuf, "w");
-        if (!tmpfile) {
-                telem_perror("Error opening local record copy temp file");
-                goto save_err;
-        }
-
-        // Save body
-        fprintf(tmpfile, "%s\n", body);
-        fclose(tmpfile);
-
-save_err:
-        free(tmpbuf);
-        return;
-}
-
-void spool_record(TelemDaemon *daemon, char *headers[], char *body)
-{
-        int ret = 0;
-        struct stat stat_buf;
-        int64_t max_spool_size = 0;
-        char *tmpbuf = NULL;
-        FILE *tmpfile = NULL;
-        int tmpfd = 0;
-
-        if (!daemon->is_spool_valid) {
-                /* If spool is not valid, simply drop the record */
-                return;
-        }
-
-        //check if the size is greater than 1 MB/spool max size
-        max_spool_size = spool_max_size_config();
-        if (max_spool_size != -1) {
-                telem_log(LOG_DEBUG, "Total size of spool dir: %ld\n", daemon->current_spool_size);
-                if (daemon->current_spool_size < 0) {
-                        telem_log(LOG_ERR, "Error getting spool directory size: %s\n",
-                                  strerror(-(int)(daemon->current_spool_size)));
-                        return;
-                } else if (daemon->current_spool_size >= (max_spool_size * 1024)) {
-                        telem_log(LOG_INFO, "Spool dir full, dropping record\n");
-                        return;
-                }
-        }
-
-        // create file with record
-        ret = asprintf(&tmpbuf, "%s/XXXXXX", spool_dir_config());
-        if (ret == -1) {
-                telem_log(LOG_ERR, "Failed to allocate memory for record name, aborting\n");
-                exit(EXIT_FAILURE);
-        }
-
-        tmpfd = mkstemp(tmpbuf);
-        if (tmpfd == -1) {
-                telem_perror("Error while creating temp file");
-                goto spool_err;
-        }
-
-        tmpfile = fdopen(tmpfd, "a");
-        if (!tmpfile) {
-                telem_perror("Error opening temp file");
-                close(tmpfd);
-                if (unlink(tmpbuf)) {
-                        telem_perror("Error deleting temp file");
-                }
-                goto spool_err;
-        }
-
-        // write headers
-        for (int i = 0; i < NUM_HEADERS; i++) {
-                fprintf(tmpfile, "%s\n", headers[i]);
-        }
-
-        //write body
-        fprintf(tmpfile, "%s\n", body);
-        fflush(tmpfile);
-        fclose(tmpfile);
-
-        /* The stat should not fail here unless it is ENOMEM */
-        if (stat(tmpbuf, &stat_buf) == 0) {
-                daemon->current_spool_size += (stat_buf.st_blocks * 512);
-        }
-
-spool_err:
-        free(tmpbuf);
-        return;
-}
-
 void add_pollfd(TelemDaemon *daemon, int fd, short events)
 {
        assert(daemon);
@@ -22,7 +22,6 @@
 #include <sys/queue.h>
 #include <stdbool.h>
 #include <inttypes.h>
-#include "journal/journal.h"

 #define TM_MACHINE_ID_EXPIRY (3 /*d*/ * 24 /*h*/ * 60 /*m*/ * 60 /*s*/)

@@ -55,26 +54,6 @@ typedef struct TelemDaemon {
        size_t current_alloc;
        /* client list head */
        client_list_head client_head;
-        /* Time of last failed post */
-        time_t bypass_http_post_ts;
-        /* Telemetry Journal*/
-        TelemJournal *record_journal;
-        /* Rate limit record and byte arrays */
-        size_t record_burst_array[TM_RATE_LIMIT_SLOTS];
-        size_t byte_burst_array[TM_RATE_LIMIT_SLOTS];
-        /* Rate Limit Configurations */
-        bool rate_limit_enabled;
-        int64_t record_burst_limit;
-        int record_window_length;
-        int64_t byte_burst_limit;
-        int byte_window_length;
-        const char *rate_limit_strategy;
-        /* Spool configuration */
-        bool is_spool_valid;
-        long current_spool_size;
-        /* Record local copy and delivery  */
-        bool record_retention_enabled;
-        bool record_server_delivery_enabled;
        char *machine_id_override;
 } TelemDaemon;

@@ -85,21 +64,6 @@ typedef struct TelemDaemon {
 */
 void initialize_daemon(TelemDaemon *daemon);

-/**
- * Initialize rate-limiting in the daemon
- *
- * @param daemon A pointer to the daemon structure
- */
-void initialize_rate_limit(TelemDaemon *daemon);
-
-/**
- * Initialize record delivery to remote and locally
- * in the daemon
- *
- * @param daemon A pointer to the daemon structure
- */
-void initialize_record_delivery(TelemDaemon *daemon);
-
 /**
 * Add poll fd struct to the array of pollfds.
 *
@@ -182,113 +146,6 @@ void terminate_client(TelemDaemon *daemon, client *cl, nfds_t index);
 */
 void process_record(TelemDaemon *daemon, client *cl);

-/**
- * Check if burst limiting is enabled
- *
- * @param burst_limit Burst limit amount. If negative rate limiting
- * is disabled
- *
- * @return true if rate limiting for burst_limit variable is enabled
- */
-bool burst_limit_enabled(int64_t burst_limit);
-
-/**
- * Checks whether record does not exceed the burst threshold
- *
- * @param current_minute The current minute of time the record processed
- * @param burst_limit Amount of records that can be passed within a window
- * @param window_length Window of time burst limit refers to
- * @param array Pointer to array that stores record or byte count
- * @param incValue Amount to increase array slot by
- *
- * @return true if record does not exceed threshold, and false otherwise
- */
-bool rate_limit_check(int current_minute, int64_t burst_limit, int
-                      window_length, size_t *array, size_t incValue);
-
-/**
- * Updates rate limit arrays with passing record information
- *
- * @param current_minute The current minute of time the record processed
- * @param window_length Window of time burst limit refers to
- * @param array Pointer to array that stores record or byte count
- * @param incValue Amount to increase array slot by
- *
- */
-void rate_limit_update(int current_minute, int window_length,
-                       size_t *array, size_t incValue);
-
-/**
- * Checks if the spool strategy is chosen for rate-limiting
- *
- * @param daemon Pointer to the daemon
- *
- * @return true if spool is choosen false if drop is choosen
- */
-bool spool_strategy_selected(TelemDaemon *daemon);
-
-/**
- * Check if daemon should direct-spool after encountering network issues.
- *
- * @param daemon Pointer to the daemon
- * @param time The time as time_t (usually from time())
- *
- * @return true if inside direct spool window, false otherwise.
- */
-bool inside_direct_spool_window(TelemDaemon *daemon, time_t time);
-
-/**
- * Set the start time of the direct spool window. Will use the
- * current time.
- *
- * @param daemon Pointer to the daemon
- *
- * @return no return value.
- */
-void start_network_bypass(TelemDaemon *daemon);
-
-/**
- * Function pointer for posting the records to the server. Replaced by a
- * mock function in the unit tests.
- *
- * @param header_values The header data read from the socket
- * @param body The payload of the record read
- * @param spool Indicate whether the record should be spooled if not
- *    successfully delivered
- *
- * @return true if the record was successfully posted
- */
-extern bool (*post_record_ptr)(char *headers[], char *body, bool spool);
-
-/**
- * Post records on a http connection to server
- *
- * @param header_values The header data read from the socket
- * @param body The payload of the record read
- * @param spool Indicate whether the record should be spooled if not
- *    successfully delivered
- *
- * @return true if the record was successfully posted
- */
-bool post_record_http(char *header_values[], char *body, bool spool);
-
-/**
- * Spool the record in case it cannot be delivered to the server
- *
- * @param headers The headers for the record
- * @param body The payload of the record
- *
- */
-void spool_record(TelemDaemon * daemon, char *headers[], char *body);
-
-/**
- * Save a copy of the record payload locally
- *
- * @param body The payload of the record
- *
- */
-void save_local_copy(TelemDaemon *daemon, char *body);
-
 /**
 * Get random machine id stored in file
 *
@@ -310,4 +167,5 @@ int update_machine_id(void);
 * or if the file is empty
 */
 char *read_machine_id_override(void);
+
 /* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
@@ -316,7 +316,7 @@ static int set_system_build_header(struct telem_ref *t_ref)

 /**
 * Sets the header reporting the machine_id. Note, the machine_id is really
- * managed by the telemetrics daemon (telemd), and populated by the daemon
+ * managed by the telemetrics daemon (telemprobd), and populated by the daemon
 * as it delivers the records. Consequently, the header is simply set to a
 * dummy value of 0xFFFFFFF.
 *
@@ -410,7 +410,7 @@ static int set_cpu_model_header(struct telem_ref *t_ref)
                        model_str_len = strlen(model_name);
                        if (model_str_len > 2) {
                                model_name = (char *)model_name + 2 * sizeof(char);
-                                model_name[model_str_len - 2] = '\0';
+                                model_name[model_str_len - 3] = '\0';
                        } else {
                                model_name = "blank";
                        }
@@ -1081,7 +1081,7 @@ int tm_set_event_id(struct telem_ref *t_ref, char *event_id)
 }

 /**
- * Write nbytes from buf to fd. Used to send records to telemd.
+ * Write nbytes from buf to fd. Used to send records to telemprobd.
 *
 * @param fd Socket fd obtained from tm_get_socket.
 * @param buf Data to be written to the socket.
@@ -31,7 +31,10 @@
 * - a library, libtelemetry, that telemetrics probes use to create telemetrics
 *   records and send them to the daemon for further processing.
 *
- * - a daemon, telemd, that prepares the records to send to a telemetrics server
+ * - telemprobd, a daemon that handles communication with probes and forwards
+ *   records to a second daemon.
+ *
+ * - telempostd, this daemon sends telemetry records to a telemetrics server
 *   (not included in this source tree), or spools the records on disk in case
 *   it's unable to successfully deliver them.
 *
@@ -0,0 +1,754 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2018 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <assert.h>
+#include <signal.h>
+#include <dirent.h>
+#include <malloc.h>
+#include <stdbool.h>
+#include <sys/stat.h>
+#include <curl/curl.h>
+#include <sys/signalfd.h>
+
+#include "log.h"
+#include "util.h"
+#include "spool.h"
+#include "iorecord.h"
+#include "retention.h"
+#include "telempostdaemon.h"
+
+/* spool window check */
+static bool inside_direct_spool_window(TelemPostDaemon *daemon, time_t current_time)
+{
+        return (current_time < daemon->bypass_http_post_ts + 1800) ? true : false;
+}
+
+/* set http bypass */
+static void start_network_bypass(TelemPostDaemon *daemon)
+{
+        daemon->bypass_http_post_ts = time(NULL);
+}
+
+/* burst limit check  */
+bool burst_limit_enabled(int64_t burst_limit)
+{
+        return (burst_limit > -1) ? true : false;
+}
+
+/* rate limit check */
+bool rate_limit_check(int current_minute, int64_t burst_limit, int
+                      window_length, size_t *array, size_t incValue)
+{
+        size_t count = 0;
+
+        /* Variable to determine last minute allowed in burst window */
+        int window_start = (TM_RATE_LIMIT_SLOTS + (current_minute - window_length + 1))
+                           % TM_RATE_LIMIT_SLOTS;
+
+        /* The modulo is not placed in the for loop itself, because if
+         * the current minute is less than the window start (from wrapping)
+         * the for loop will never be entered.
+         */
+
+        /* Counts all elements within burst_window in array */
+        for (int i = window_start; i < (window_start + window_length); i++) {
+                count += array[i % TM_RATE_LIMIT_SLOTS];
+
+                if ((array[i % TM_RATE_LIMIT_SLOTS] + incValue) >= SIZE_MAX) {
+                        /* Exceeds maximum size array can hold */
+                        return false;
+                }
+        }
+        /* Include current record being processed to count */
+        count += incValue;
+
+        /* Determines whether the count has exceeded the limit or not */
+        return (count > burst_limit) ? false : true;
+}
+
+/* spool strategy check */
+bool spool_strategy_selected(TelemPostDaemon *daemon)
+{
+        /* Performs action depending on strategy choosen */
+        return (strcmp(daemon->rate_limit_strategy, "spool") == 0) ? true : false;
+}
+
+/* rate limit check  */
+void rate_limit_update(int current_minute, int window_length, size_t *array,
+                       size_t incValue)
+{
+        /* Variable to determine the amount of zero'd out spots needed */
+        int blank_slots = (TM_RATE_LIMIT_SLOTS - window_length);
+
+        /* Update specified array depending on increment value */
+        array[current_minute] += incValue;
+        /* Zero out expired records for record array */
+        for (int i = current_minute + 1; i < (blank_slots + current_minute + 1); i++) {
+                array[i % TM_RATE_LIMIT_SLOTS] = 0;
+        }
+}
+
+static void set_pollfd(TelemPostDaemon *daemon, int fd, enum fdindex i, short events)
+{
+        assert(daemon);
+        assert(fd);
+
+        daemon->pollfds[i].fd = fd;
+        daemon->pollfds[i].events = events;
+        daemon->pollfds[i].revents = 0;
+}
+
+static void initialize_signals(TelemPostDaemon *daemon)
+{
+        int sigfd;
+        sigset_t mask;
+
+        sigemptyset(&mask);
+
+        if (sigaddset(&mask, SIGHUP) != 0) {
+                telem_perror("Error adding signal SIGHUP to mask");
+                exit(EXIT_FAILURE);
+        }
+
+        if (sigaddset(&mask, SIGINT) != 0) {
+                telem_perror("Error adding signal SIGINT to mask");
+                exit(EXIT_FAILURE);
+        }
+
+        if (sigaddset(&mask, SIGTERM) != 0) {
+                telem_perror("Error adding signal SIGTERM to mask");
+                exit(EXIT_FAILURE);
+        }
+
+        if (sigaddset(&mask, SIGPIPE) != 0) {
+                telem_perror("Error adding signal SIGPIPE to mask");
+                exit(EXIT_FAILURE);
+        }
+
+        if (sigprocmask(SIG_BLOCK, &mask, NULL) == -1) {
+                telem_perror("Error changing signal mask with SIG_BLOCK");
+                exit(EXIT_FAILURE);
+        }
+
+        sigfd = signalfd(-1, &mask, 0);
+        if (sigfd == -1) {
+                telem_perror("Error creating the signalfd");
+                exit(EXIT_FAILURE);
+        }
+
+        daemon->sfd = sigfd;
+
+        set_pollfd(daemon, sigfd, signlfd, POLLIN);
+}
+
+static void initialize_rate_limit(TelemPostDaemon *daemon)
+{
+        for (int i = 0; i < TM_RATE_LIMIT_SLOTS; i++) {
+                daemon->record_burst_array[i] = 0;
+                daemon->byte_burst_array[i] = 0;
+        }
+        daemon->rate_limit_enabled = rate_limit_enabled_config();
+        daemon->record_burst_limit = record_burst_limit_config();
+        daemon->record_window_length = record_window_length_config();
+        daemon->byte_burst_limit = byte_burst_limit_config();
+        daemon->byte_window_length = byte_window_length_config();
+        daemon->rate_limit_strategy = rate_limit_strategy_config();
+}
+
+static void initialize_record_delivery(TelemPostDaemon *daemon)
+{
+        daemon->record_retention_enabled = record_retention_enabled_config();
+        daemon->record_server_delivery_enabled = record_server_delivery_enabled_config();
+}
+
+void initialize_daemon(TelemPostDaemon *daemon)
+{
+        assert(daemon);
+
+        daemon->bypass_http_post_ts = 0;
+        daemon->is_spool_valid = is_spool_valid();
+        daemon->record_journal = open_journal(JOURNAL_PATH);
+        daemon->fd = inotify_init();
+        if (daemon->fd < 0) {
+                telem_perror("Error initializing inotify");
+                exit(EXIT_FAILURE);
+        }
+        daemon->wd = inotify_add_watch(daemon->fd, spool_dir_config(), IN_CLOSE_WRITE);
+
+        initialize_signals(daemon);
+        set_pollfd(daemon, daemon->fd, watchfd, POLLIN);
+
+        initialize_rate_limit(daemon);
+        initialize_record_delivery(daemon);
+        /* Register record retention delete action as a callback to prune entry */
+        if (daemon->record_journal != NULL && daemon->record_retention_enabled) {
+                daemon->record_journal->prune_entry_callback = &delete_record_by_id;
+        }
+        daemon->current_spool_size = 0;
+}
+
+size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
+{
+        telem_log(LOG_DEBUG, "Received data:\n%s\n", ptr);
+        return size * nmemb;
+}
+
+bool post_record_http(char *headers[], char *body)
+{
+        CURL *curl;
+        int res = 0;
+        char *content = "Content-Type: application/text";
+        struct curl_slist *custom_headers = NULL;
+        char errorbuf[CURL_ERROR_SIZE];
+        long http_response = 0;
+        const char *cert_file = get_cainfo_config();
+        const char *tid_header = get_tidheader_config();
+
+        // Initialize the libcurl global environment once per POST. This lets us
+        // clean up the environment after each POST so that when the daemon is
+        // sitting idle, it will be consuming as little memory as possible.
+        curl_global_init(CURL_GLOBAL_ALL);
+
+        curl = curl_easy_init();
+        if (!curl) {
+                telem_log(LOG_ERR, "curl_easy_init(): Unable to start libcurl"
+                          " easy session, exiting\n");
+                exit(EXIT_FAILURE);
+                /* TODO: check if memory needs to be released */
+        }
+
+        // Errors for any curl_easy_* functions will store nice error messages
+        // in errorbuf, so send log messages with errorbuf contents
+        curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorbuf);
+
+        curl_easy_setopt(curl, CURLOPT_URL, server_addr_config());
+        curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 5L);
+        curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10L);
+        curl_easy_setopt(curl, CURLOPT_POST, 1);
+#ifdef DEBUG
+        curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
+#endif
+
+        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);
+
+        for (int i = 0; i < NUM_HEADERS; i++) {
+                custom_headers = curl_slist_append(custom_headers, headers[i]);
+        }
+        custom_headers = curl_slist_append(custom_headers, tid_header);
+        // This should be set by probes/libtelemetry in the future
+        custom_headers = curl_slist_append(custom_headers, content);
+
+        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, custom_headers);
+        curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body);
+        curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, strlen(body));
+        curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_TRY);
+
+        if (strlen(cert_file) > 0) {
+                if (access(cert_file, F_OK) != -1) {
+                        curl_easy_setopt(curl, CURLOPT_CAINFO, cert_file);
+                        telem_log(LOG_INFO, "cafile was set to %s\n", cert_file);
+                }
+        }
+
+        telem_log(LOG_DEBUG, "Executing curl operation...\n");
+        errorbuf[0] = 0;
+        res = curl_easy_perform(curl);
+        curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_response);
+
+        if (res) {
+                size_t len = strlen(errorbuf);
+                if (len) {
+                        telem_log(LOG_DEBUG, "Failed sending record: %s%s", errorbuf,
+                                  ((errorbuf[len - 1] != '\n') ? "\n" : ""));
+                } else {
+                        telem_log(LOG_DEBUG, "Failed sending record: %s\n",
+                                  curl_easy_strerror(res));
+                }
+        } else if (http_response != 201 && http_response != 200) {
+                /*  201 means the record was  successfully created
+                 *  200 is a generic "ok".
+                 */
+                telem_log(LOG_ERR, "Encountered error %ld on the server\n",
+                          http_response);
+                // We treat HTTP error codes the same as libcurl errors
+                res = 1;
+        } else {
+                telem_log(LOG_INFO, "Record sent successfully\n");
+        }
+
+        curl_slist_free_all(custom_headers);
+        curl_easy_cleanup(curl);
+
+        curl_global_cleanup();
+
+        return res ? false : true;
+}
+
+static void save_local_copy(TelemPostDaemon *daemon, char *body)
+{
+        int ret = 0;
+        char *tmpbuf = NULL;
+        FILE *tmpfile = NULL;
+
+        if (daemon == NULL || daemon->record_journal == NULL ||
+            daemon->record_journal->latest_record_id == NULL) {
+                return;
+        }
+
+        ret = asprintf(&tmpbuf, "%s/%s", RECORD_RETENTION_DIR,
+                       daemon->record_journal->latest_record_id);
+        if (ret == -1) {
+                telem_log(LOG_ERR, "Failed to allocate memory for record full path, aborting\n");
+                return;
+        }
+
+        tmpfile = fopen(tmpbuf, "w");
+        if (!tmpfile) {
+                telem_perror("Error opening local record copy temp file");
+                goto save_err;
+        }
+
+        // Save body
+        fprintf(tmpfile, "%s\n", body);
+        fclose(tmpfile);
+
+save_err:
+        free(tmpbuf);
+        return;
+}
+
+static void save_entry_to_journal(TelemPostDaemon *daemon, time_t t_stamp, char *headers[])
+{
+        char *classification_value;
+        char *event_id_value;
+
+        if (get_header_value(headers[TM_CLASSIFICATION], &classification_value) &&
+            get_header_value(headers[TM_EVENT_ID], &event_id_value)) {
+                if (new_journal_entry(daemon->record_journal, classification_value, t_stamp, event_id_value) != 0) {
+                        telem_log(LOG_INFO, "new_journal_entry in process_record: failed saving record entry\n");
+                }
+        }
+        free(classification_value);
+        free(event_id_value);
+}
+
+/* Check window length conf values */
+static bool windows_length_value_check(TelemPostDaemon *daemon)
+{
+        if (daemon->record_window_length == -1 ||
+            daemon->byte_window_length == -1) {
+                telem_log(LOG_ERR, "Invalid value for window length\n");
+                return false;
+        }
+
+        return true;
+}
+
+/* Rate limiting checks */
+static void rate_limit_checks(TelemPostDaemon *daemon, bool *record_check_passed, bool *byte_check_passed)
+{
+        time_t temp = time(NULL);
+        struct tm *tm_s = localtime(&temp);
+        int current_minute;
+        current_minute = tm_s->tm_min;
+        bool record_burst_enabled = burst_limit_enabled(daemon->record_burst_limit);
+        bool byte_burst_enabled =  burst_limit_enabled(daemon->byte_burst_limit);
+
+        /* Checks if entirety of rate limiting is enabled */
+        if (daemon->rate_limit_enabled) {
+                /* Checks whether record and byte bursts are enabled individually */
+                record_burst_enabled = burst_limit_enabled(daemon->record_burst_limit);
+                byte_burst_enabled = burst_limit_enabled(daemon->byte_burst_limit);
+
+                if (record_burst_enabled) {
+                        *record_check_passed = rate_limit_check(current_minute,
+                                                                daemon->record_burst_limit, daemon->record_window_length,
+                                                                daemon->record_burst_array, TM_RECORD_COUNTER);
+                }
+                if (byte_burst_enabled) {
+                        *byte_check_passed = rate_limit_check(current_minute,
+                                                              daemon->byte_burst_limit, daemon->byte_window_length,
+                                                              daemon->byte_burst_array, RECORD_SIZE_LEN);
+                }
+                /* If both record and byte burst disabled, rate limiting disabled */
+                if (!record_burst_enabled && !byte_burst_enabled) {
+                        daemon->rate_limit_enabled = false;
+                }
+        }
+
+}
+
+/* Wrapper for save local copy */
+static void apply_retention_policies(TelemPostDaemon *daemon, char *body)
+{
+        if (daemon->record_retention_enabled) {
+                save_local_copy(daemon, body);
+        }
+}
+
+/* Deliver record to backend if rate limiting policies are met otherwise
+ * spool record for future delivery */
+static bool deliver_record(TelemPostDaemon *daemon, char *headers[], char *body)
+{
+
+        bool ret = false;
+        int current_minute;
+        bool do_spool = false;
+        bool record_sent = false;
+        time_t temp = time(NULL);
+        struct tm *tm_s = localtime(&temp);
+        current_minute = tm_s->tm_min;
+        /* Checks flags */
+        bool record_check_passed = true;
+        bool byte_check_passed = true;
+        bool record_burst_enabled = burst_limit_enabled(daemon->record_burst_limit);
+        bool byte_burst_enabled =  burst_limit_enabled(daemon->byte_burst_limit);
+
+        /* Perform record and byte rate limiting checks */
+        rate_limit_checks(daemon, &record_check_passed, &byte_check_passed);
+
+        /* Sends record if rate limiting is disabled, or all checks passed */
+        if (!daemon->rate_limit_enabled || (record_check_passed && byte_check_passed)) {
+                /* Send the record as https post */
+                record_sent = post_record_ptr(headers, body);
+                /**
+                 * This is the only point where an error condition could be returned
+                 * if the record was not sent
+                 * */
+                ret = record_sent;
+        }
+        // Get rate-limit strategy
+        do_spool = spool_strategy_selected(daemon);
+
+        // Drop record
+        if (!record_sent && !do_spool) {
+                // Not an error condition
+                ret = true;
+        }
+        // Spool Record
+        else if (!record_sent && do_spool) {
+                start_network_bypass(daemon);
+                telem_log(LOG_INFO, "process_record: initializing direct-spool window\n");
+                // False will keep record around
+                ret = false;
+        } else {
+                /* Updates rate limiting arrays if record sent */
+                if (record_burst_enabled) {
+                        rate_limit_update(current_minute, daemon->record_window_length,
+                                          daemon->record_burst_array, TM_RECORD_COUNTER);
+                }
+                if (byte_burst_enabled) {
+                        rate_limit_update(current_minute, daemon->byte_window_length,
+                                          daemon->byte_burst_array, RECORD_SIZE_LEN);
+                }
+        }
+
+        return ret;
+}
+
+bool process_staged_record(char *filename, bool is_retry, TelemPostDaemon *daemon)
+{
+        int k;
+        bool ret = false;
+        char *headers[NUM_HEADERS];
+        char *body = NULL;
+        struct stat buf;
+        time_t current_time = time(NULL);
+        int64_t max_spool_size = 0;
+
+        for (k = 0; k < NUM_HEADERS; k++) {
+                headers[k] = NULL;
+        }
+
+        /** Load record **/
+        if ((ret = read_record(filename, headers, &body)) == false) {
+                telem_log(LOG_WARNING, "unable to read record\n");
+                ret = true; // Record corrupted? true will remove record
+                goto end_processing_file;
+        }
+
+        /** Get file information  **/
+        if (stat(filename, &buf) == -1) {
+                telem_perror("Processing staged file unable to stat record in spool");
+                ret = true; // true to remove it
+                goto end_processing_file;
+        }
+
+        /** Update spool directory size **/
+        (daemon->current_spool_size) += (buf.st_blocks * 512);
+
+        /** Check that record is not expired **/
+        if (!S_ISREG(buf.st_mode) ||
+            (current_time - buf.st_mtime > (record_expiry_config() * 60)) ||
+            (buf.st_uid  != getuid())) {
+                ret = true; // Expired, true to remove it
+                goto end_processing_file;
+        }
+
+        /* Retries should not be recorded */
+        if (is_retry == false) {
+                /** Journal entry **/
+                save_entry_to_journal(daemon, current_time, headers);
+                /** Record retention **/
+                apply_retention_policies(daemon, body);
+        }
+
+        /** Record delivery **/
+        if (!daemon->record_server_delivery_enabled) {
+#ifdef DEBUG
+                telem_log(LOG_WARNING, "record server delivery disabled\n");
+#endif
+                // Not an error condition
+                ret = true;
+                goto end_processing_file;
+        }
+
+        /** Spool policies **/
+        if (inside_direct_spool_window(daemon, time(NULL))) {
+#ifdef DEBUG
+                telem_log(LOG_INFO, "process_record: delivering directly to spool\n");
+#endif
+                /* Check spool max size conf */
+                max_spool_size = spool_max_size_config();
+                if (max_spool_size != -1 &&
+                    daemon->current_spool_size >= (max_spool_size * 1024)) {
+                        // Drop record
+                        telem_log(LOG_INFO, "Spool dir full, dropping record\n");
+                        ret = true;
+                } else {
+                        // Keep record, non error condition
+                        ret = false;
+                }
+                goto end_processing_file;
+        }
+
+        /** Check window_length **/
+        if (windows_length_value_check(daemon) == false) {
+                exit(EXIT_FAILURE);
+        }
+
+        /** Deliver or spool **/
+        ret = deliver_record(daemon, headers, body);
+
+end_processing_file:
+        /** Update spool size if record will be removed **/
+        if (ret) {
+                (daemon->current_spool_size) -= (buf.st_blocks * 512);
+        }
+        telem_log(LOG_DEBUG, "spool_size: %ld\n", daemon->current_spool_size);
+        free(body);
+
+        for (k = 0; k < NUM_HEADERS; k++) {
+                free(headers[k]);
+        }
+
+        return ret;
+}
+
+static int directory_dot_filter(const struct dirent *entry)
+{
+        if ((strcmp(entry->d_name, ".") == 0) || (strcmp(entry->d_name, "..") == 0)) {
+                return 0;
+        } else {
+                return 1;
+        }
+}
+
+int staging_records_loop(TelemPostDaemon *daemon)
+{
+        int ret;
+        int processed;
+        int numentries;
+        struct dirent **namelist;
+
+        numentries = scandir(spool_dir_config(), &namelist, directory_dot_filter, NULL);
+        processed = 0;
+
+        if (numentries == 0) {
+                telem_log(LOG_DEBUG, "No entries in staging\n");
+                return numentries;
+        } else if (numentries < 0) {
+                telem_perror("Error while scanning staging");
+                return numentries;
+        }
+
+        for (int i = 0; i < numentries; i++) {
+                char *record_path;
+                telem_log(LOG_DEBUG, "Processing staged record: %s\n",
+                          namelist[i]->d_name);
+                ret = asprintf(&record_path, "%s/%s", spool_dir_config(), namelist[i]->d_name);
+                if (ret == -1) {
+                        telem_log(LOG_ERR, "Failed to allocate memory for staging record full path\n");
+                        exit(EXIT_FAILURE);
+                }
+                if (process_staged_record(record_path, true, daemon)) {
+                        unlink(record_path);
+                        processed++;
+                }
+                free(record_path);
+        }
+
+        for (int i = 0; i < numentries; i++) {
+                free(namelist[i]);
+        }
+        free(namelist);
+
+        return numentries - processed;
+}
+
+void run_daemon(TelemPostDaemon *daemon)
+{
+        int ret;
+        int retry_attempt = MAX_RETRY_ATTEMPTS;
+        int spool_process_time = spool_process_time_config();
+        bool daemon_recycling_enabled = daemon_recycling_enabled_config();
+        time_t last_spool_run_time = time(NULL);
+        time_t last_daemon_start_time = time(NULL);
+
+        assert(daemon);
+        assert(daemon->pollfds);
+        assert(daemon->pollfds[signlfd].fd);
+        assert(daemon->pollfds[watchfd].fd);
+
+        /* Post at boot failed initialize retries variable */
+        if (daemon->bypass_http_post_ts != 0) {
+                retry_attempt = 1;
+        }
+
+        while (1) {
+                int retry_delay = spool_process_time;
+                malloc_trim(0);
+
+                if (retry_attempt < MAX_RETRY_ATTEMPTS) {
+                        retry_delay = retry_attempt * retry_attempt;
+                        daemon->bypass_http_post_ts = 0;
+                        telem_log(LOG_INFO, "Record delivery failed will retry in %d seconds",
+                                  retry_delay);
+                }
+
+                ret = poll(daemon->pollfds, NFDS, retry_delay * 1000);
+                if (ret == -1) {
+                        telem_perror("Failed to poll daemon file descriptors");
+                        break;
+                } else if (ret != 0) {
+
+                        if (daemon->pollfds[signlfd].revents != 0) {
+                                struct signalfd_siginfo fdsi;
+                                ssize_t s;
+                                s = read(daemon->sfd, &fdsi, sizeof(struct signalfd_siginfo));
+                                if (s != sizeof(struct signalfd_siginfo)) {
+                                        telem_perror("Error while reading from the signal"
+                                                     "file descriptor");
+                                        exit(EXIT_FAILURE);
+                                }
+
+                                if (fdsi.ssi_signo == SIGTERM || fdsi.ssi_signo == SIGINT) {
+                                        telem_log(LOG_INFO, "Received either a \
+                                                                     SIGINT/SIGTERM signal\n");
+                                        break;
+                                }
+                        } else if (daemon->pollfds[watchfd].revents != 0) {
+
+                                int ret = 0;
+                                ssize_t i = 0;
+                                ssize_t length = 0;
+                                char buffer[BUFFER_LEN];
+
+                                length = read(daemon->fd, buffer, BUFFER_LEN);
+                                if (length < 0) {
+                                        telem_perror("Error while reading from inotify"
+                                                     "watcher");
+                                        exit(EXIT_FAILURE);
+                                }
+
+                                while (i < length) {
+                                        struct inotify_event *event = (struct inotify_event *)&buffer[i];
+
+                                        if (event->len) {
+                                                if (event->mask & IN_CLOSE_WRITE && !(event->mask & IN_ISDIR)) {
+                                                        char *record_name = NULL;
+
+                                                        /* Retrieve foldername from watch id?  */
+                                                        ret = asprintf(&record_name, "%s/%s", spool_dir_config(), event->name);
+                                                        if (ret == -1) {
+                                                                telem_log(LOG_ERR, "Failed to allocate memory for record full path, aborting\n");
+                                                                exit(EXIT_FAILURE);
+                                                        }
+                                                        /* Process inotify event */
+                                                        if (process_staged_record(record_name, false, daemon)) {
+                                                                unlink(record_name);
+                                                        }
+                                                        free(record_name);
+                                                }
+                                        }
+
+                                        i += (ssize_t)EVENT_SIZE + event->len;
+                                }
+                        }
+                } else {
+                        time_t now = time(NULL);
+                        /* time to recycle the daemon has elapsed*/
+                        if (daemon_recycling_enabled &&
+                            difftime(now, last_daemon_start_time) >= TM_DAEMON_EXIT_TIME) {
+                                /* Exit */
+                                telem_log(LOG_INFO, "Telemetry post daemon exiting for recycling\n");
+                                break;
+                        }
+                        /* Check if this is a retry */
+                        if (retry_attempt < MAX_RETRY_ATTEMPTS) {
+                                if (staging_records_loop(daemon) == 0) {
+                                        retry_attempt = MAX_RETRY_ATTEMPTS + 1;
+                                } else {
+                                        retry_attempt++;
+                                }
+                        } else if (retry_attempt == MAX_RETRY_ATTEMPTS) {
+                                retry_attempt = MAX_RETRY_ATTEMPTS + 1;
+                                telem_log(LOG_ERR, "Record deliver failed after %d attempts",
+                                          MAX_RETRY_ATTEMPTS);
+                        }
+                        /* Check spool  */
+                        if (difftime(now, last_spool_run_time) >= spool_process_time) {
+                                spool_records_loop(&(daemon->current_spool_size));
+                                last_spool_run_time = time(NULL);
+                        }
+                }
+
+                /* Check journal records and prune if needed */
+                ret = prune_journal(daemon->record_journal, JOURNAL_TMPDIR);
+                if (ret != 0) {
+                        telem_log(LOG_WARNING, "Unable to prune journal\n");
+                }
+        }
+}
+
+void close_daemon(TelemPostDaemon *daemon)
+{
+
+        if (daemon->fd) {
+                if (daemon->wd) {
+                        inotify_rm_watch(daemon->fd, daemon->wd);
+                }
+                close(daemon->fd);
+        }
+
+        close_journal(daemon->record_journal);
+}
+
+/* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
@@ -0,0 +1,134 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2018 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
+#define EVENT_SIZE sizeof(struct inotify_event)
+#define BUFFER_LEN 1024 * (EVENT_SIZE + 16)
+#define NFDS 2
+#define TM_RATE_LIMIT_SLOTS (1 /*h*/ * 60 /*m*/)
+#define TM_RECORD_COUNTER (1)
+#define MAX_RETRY_ATTEMPTS 8
+
+#include <poll.h>
+#include <stdbool.h>
+#include <sys/inotify.h>
+
+#include "common.h"
+#include "journal/journal.h"
+#include "configuration.h"
+
+enum fdindex {signlfd, watchfd};
+
+typedef struct TelemPostDaemon {
+        int fd;
+        int wd;
+        int sfd;
+        char event_buffer[BUFFER_LEN];
+        struct pollfd pollfds[NFDS];
+        /* Telemetry Journal*/
+        TelemJournal *record_journal;
+        /* Time of last failed post */
+        time_t bypass_http_post_ts;
+        /* Rate limit record and byte arrays */
+        size_t record_burst_array[TM_RATE_LIMIT_SLOTS];
+        size_t byte_burst_array[TM_RATE_LIMIT_SLOTS];
+        /* Rate Limit Configurations */
+        bool rate_limit_enabled;
+        int64_t record_burst_limit;
+        int record_window_length;
+        int64_t byte_burst_limit;
+        int byte_window_length;
+        const char *rate_limit_strategy;
+        /* Spool configuration */
+        bool is_spool_valid;
+        long current_spool_size;
+        /* Record local copy and delivery  */
+        bool record_retention_enabled;
+        bool record_server_delivery_enabled;
+} TelemPostDaemon;
+
+/**
+ * Initializes daemon
+ *
+ * @param daemon a pointer to telemetry post daemon
+ */
+void initialize_daemon(TelemPostDaemon *daemon);
+
+/**
+ * Starts daemon
+ *
+ * @param daemon a pointer to telemetry post daemon
+ */
+void run_daemon(TelemPostDaemon *daemon);
+
+/**
+ * Cleans up inotify descriptors
+ *
+ * @param daemon a pointer to telemetry post daemon
+ */
+void close_daemon(TelemPostDaemon *daemon);
+
+/**
+ * Processed record written on disk
+ *
+ * @param filename a pointor to record on disk
+ * @param is_retry a boolean value that indicates if
+ *        the record has been previously processed.
+ * @param daemon post to telemetry post daemon
+ */
+bool process_staged_record(char *filename, bool is_retry, TelemPostDaemon *daemon);
+
+/**
+ * Scans staging directory to process files that were
+ * missed by file watcher
+ *
+ * @param daemon a pointer to telemetry post daemon
+ * @return the number of records that were removed from spool
+ */
+int staging_records_loop(TelemPostDaemon *daemon);
+
+/**
+ * Posts a record to backend
+ *
+ * @param headers a pointer to an array with keys and values
+ * @param body a pointer to the payload
+ */
+bool post_record_http(char *headers[], char *body);
+
+/**
+ * Pointer to function to isolate backend call during
+ * unit testing.
+ *
+ * @param headers pointer to array of keys
+ * @param body a pinter to payload
+ * */
+extern bool (*post_record_ptr)(char *headers[], char *body);
+
+/** Helper functions **/
+/* rate limit check */
+bool rate_limit_check(int current_minute, int64_t burst_limit,
+                      int window_length, size_t *array, size_t incValue);
+
+/* burst limit check  */
+bool burst_limit_enabled(int64_t burst_limit);
+
+/* rate limit check  */
+void rate_limit_update(int current_minute, int window_length, size_t *array,
+                       size_t incValue);
+
+/* spool strategy check */
+bool spool_strategy_selected(TelemPostDaemon *daemon);
+
+/* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
@@ -34,8 +34,10 @@
 #include "common.h"
 #include "util.h"

-bool get_header(const char *haystack, const char *needle, char **line, size_t len)
+bool get_header(const char *haystack, const char *needle, char **line)
 {
+        size_t len = strlen(needle);
+
        if (haystack && !strncmp(haystack, needle, len)) {
                *line = strdup(haystack);
                return true;
@@ -22,7 +22,7 @@
 void *reallocate(void **addr, size_t *allocated, size_t requested);

 /* Check if haystacks begins with needle and return a copy of haystack */
-bool get_header(const char *haystack, const char *needle, char **line, size_t len);
+bool get_header(const char *haystack, const char *needle, char **line);

 /* Get the value of a header */
 bool get_header_value(const char *header, char **value);
@@ -26,7 +26,8 @@ static struct TelemJournal *journal = NULL;
 static char *eid = "00007766547776eb7fc478eb0eb43e43";
 static int K = 20;

-void teardown(void) {
+void teardown(void)
+{
        remove(journal_file);
 }

@@ -190,4 +190,3 @@ int main(void)
 }

 /* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
-
@@ -0,0 +1,443 @@
+/*
+ * This program is part of the Clear Linux Project
+ *
+ * Copyright 2018 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms and conditions of the GNU Lesser General Public License, as
+ * published by the Free Software Foundation; either version 2.1 of the License,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT ANY
+ * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ * A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
+ * details.
+ */
+
+#include <check.h>
+#include <sys/socket.h>
+#include <sys/fcntl.h>
+#include <stdlib.h>
+#include <sys/queue.h>
+#include <unistd.h>
+
+#include "configuration.h"
+#include "telempostdaemon.h"
+#include "common.h"
+
+TelemPostDaemon tdaemon;
+
+bool dummy_post(char *headers[], char *body)
+{
+        return true;
+}
+
+bool (*post_record_ptr)(char *headers[], char *body) = dummy_post;
+
+void setup(void)
+{
+        char *config_file = ABSTOPSRCDIR "/src/data/example.conf";
+        set_config_file(config_file);
+
+        initialize_daemon(&tdaemon);
+}
+
+START_TEST(check_daemon_is_initialized)
+{
+        setup();
+
+        ck_assert(tdaemon.rate_limit_enabled == true);
+        ck_assert(tdaemon.record_burst_limit == 100);
+        ck_assert(tdaemon.record_window_length == 15);
+        ck_assert(tdaemon.byte_burst_limit == 1000);
+        ck_assert(tdaemon.byte_window_length == 20);
+        ck_assert_str_eq(tdaemon.rate_limit_strategy, "spool");
+}
+END_TEST
+
+START_TEST(check_handle_client_with_no_data)
+{
+        setup();
+
+        bool success;
+        char *filename = ABSTOPSRCDIR "/tests/telempostd/empty_message";
+
+        success = process_staged_record(filename, false, &tdaemon);
+        // Return true to remove corrupted record
+        ck_assert(success == true);
+}
+END_TEST
+
+START_TEST(check_handle_client_with_incorrect_data)
+{
+        setup();
+
+        bool success;
+        char *filename = ABSTOPSRCDIR "/tests/telempostd/incorrect_message";
+
+        success = process_staged_record(filename, false, &tdaemon);
+        // Return true to remove corrupted record
+        ck_assert(success == true);
+}
+END_TEST
+
+START_TEST(check_process_record_with_correct_size_and_data)
+{
+        setup();
+
+        bool success;
+        char *filename = ABSTOPSRCDIR "/tests/telempostd/correct_message";
+
+        success = process_staged_record(filename, false, &tdaemon);
+        ck_assert(success == true);
+}
+END_TEST
+
+START_TEST(check_process_record_with_incorrect_headers)
+{
+        setup();
+
+        bool success;
+        char *filename = ABSTOPSRCDIR "/tests/telempostd/incorrect_headers";
+
+        success = process_staged_record(filename, false, &tdaemon);
+        // Return true to remove corrupted record
+        ck_assert(success == true);
+}
+END_TEST
+
+START_TEST(check_rate_limit_enabled_functions)
+{
+        setup();
+
+        bool record_check_passed = true;
+        bool byte_check_passed = true;
+        bool record_burst_enabled = true;
+        bool byte_burst_enabled = true;
+        tdaemon.record_burst_limit = 4;
+        tdaemon.byte_burst_limit = 2999;
+        tdaemon.rate_limit_enabled = true;
+
+        bool burst_limit_enabled(int64_t burst_limit)
+        {
+                return (burst_limit > -1) ? true : false;
+        }
+
+        /* Checks whether record burst rate limiting is enabled */
+        if (burst_limit_enabled(tdaemon.record_burst_limit)) {
+                record_check_passed = false;
+        } else {
+                /* If not enabled, does not need to be checked */
+                record_burst_enabled = false;
+        }
+        /* Checks whether byte burst rate limiting is enabled */
+        if (burst_limit_enabled(tdaemon.byte_burst_limit)) {
+                byte_check_passed = false;
+        } else {
+                /* If not enabled, does not need to be checked */
+                byte_burst_enabled = false;
+        }
+        if (!record_burst_enabled && !byte_burst_enabled) {
+                tdaemon.rate_limit_enabled = false;
+        }
+
+        ck_assert(record_check_passed == false);
+        ck_assert(byte_check_passed == false);
+        ck_assert(tdaemon.rate_limit_enabled == true);
+}
+END_TEST
+
+START_TEST(check_rate_limit_records_that_pass)
+{
+        setup();
+
+        int current_minute = 3;
+        tdaemon.record_burst_array[1] = 10;
+        tdaemon.record_burst_array[7] = 10;
+        tdaemon.record_burst_array[10] = 20;
+        tdaemon.record_burst_array[55] = 10;
+        tdaemon.record_burst_limit = 30;
+        tdaemon.record_window_length = 15;
+
+        bool checker;
+        checker = rate_limit_check(current_minute, tdaemon.record_burst_limit,
+                                   tdaemon.record_window_length, tdaemon.record_burst_array, 1);
+
+        ck_assert(checker == true);
+}
+END_TEST
+
+START_TEST(check_rate_limit_records_that_do_not_pass)
+{
+        setup();
+
+        int current_minute = 24;
+        tdaemon.record_burst_array[3] = 10;
+        tdaemon.record_burst_array[4] = 10;
+        tdaemon.record_burst_array[6] = 10;
+        tdaemon.record_burst_array[27] = 10;
+        tdaemon.record_burst_limit = 30;
+        tdaemon.record_window_length = 30;
+
+        bool checker;
+        checker = rate_limit_check(current_minute, tdaemon.record_burst_limit,
+                                   tdaemon.record_window_length, tdaemon.record_burst_array, 1);
+
+        ck_assert(checker == false);
+}
+END_TEST
+
+START_TEST(check_rate_limit_bytes_that_pass)
+{
+        setup();
+
+        size_t incValue = 20000;
+
+        int current_minute = 3;
+        tdaemon.byte_burst_array[1] = 32000;
+        tdaemon.byte_burst_array[7] = 10000;
+        tdaemon.byte_burst_array[10] = 45000;
+        tdaemon.byte_burst_array[55] = 11000;
+        tdaemon.byte_burst_limit = 64000;
+        tdaemon.byte_window_length = 15;
+
+        bool checker;
+        checker = rate_limit_check(current_minute, tdaemon.byte_burst_limit,
+                                   tdaemon.byte_window_length, tdaemon.byte_burst_array, incValue);
+
+        ck_assert(checker == true);
+}
+END_TEST
+
+START_TEST(check_rate_limit_bytes_that_do_not_pass)
+{
+        setup();
+
+        size_t incValue = 80000;
+
+        int current_minute = 15;
+        tdaemon.byte_burst_array[1] = 32000;
+        tdaemon.byte_burst_array[7] = 10000;
+        tdaemon.byte_burst_array[10] = 32300;
+        tdaemon.byte_burst_array[55] = 31000;
+        tdaemon.byte_burst_limit = 100000;
+        tdaemon.byte_window_length = 15;
+
+        bool checker;
+        checker = rate_limit_check(current_minute, tdaemon.byte_burst_limit,
+                                   tdaemon.byte_window_length, tdaemon.byte_burst_array, incValue);
+
+        ck_assert(checker == false);
+}
+END_TEST
+
+START_TEST(check_update_record_array)
+{
+        setup();
+        int current_minute = 7;
+        tdaemon.record_window_length = 15;
+        rate_limit_update(current_minute, tdaemon.record_window_length,
+                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
+
+        rate_limit_update(current_minute, tdaemon.record_window_length,
+                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
+
+        current_minute = 45;
+        rate_limit_update(current_minute, tdaemon.record_window_length,
+                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
+
+        current_minute = 47;
+        rate_limit_update(current_minute, tdaemon.record_window_length,
+                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
+        rate_limit_update(current_minute, tdaemon.record_window_length,
+                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
+
+        for (int i = 0; i < 60; i++) {
+                if (i != 45 && i != 47) {
+                        ck_assert_msg(tdaemon.record_burst_array[i] == 0, "array at %d is %d\n",
+                                      i, tdaemon.record_burst_array[i]);
+                } else {
+                        ck_assert_msg(tdaemon.record_burst_array[45] == 1, "array: %d is %d\n",
+                                      i, tdaemon.record_burst_array[i]);
+                        ck_assert_msg(tdaemon.record_burst_array[47] == 2, "array: %d is %d\n",
+                                      i, tdaemon.record_burst_array[i]);
+                }
+        }
+}
+END_TEST
+
+START_TEST(check_update_byte_array)
+{
+        setup();
+
+        int current_minute = 7;
+        tdaemon.byte_window_length = 15;
+        size_t record_size = 32000;
+
+        rate_limit_update(current_minute, tdaemon.byte_window_length,
+                          tdaemon.byte_burst_array, record_size);
+        rate_limit_update(current_minute, tdaemon.byte_window_length,
+                          tdaemon.byte_burst_array, record_size);
+
+        current_minute = 45;
+        rate_limit_update(current_minute, tdaemon.byte_window_length,
+                          tdaemon.byte_burst_array, record_size);
+
+        current_minute = 47;
+        rate_limit_update(current_minute, tdaemon.byte_window_length,
+                          tdaemon.byte_burst_array, record_size);
+        rate_limit_update(current_minute, tdaemon.byte_window_length,
+                          tdaemon.byte_burst_array, record_size);
+        rate_limit_update(current_minute, tdaemon.byte_window_length,
+                          tdaemon.byte_burst_array, record_size);
+
+        for (int i = 0; i < 60; i++) {
+                if (i != 45 && i != 47) {
+                        ck_assert_msg(tdaemon.byte_burst_array[i] == 0, "array at %d is %d\n",
+                                      i, tdaemon.record_burst_array[i]);
+                } else {
+                        ck_assert_msg(tdaemon.byte_burst_array[45] == 32000, "bytearray: %d is %d\n",
+                                      i, tdaemon.record_burst_array[i]);
+                        ck_assert_msg(tdaemon.byte_burst_array[47] == 96000, "bytearray: %d is %d\n",
+                                      i, tdaemon.record_burst_array[i]);
+                }
+        }
+}
+END_TEST
+
+START_TEST(check_strategy_spool_option)
+{
+        setup();
+
+        tdaemon.rate_limit_strategy = "spool";
+        bool do_spool = true;
+        char *ret = NULL;
+        bool record_sent = false;
+
+        // Get rate-limit strategy
+        do_spool = spool_strategy_selected(&tdaemon);
+
+        // Drop record
+        if (!record_sent && !do_spool) {
+                ret = "dropped";
+        }
+        // Spool Record
+        else if (!record_sent && do_spool) {
+                ret = "spooled";
+        } else {
+                ret = "sent";
+        }
+
+        ck_assert_str_eq(ret, "spooled");
+
+}
+END_TEST
+
+START_TEST(check_strategy_drop_option)
+{
+        setup();
+
+        tdaemon.rate_limit_strategy = "drop";
+        bool do_spool = true;
+        char *ret = NULL;
+        bool record_sent = false;
+
+        // Get rate-limit strategy
+        do_spool = spool_strategy_selected(&tdaemon);
+
+        // Drop record
+        if (!record_sent && !do_spool) {
+                ret = "dropped";
+        }
+        // Spool Record
+        else if (!record_sent && do_spool) {
+                ret = "spooled";
+        } else {
+                ret = "sent";
+        }
+
+        ck_assert_str_eq(ret, "dropped");
+
+}
+END_TEST
+
+START_TEST(check_strategy_if_record_sent)
+{
+        setup();
+
+        tdaemon.rate_limit_strategy = "drop";
+        bool do_spool = true;
+        char *ret = NULL;
+        bool record_sent = true;
+
+        // Get rate-limit strategy
+        do_spool = spool_strategy_selected(&tdaemon);
+
+        // Drop record
+        if (!record_sent && !do_spool) {
+                ret = "dropped";
+        }
+        // Spool Record
+        else if (!record_sent && do_spool) {
+                ret = "spooled";
+        } else {
+                ret = "sent";
+        }
+
+        ck_assert_str_eq(ret, "sent");
+
+}
+END_TEST
+
+Suite *config_suite(void)
+{
+        // A suite is comprised of test cases, defined below
+        Suite *s = suite_create("postd");
+
+        // Individual unit tests are added to "test cases"
+        TCase *t = tcase_create("postd");
+        tcase_add_test(t, check_daemon_is_initialized);
+        tcase_add_test(t, check_handle_client_with_no_data);
+        tcase_add_test(t, check_handle_client_with_incorrect_data);
+        tcase_add_test(t, check_process_record_with_correct_size_and_data);
+        tcase_add_test(t, check_process_record_with_incorrect_headers);
+        tcase_add_test(t, check_rate_limit_enabled_functions);
+        tcase_add_test(t, check_rate_limit_records_that_pass);
+        tcase_add_test(t, check_rate_limit_records_that_do_not_pass);
+        tcase_add_test(t, check_rate_limit_bytes_that_pass);
+        tcase_add_test(t, check_rate_limit_bytes_that_do_not_pass);
+        tcase_add_test(t, check_update_record_array);
+        tcase_add_test(t, check_update_byte_array);
+        tcase_add_test(t, check_strategy_spool_option);
+        tcase_add_test(t, check_strategy_drop_option);
+        tcase_add_test(t, check_strategy_if_record_sent);
+
+        suite_add_tcase(s, t);
+
+        return s;
+}
+
+int main(void)
+{
+        Suite *s;
+        SRunner *sr;
+
+        s = config_suite();
+        sr = srunner_create(s);
+
+        // Use the TAP driver for now, so that each
+        // unit test will PASS/FAIL in the log output.
+        srunner_set_log(sr, NULL);
+        srunner_set_tap(sr, "-");
+
+        srunner_run_all(sr, CK_SILENT);
+        // failed = srunner_ntests_failed(sr);
+        srunner_free(sr);
+
+        // if you want the TAP driver to report a hard error based
+        // on certain conditions (e.g. number of failed tests, etc.),
+        // return non-zero here instead.
+        return 0;
+}
+
+/* vi: set ts=8 sw=8 sts=4 et tw=80 cino=(0: */
@@ -26,7 +26,6 @@
 #include "common.h"

 TelemDaemon tdaemon;
-static bool post_was_called = false;

 char *get_serialized_record(char *headers, char *post_body, size_t *record_size)
 {
@@ -54,14 +53,6 @@ char *get_serialized_record(char *headers, char *post_body, size_t *record_size)
        return data;
 }

-bool dummy_post(char *headers[], char *body, bool spool)
-{
-        post_was_called = true;
-        return post_was_called;
-}
-
-bool (*post_record_ptr)(char *[], char *, bool) = dummy_post;
-
 void setup(void)
 {
        char *config_file = ABSTOPSRCDIR "/src/data/example.conf";
@@ -76,12 +67,6 @@ START_TEST(check_daemon_is_initialized)

        ck_assert(tdaemon.nfds == 0);
        ck_assert(tdaemon.pollfds == NULL);
-        ck_assert(tdaemon.rate_limit_enabled == true);
-        ck_assert(tdaemon.record_burst_limit == 100);
-        ck_assert(tdaemon.record_window_length == 15);
-        ck_assert(tdaemon.byte_burst_limit == 1000);
-        ck_assert(tdaemon.byte_window_length == 20);
-        ck_assert_str_eq(tdaemon.rate_limit_strategy, "spool");
 }
 END_TEST

@@ -269,12 +254,10 @@ START_TEST(check_handle_client_with_correct_size)
        memcpy(buf + 2 * sizeof(uint32_t), data, strlen(data) + 1);
        ck_assert(strcmp(data, buf + 2 * sizeof(int32_t)) == 0);

-        post_was_called = false;
        ssize_t ret = write(server_fd, buf, 2 * sizeof(uint32_t) + size + 1);
        ck_assert(ret != -1);
        processed = handle_client(&tdaemon, 0, cl);
        ck_assert(processed == true);
-        ck_assert(post_was_called == false);

        ck_assert_msg(is_client_list_empty(&(tdaemon.client_head)), "Failed to remove client with no data\n");
        ck_assert_msg(tdaemon.nfds == 0, "Failed to remove poll fd for client with n data\n");
@@ -307,14 +290,12 @@ START_TEST(check_process_record_with_correct_size_and_data)
        ck_assert_msg(cl != NULL, "failed to malloc client");
        add_pollfd(&tdaemon, client_fd, POLLIN | POLLPRI);

-        post_was_called = false;
        record = get_serialized_record(headers, post_body, &record_size);
        ssize_t ret = write(server_fd, record, record_size);
        ck_assert(ret == record_size);

        processed = handle_client(&tdaemon, 0, cl);
        ck_assert(processed == true);
-        ck_assert(post_was_called == true);
        ck_assert_msg(is_client_list_empty(&(tdaemon.client_head)), "Failed to remove client with correct data\n");
        ck_assert_msg(tdaemon.nfds == 0, "Failed to remove poll fd for client with correct data\n");
        close(server_fd);
@@ -341,14 +322,12 @@ START_TEST(check_process_record_with_incorrect_headers)
        ck_assert_msg(cl != NULL, "failed to malloc client");
        add_pollfd(&tdaemon, client_fd, POLLIN | POLLPRI);

-        post_was_called = false;
        record = get_serialized_record(headers, post_body, &record_size);
        ssize_t ret = write(server_fd, record, record_size);
        ck_assert(ret == record_size);

        processed = handle_client(&tdaemon, 0, cl);
        ck_assert(processed == true);
-        ck_assert(post_was_called == false);
        ck_assert_msg(is_client_list_empty(&(tdaemon.client_head)), "Failed to remove client with incorrect headers\n");
        ck_assert_msg(tdaemon.nfds == 0, "Failed to remove poll fd for client with incorrect headers\n");
        close(server_fd);
@@ -356,296 +335,13 @@ START_TEST(check_process_record_with_incorrect_headers)
 }
 END_TEST

-START_TEST(check_rate_limit_enabled_functions)
-{
-        setup();
-
-        bool record_check_passed = true;
-        bool byte_check_passed = true;
-        bool record_burst_enabled = true;
-        bool byte_burst_enabled = true;
-        tdaemon.record_burst_limit = 4;
-        tdaemon.byte_burst_limit = 2999;
-        tdaemon.rate_limit_enabled = true;
-
-        bool burst_limit_enabled(int64_t burst_limit)
-        {
-                return (burst_limit > -1) ? true : false;
-        }
-
-        /* Checks whether record burst rate limiting is enabled */
-        if (burst_limit_enabled(tdaemon.record_burst_limit)) {
-                record_check_passed = false;
-        } else {
-                /* If not enabled, does not need to be checked */
-                record_burst_enabled = false;
-        }
-        /* Checks whether byte burst rate limiting is enabled */
-        if (burst_limit_enabled(tdaemon.byte_burst_limit)) {
-                byte_check_passed = false;
-        } else {
-                /* If not enabled, does not need to be checked */
-                byte_burst_enabled = false;
-        }
-        if (!record_burst_enabled && !byte_burst_enabled) {
-                tdaemon.rate_limit_enabled = false;
-        }
-
-        ck_assert(record_check_passed == false);
-        ck_assert(byte_check_passed == false);
-        ck_assert(tdaemon.rate_limit_enabled == true);
-}
-END_TEST
-
-START_TEST(check_rate_limit_records_that_pass)
-{
-        setup();
-
-        int current_minute = 3;
-        tdaemon.record_burst_array[1] = 10;
-        tdaemon.record_burst_array[7] = 10;
-        tdaemon.record_burst_array[10] = 20;
-        tdaemon.record_burst_array[55] = 10;
-        tdaemon.record_burst_limit = 30;
-        tdaemon.record_window_length = 15;
-
-        bool checker;
-        checker = rate_limit_check(current_minute, tdaemon.record_burst_limit,
-                                   tdaemon.record_window_length, tdaemon.record_burst_array, 1);
-
-        ck_assert(checker == true);
-}
-END_TEST
-
-START_TEST(check_rate_limit_records_that_do_not_pass)
-{
-        setup();
-
-        int current_minute = 24;
-        tdaemon.record_burst_array[3] = 10;
-        tdaemon.record_burst_array[4] = 10;
-        tdaemon.record_burst_array[6] = 10;
-        tdaemon.record_burst_array[27] = 10;
-        tdaemon.record_burst_limit = 30;
-        tdaemon.record_window_length = 30;
-
-        bool checker;
-        checker = rate_limit_check(current_minute, tdaemon.record_burst_limit,
-                                   tdaemon.record_window_length, tdaemon.record_burst_array, 1);
-
-        ck_assert(checker == false);
-}
-END_TEST
-
-START_TEST(check_rate_limit_bytes_that_pass)
-{
-        setup();
-
-        size_t incValue = 20000;
-
-        int current_minute = 3;
-        tdaemon.byte_burst_array[1] = 32000;
-        tdaemon.byte_burst_array[7] = 10000;
-        tdaemon.byte_burst_array[10] = 45000;
-        tdaemon.byte_burst_array[55] = 11000;
-        tdaemon.byte_burst_limit = 64000;
-        tdaemon.byte_window_length = 15;
-
-        bool checker;
-        checker = rate_limit_check(current_minute, tdaemon.byte_burst_limit,
-                                   tdaemon.byte_window_length, tdaemon.byte_burst_array, incValue);
-
-        ck_assert(checker == true);
-}
-END_TEST
-
-START_TEST(check_rate_limit_bytes_that_do_not_pass)
-{
-        setup();
-
-        size_t incValue = 80000;
-
-        int current_minute = 15;
-        tdaemon.byte_burst_array[1] = 32000;
-        tdaemon.byte_burst_array[7] = 10000;
-        tdaemon.byte_burst_array[10] = 32300;
-        tdaemon.byte_burst_array[55] = 31000;
-        tdaemon.byte_burst_limit = 100000;
-        tdaemon.byte_window_length = 15;
-
-        bool checker;
-        checker = rate_limit_check(current_minute, tdaemon.byte_burst_limit,
-                                   tdaemon.byte_window_length, tdaemon.byte_burst_array, incValue);
-
-        ck_assert(checker == false);
-}
-END_TEST
-
-START_TEST(check_update_record_array)
-{
-        setup();
-        int current_minute = 7;
-        tdaemon.record_window_length = 15;
-        rate_limit_update(current_minute, tdaemon.record_window_length,
-                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
-
-        rate_limit_update(current_minute, tdaemon.record_window_length,
-                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
-
-        current_minute = 45;
-        rate_limit_update(current_minute, tdaemon.record_window_length,
-                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
-
-        current_minute = 47;
-        rate_limit_update(current_minute, tdaemon.record_window_length,
-                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
-        rate_limit_update(current_minute, tdaemon.record_window_length,
-                          tdaemon.record_burst_array, TM_RECORD_COUNTER);
-
-        for (int i = 0; i < 60; i++) {
-                if (i != 45 && i != 47) {
-                        ck_assert_msg(tdaemon.record_burst_array[i] == 0, "array at %d is %d\n",
-                                      i, tdaemon.record_burst_array[i]);
-                } else {
-                        ck_assert_msg(tdaemon.record_burst_array[45] == 1, "array: %d is %d\n",
-                                      i, tdaemon.record_burst_array[i]);
-                        ck_assert_msg(tdaemon.record_burst_array[47] == 2, "array: %d is %d\n",
-                                      i, tdaemon.record_burst_array[i]);
-                }
-        }
-}
-END_TEST
-
-START_TEST(check_update_byte_array)
-{
-        setup();
-
-        int current_minute = 7;
-        tdaemon.byte_window_length = 15;
-        size_t record_size = 32000;
-
-        rate_limit_update(current_minute, tdaemon.byte_window_length,
-                          tdaemon.byte_burst_array, record_size);
-        rate_limit_update(current_minute, tdaemon.byte_window_length,
-                          tdaemon.byte_burst_array, record_size);
-
-        current_minute = 45;
-        rate_limit_update(current_minute, tdaemon.byte_window_length,
-                          tdaemon.byte_burst_array, record_size);
-
-        current_minute = 47;
-        rate_limit_update(current_minute, tdaemon.byte_window_length,
-                          tdaemon.byte_burst_array, record_size);
-        rate_limit_update(current_minute, tdaemon.byte_window_length,
-                          tdaemon.byte_burst_array, record_size);
-        rate_limit_update(current_minute, tdaemon.byte_window_length,
-                          tdaemon.byte_burst_array, record_size);
-
-        for (int i = 0; i < 60; i++) {
-                if (i != 45 && i != 47) {
-                        ck_assert_msg(tdaemon.byte_burst_array[i] == 0, "array at %d is %d\n",
-                                      i, tdaemon.record_burst_array[i]);
-                } else {
-                        ck_assert_msg(tdaemon.byte_burst_array[45] == 32000, "bytearray: %d is %d\n",
-                                      i, tdaemon.record_burst_array[i]);
-                        ck_assert_msg(tdaemon.byte_burst_array[47] == 96000, "bytearray: %d is %d\n",
-                                      i, tdaemon.record_burst_array[i]);
-                }
-        }
-}
-END_TEST
-
-START_TEST(check_strategy_spool_option)
-{
-        setup();
-
-        tdaemon.rate_limit_strategy = "spool";
-        bool do_spool = true;
-        char *ret = NULL;
-        bool record_sent = false;
-
-        // Get rate-limit strategy
-        do_spool = spool_strategy_selected(&tdaemon);
-
-        // Drop record
-        if (!record_sent && !do_spool) {
-                ret = "dropped";
-        }
-        // Spool Record
-        else if (!record_sent && do_spool) {
-                ret = "spooled";
-        } else {
-                ret = "sent";
-        }
-
-        ck_assert_str_eq(ret, "spooled");
-
-}
-END_TEST
-
-START_TEST(check_strategy_drop_option)
-{
-        setup();
-
-        tdaemon.rate_limit_strategy = "drop";
-        bool do_spool = true;
-        char *ret = NULL;
-        bool record_sent = false;
-
-        // Get rate-limit strategy
-        do_spool = spool_strategy_selected(&tdaemon);
-
-        // Drop record
-        if (!record_sent && !do_spool) {
-                ret = "dropped";
-        }
-        // Spool Record
-        else if (!record_sent && do_spool) {
-                ret = "spooled";
-        } else {
-                ret = "sent";
-        }
-
-        ck_assert_str_eq(ret, "dropped");
-
-}
-END_TEST
-
-START_TEST(check_strategy_if_record_sent)
-{
-        setup();
-
-        tdaemon.rate_limit_strategy = "drop";
-        bool do_spool = true;
-        char *ret = NULL;
-        bool record_sent = true;
-
-        // Get rate-limit strategy
-        do_spool = spool_strategy_selected(&tdaemon);
-
-        // Drop record
-        if (!record_sent && !do_spool) {
-                ret = "dropped";
-        }
-        // Spool Record
-        else if (!record_sent && do_spool) {
-                ret = "spooled";
-        } else {
-                ret = "sent";
-        }
-
-        ck_assert_str_eq(ret, "sent");
-
-}
-END_TEST
-
 Suite *config_suite(void)
 {
        // A suite is comprised of test cases, defined below
-        Suite *s = suite_create("daemon");
+        Suite *s = suite_create("probd");

        // Individual unit tests are added to "test cases"
-        TCase *t = tcase_create("daemon");
+        TCase *t = tcase_create("probd");
        tcase_add_test(t, check_add_del_poll_fd);
        tcase_add_test(t, check_daemon_is_initialized);
        tcase_add_test(t, check_add_remove_client);
@@ -655,16 +351,6 @@ Suite *config_suite(void)
        tcase_add_test(t, check_handle_client_with_correct_size);
        tcase_add_test(t, check_process_record_with_correct_size_and_data);
        tcase_add_test(t, check_process_record_with_incorrect_headers);
-        tcase_add_test(t, check_rate_limit_enabled_functions);
-        tcase_add_test(t, check_rate_limit_records_that_pass);
-        tcase_add_test(t, check_rate_limit_records_that_do_not_pass);
-        tcase_add_test(t, check_rate_limit_bytes_that_pass);
-        tcase_add_test(t, check_rate_limit_bytes_that_do_not_pass);
-        tcase_add_test(t, check_update_record_array);
-        tcase_add_test(t, check_update_byte_array);
-        tcase_add_test(t, check_strategy_spool_option);
-        tcase_add_test(t, check_strategy_drop_option);
-        tcase_add_test(t, check_strategy_if_record_sent);

        suite_add_tcase(s, t);

@@ -12,7 +12,8 @@ TESTS = $(check_PROGRAMS) $(dist_check_SCRIPTS)

 check_PROGRAMS = \
 	%D%/check_config \
-	%D%/check_daemon \
+	%D%/check_probd \
+	%D%/check_postd \
 	%D%/check_probes \
 	%D%/check_ncb64  \
 	%D%/check_journal \
@@ -32,27 +33,63 @@ dist_check_SCRIPTS = \
 	@CHECK_LIBS@ \
 	$(top_builddir)/src/libtelem-shared.la

-%C%_check_daemon_SOURCES = \
-	%D%/check_daemon.c \
+%C%_check_probd_SOURCES = \
+	%D%/check_probd.c \
 	src/telemdaemon.c \
 	src/telemdaemon.h \
+	src/iorecord.h \
+	src/iorecord.c \
 	src/journal/journal.c \
 	src/journal/journal.h

-%C%_check_daemon_CFLAGS = \
+%C%_check_probd_CFLAGS = \
 	$(AM_CFLAGS) \
 	@CHECK_CFLAGS@ \
 	@CURL_CFLAGS@
-%C%_check_daemon_LDADD = \
+%C%_check_probd_LDADD = \
 	@CHECK_LIBS@ \
 	@CURL_LIBS@ \
 	$(top_builddir)/src/libtelem-shared.la

 if LOG_SYSTEMD
 if HAVE_SYSTEMD_JOURNAL
-%C%_check_daemon_CFLAGS += \
+%C%_check_probd_CFLAGS += \
        $(SYSTEMD_JOURNAL_CFLAGS)
-%C%_check_daemon_LDADD += \
+%C%_check_probd_LDADD += \
+        $(SYSTEMD_JOURNAL_LIBS)
+endif
+endif
+
+%C%_check_postd_SOURCES = \
+	%D%/check_postd.c \
+	src/spool.c \
+	src/iorecord.c \
+	src/retention.c \
+        src/telempostdaemon.c \
+        src/telempostdaemon.h \
+        src/journal/journal.c \
+        src/journal/journal.h
+
+EXTRA_DIST += \
+	%D%/telempostd/correct_message \
+	%D%/telempostd/incorrect_headers \
+	%D%/telempostd/empty_message \
+	%D%/telempostd/incorrect_message
+
+%C%_check_postd_CFLAGS = \
+        $(AM_CFLAGS) \
+        @CHECK_CFLAGS@ \
+        @CURL_CFLAGS@
+%C%_check_postd_LDADD = \
+        @CHECK_LIBS@ \
+        @CURL_LIBS@ \
+        $(top_builddir)/src/libtelem-shared.la
+
+if LOG_SYSTEMD
+if HAVE_SYSTEMD_JOURNAL
+%C%_check_postd_CFLAGS += \
+        $(SYSTEMD_JOURNAL_CFLAGS)
+%C%_check_postd_LDADD += \
        $(SYSTEMD_JOURNAL_LIBS)
 endif
 endif
@@ -0,0 +1,16 @@
+record_format_version: 1
+classification: crash/kernel/bug
+severity: 0
+machine_id: 1234
+creation_timestamp: 1418672344
+arch: x86_64
+host_type: macbookpro
+build: 200
+kernel_version: 3.15
+payload_format_version: 1
+system_name: clear-linux-os
+board_name: Qemu|Intel
+cpu_model: Intel(R) Core(TM) i7-5650U CPU @ 2.20GHz
+bios_version: Qemu
+event_id: 3a2d799826edc6266d72824d2aac6763
+test message
@@ -0,0 +1,10 @@
+record_format_version: 1
+nclassificatioooon: crash/kernel/bug
+severity: 0
+machine_id: 1234
+creation_timestamp: 1531420323
+arch: x86_64
+host_type: macbookpro
+build: 23550
+kernel_version: 4.17.4-589.native
+test message
@@ -0,0 +1 @@
+test
Author	SHA1	Message	Date
Alex Jaramillo	d8d3cdee49	Release v2.0.1 Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-09-25 15:57:03 +00:00
Alex Jaramillo	839ee5c126	Retry logic for post failure on boot This change alters the logic of post telemetry message to backend to handle cases when the network or name resolution service is not available when the daemon starts. Currently if a message is not delivered it will be spooled and it should have to wait 900 seconds (default value) for a future loop to process this message. With this change when the daemon is started and is unable to deliver telemetry messages it will atempt to re-send those messages using a t*t delay in seconds up to t = 7 (or 1, 4, 9, 16, 25, 36, 49 seconds). Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-09-24 15:18:12 -07:00
Alex Jaramillo	a08a89ab7f	Fix send record check bug This change fixes a bug where the telem-record-gen probe returns 1 though the execution of the probe was successful. Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-09-06 13:33:39 -07:00
Alex Jaramillo	f81e4c7fba	Telemetry refactoring This change adds a daemon that handles telemetry record retention, record reporting, and record limiting policies. This new daemon receives records in a folder that is monitored by inotify. Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-09-06 09:57:01 -07:00
Alex Jaramillo	0d07d50a1f	Addressing comments	2018-08-27 10:40:12 -07:00
Alex Jaramillo	3b096674dc	Addresing comment	2018-08-27 10:40:12 -07:00
Alex Jaramillo	c946c7dae9	Clip new line in model string	2018-08-27 10:40:12 -07:00
Alex Jaramillo	91a5c82bd3	Fix memory leak	2018-08-27 10:40:12 -07:00
Alex Jaramillo	c02f74be5b	Add print only option	2018-08-27 10:40:12 -07:00
Patrick McCarty	2082befdb7	Revert "Removing crash dump configuration" Installing a crashprobe sysctl conf file by default is, in my opinion, the most useful way to enable the crashprobe for Linux distributions. The fact that Clear Linux does not use it anymore does not necessarily dictate policy for telemetrics-client itself. My preference would be to simply exclude the conf file in the Clear Linux telemetrics-client package, since that is very easy to do with autospec. Also, the new crash handling implementation in Clear Linux requires a systemd patch that won't be upstreamed, so this is another good reason to install the conf by default. This reverts commit `d1425173a9`. Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>	2018-07-30 16:20:05 -07:00
Patrick McCarty	f2206d1cc8	crash probe: retry processing when errors occur It is possible that transient errors may be raised when processing the core file, so speculatively retry for any processing errors that appear for the first run. Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>	2018-07-27 20:54:49 -07:00
Alex Jaramillo	d1425173a9	Removing crash dump configuration This change removes the crash probe exclusive use of the core dump reported by the kernel. Other component above telemetry will handle this to properly report core dumps to telemetry (when available). Signed-off-by: Alex Jaramillo <alex.v.jaramillo@intel.com>	2018-07-27 15:25:01 -07:00