diff --git a/source/clear-linux/guides/guides.rst b/source/clear-linux/guides/guides.rst index f2c65575..963653f3 100644 --- a/source/clear-linux/guides/guides.rst +++ b/source/clear-linux/guides/guides.rst @@ -19,3 +19,4 @@ after completing the |CL| :ref:`installation `. maintenance/maintenance network/network deploy-at-scale + telemetry diff --git a/source/clear-linux/guides/telemetry/telemctl.rst b/source/clear-linux/guides/telemetry/telemctl.rst new file mode 100644 index 00000000..4fa0599c --- /dev/null +++ b/source/clear-linux/guides/telemetry/telemctl.rst @@ -0,0 +1,71 @@ +.. _telemctl: + +Using telemctl options +###################### + +The |CL-ATTR| telemetry client provides an admin tool called telemctl for +managing the telemetry services and probes. The tool is located in +:file:` /usr/bin`. Running it with no argument results in the following: + +.. code-block:: bash + + sudo telemctl + +.. code-block:: console + + /usr/bin/telemctl - Control actions for telemetry services + stop Stops all running telemetry services + start Starts all telemetry services + restart Restarts all telemetry services + is-active Checks if telemprobd and telempostd are active + opt-in Opts in to telemetry, and starts telemetry services + opt-out Opts out of telemetry, and stops telemetry services + journal Prints telemetry journal contents. Use -h argument for more + options + +telemctl commands: +****************** + +start/stop/restart +================== + +The commands to start, stop and restart the telemetry services manage all +required services and probes on the system. There is no need to separately +start/stop/restart the two client daemons **telemprobd** and **telempostd**. +The **restart** command option will call **telemctl stop** followed by ** +telemctl start** + +is-active +========= + +The `is-active` option reports whether the two client daemons are active. +This is useful to verify that the **opt-in** and **opt-out** options have +taken effect, or to ensure that telemetry is functioning on the system. +Note that both daemons are verified. + +.. code-block:: bash + + sudo telemctl is-active + +.. code-block:: console + + telemprobd : active + telempostd : active + +.. include:: ./telemetry-enable.rst + :start-after: incl-opt-in-out-telemetry: + :end-before: Remove the telemetry software bundle + +.. note:: + + To opt-in but not immediately start telemetry services, run the command + :command:`sudo telemctl stop` after the :command:`opt-in` command is + entered. Once you are ready to start the service, enter the command + :command:`sudo telemctl start`. + +Next steps +========== + +Learn to read records: + +* :ref:`telemetry-journal` diff --git a/source/clear-linux/guides/telemetry/telemetry-api.rst b/source/clear-linux/guides/telemetry/telemetry-api.rst new file mode 100644 index 00000000..595ac8fd --- /dev/null +++ b/source/clear-linux/guides/telemetry/telemetry-api.rst @@ -0,0 +1,170 @@ +.. _telemetry-api + +The telemetry API +################## + +Installing the ``telemetry`` bundle includes the libtelemetry C library, +which exposes an API used by the telemprobd and telempostd daemons. You +can use these in your applications as well. The API documentation is found +in the :file:`telemetry.h` file in `Telemetrics client`_ repository. + +Creating records with telem-record-gen +************************************** + +The telemetrics-client package provides a record generator tool called +``telem-record-gen``. This tool can be used to create records from shell +scripts, etc., when writing a probe in C is not desirable. Records are sent +to the backend server, and can also be echoed to stdout. + +telem-record-gen usage +====================== + +.. code-block:: bash + + telem-record-gen [OPTIONS] - create and send a custom telemetry record + +.. code-block:: console + + Help Options: + -h, --help Show help options + + Application Options: + -V, --version Print the program version + -s, --severity Severity level (1-4) - (default 1) + -c, --class Classification level_1/level_2/level_3 (required) + -p, --payload Record body (max size = 8k) (required) + -P, --payload-file File to read payload from + -R, --record-version Version number for format of payload (default 1) + -e, --event-id Event id to use in the record + -o, --echo Echo record to stdout + -n, --no-post Do not post record just print + +The :command:`-c` and :command:`-p` options are required; defaults are +supplied for most other options. The maximum payload size is 8k +(8192 bytes). Excess is ignored, regardless of source (file/commandline/ +stdin). An empty payload is allowed, but even an empty payload must be +specified in one of the three ways shown below. + +telem-record-gen examples +========================= + +There are three ways to supply the payload to the record. + +#. On the command line, use the :command:`-p ` option: + + .. code-block:: bash + + telem-record-gen -c a/b/c -n -o -p 'payload goes here' + + .. code-block:: console + + record_format_version: 4 + classification: a/b/c + severity: 1 + machine_id: FFFFFFFF + creation_timestamp: 1539023189 + arch: x86_64 + host_type: innotek GmbH|VirtualBox|1.2 + build: 25180 + kernel_version: 4.14.71-404.lts + payload_format_version: 1 + system_name: clear-linux-os + board_name: VirtualBox|Oracle Corporation + cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz + bios_version: VirtualBox + event_id: 2236710e4fc11e4a646ce956c7802788 + + payload goes here + +#. Specify a file that contains the payload with the option + :command:'-P path/to/file'. + + .. code-block:: bash + + telem-record-gen -c a/b/c -n -o -P ./payload_file.txt + + .. code-block:: console + + record_format_version: 4 + classification: a/b/c + severity: 1 + machine_id: FFFFFFFF + creation_timestamp: 1539023621 + arch: x86_64 + host_type: innotek GmbH|VirtualBox|1.2 + build: 25180 + kernel_version: 4.14.71-404.lts + payload_format_version: 1 + system_name: clear-linux-os + board_name: VirtualBox|Oracle Corporation + cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz + bios_version: VirtualBox + event_id: d73d6040afd7693cccdfece479df9795 + + payload read from file + +#. If the :command:`-p` or :command:`-P` options are absent, the tool reads + from stdin so you can use it in a HEREDOC in scripts. + + .. code-block:: bash + + telem-record-gen -c a/b/c -n -o << HEOF + payload read from stdin + HEOF + + .. code-block:: console + + record_format_version: 4 + classification: a/b/c + severity: 1 + machine_id: FFFFFFFF + creation_timestamp: 1539023621 + arch: x86_64 + host_type: innotek GmbH|VirtualBox|1.2 + build: 25180 + kernel_version: 4.14.71-404.lts + payload_format_version: 1 + system_name: clear-linux-os + board_name: VirtualBox|Oracle Corporation + cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz + bios_version: VirtualBox + event_id: 2f070e8e71679f2b1f28794e3a6c42ee + + payload read from stdin + + .. note:: + + Although only the classification and payload are specified, the tool supplies values for the remaining values. + +Telemetry records and the REST API +================================== + +If you have not configured the telemetry client to keep records locally, you +can view them using the Web UI of the server, or you can query them from the +server using the REST API provided by |CL| telemetrics. The API is +available at :file:`/api/records`, and when queried, returns a JSON +response that contains a list of records. There are several parameters for +filtering queries, similar to the filters available through the telemetryui Records view. + +* classification: The classification of the record +* severity: The severity of the record. Restricted to integer value +* machine_id: The id of the machine where this record was generated on +* build: The build on which the record was generated. Restricted to 256 + characters. +* created_in_days: causes the query to return records created after the last + given days +* created_in_sec: returns the records created after the last given seconds +* limit: The maximum number of records to be returned. + +Next Steps +========== + +* :ref:`telemetry-backend` +* `Telemetrics client`_ + +Related topics +============== + +* :ref:`telemetry-about` + +.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/ \ No newline at end of file diff --git a/source/clear-linux/guides/telemetry/telemetry-config.rst b/source/clear-linux/guides/telemetry/telemetry-config.rst new file mode 100644 index 00000000..4a09d914 --- /dev/null +++ b/source/clear-linux/guides/telemetry/telemetry-config.rst @@ -0,0 +1,101 @@ +.. _telemetry-config: + +Telemetry Client Configuration +############################## + +The telemetry client will look for the configuration file located at +:file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the +file does not exist, the client will use the default configuration located +at :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. To modify or +customize the configuration, copy the file from +:file:`/usr/share/defaults/telemetrics` to +:file:`/etc/telemetrics` and edit it. + +Configuration Options +********************* +The client uses the following configuration options from the config file: + +* **server**: This specifies the web server to which telempostd sends the + telemetry records. +* **socket_path**: This specifies the path of the unix domain socket that + the telemprobd listens on for connections from the probes. +* **spool_dir**: This configuration option is related to spooling. If the + daemon is not able to send the telemetry records to the backend server due + to reasons such as the network availability, then it stores the records in + a spool directory. This option specifies that path of the spool directory. + This directory should be owned by the same user as the daemon. + + - mkdir -p /var/spool/telemetry + - chown -R telemetry:telemetry /var/spool/telemetry + - systemctl restart telemprobd.service + +* **record_expiry**: This is the time in minutes after which the records in + the spool directory are deleted by the daemon. +* **spool_process_time**: This specifies the time interval in seconds that + the daemon waits for before checking the spool directory for records. The + daemon picks up the records in the order of modification date and tries to + send the record to the server. It sends a maximum of 10 records at a time. + If it was able to send a record successfully, it deletes the record from + the spool. If the daemon finds a record older than the "record_expiry" + time, then it deletes that record. The daemon looks at a maximum of 20 + records in a single spool run loop. +* **rate_limit_enabled**: This determines whether rate-limiting is enabled or + disabled. When enabled, there is a threshold on both records sent within a + window of time, and record bytes sent within a window a time. +* **record_burst_limit**: This is the maximum amount of records allowed to be + passed by the daemon within the record_window_length of time. If set to + -1, the rate-limiting for record bursts is disabled. +* **record_window_length**: The time in minutes (0-59) that + establishes the window length for the record_burst_limit. EX: if + record_burst_window=1000 and record_window_length=15, then no more than + 1000 records can be passed within any given fifteen minute window. +* **byte_burst_limit**: This is the maximum amount of bytes that can be + passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled. +* **byte_window_length**: This is the time, in minutes (0-59), that + establishes the window length for the byte_burst_limit. +* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting + threshold has been reached. Currently the options are 'drop' or 'spool', + with spool being the default. If spool is chosen, records will be spooled + and sent at a later time. +* **record_retention_enabled**: When this key is enabled (true) the daemon + saves a copy of the payload on disk from all valid records. To avoid the + excessive use of disk space only the latest 100 records are kept. The + default value for this configuration key is false. +* **record_server_delivery_enabled**: This key controls the delivery of + records to server; when enabled (default value), the record will be posted + to the address in the configuration file. If this configuration key is + disabled (false), records will not be spooled or posted to backend. This + configuration key can be used in combination with record_retention_enabled + to keep copies of telemetry records locally only. + + .. note:: + + Configuration options may change as the telemetry client evolves. + Please use the comments in the file itself as the most accurate + reference for configuration. + +Setting a static machine id +=========================== + +The machine id reported by the telemetry client is rotated every 3 days for +privacy reasons. If you wish to have a static machine id for testing +purposes, you can opt in by creating a static machine id file named +"opt-in-static-machine-id" under the directory :file:`/etc/telemetrics/`. +Where "unique machine ID" is your desired static machine ID. + +.. code-block:: bash + + sudo mkdir -p /etc/telemetrics + +.. code-block:: bash + + sudo echo "unique machine ID" > /etc/telemetrics/opt-in-static-machine-id + +.. note:: + + The machine id mentioned here is not the same as the system hostname. Learn how to :ref:`hostname`: + +Next steps +========== + +* :ref:`telemctl` diff --git a/source/clear-linux/guides/telemetry/telemetry-enable.rst b/source/clear-linux/guides/telemetry/telemetry-enable.rst new file mode 100644 index 00000000..caeccda6 --- /dev/null +++ b/source/clear-linux/guides/telemetry/telemetry-enable.rst @@ -0,0 +1,114 @@ +.. _telemetry-enable: + +Install and enable telemetry in |CL-ATTR| +######################################### + +Telemetry enables developers to observe and proactively address issues on +|CL-ATTR| before end users are impacted. The telemetry functionality +is maintained in the ``telemetrics`` software bundle. + +.. note:: + + The telemetry functionality adheres to `Intel privacy policies`_ + regarding the collection and use of :abbr:`PII (Personally Identifiable + Information)` and is open source. Specifically, no intentionally + identifiable information about the user or system owner is collected. + +End users may enable or disable the telemetry component of |CL| or even +redirect where the records go if they wish to collect records for themselves. + +Install the telemetry software bundle +************************************* + +During the initial installation of |CL|, you are requested to join the +stability enhancement program and allow |CL| to collect anonymous reports +to improve system stability. If you choose not to join this program, then the +telemetry software bundle is not added to your system. + +To install the telemetry bundle, enter the following command as either the +root user or with :command:`sudo` privileges: + +.. code-block:: bash + + sudo swupd bundle-add telemetrics + +This adds the telemetrics-client to your system, and you will automatically +opt-in for the service. + +Enable telemetry +================ + +To start telemetry on your system, run the following command: + +.. code-block:: bash + + sudo telemctl start + +This enables and starts the :command:`telemprobd` and :command:`telempostd` +daemons. Your system will begin to send telemetry data to the server defined +in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not +exist, the :command:`telemprobd` and :command:`telempostd` daemons will use +the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. + +Disable telemetry +================= + +To disable both of the telemetry daemons, run the following command: + +.. code-block:: bash + + sudo telemctl stop + +.. _incl-opt-in-out-telemetry: + +Opt-in to telemetry +=================== + +To opt-in to the telemetry services, simply enter the opt-in +command and start the service: + +.. code-block:: bash + + sudo telemctl opt-in + +This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists, +and starts the telemetry services. + +.. note:: + + To opt-in but not immediately start telemetry services, you will need to + run the command :command:`sudo telemctl stop` after the :command:`opt-in` + command is entered. Once you are ready to start the service, enter the + command :command:`sudo telemctl start`. + +Opt-out of telemetry +==================== + +To stop sending telemetrics data from your system, opt out of the +telemetry service: + +.. code-block:: bash + + sudo telemctl opt-out + +This creates the file :file:`/etc/telemetrics/opt-out` and stops the +telemetry services. + +.. _incl-opt-in-out-telemetry-end: + +Remove the telemetry software bundle +==================================== + +To completely remove telemetrics from your system, use the :command:`swupd` +command to remove the telemetry software bundle: + +.. code-block:: bash + + sudo swupd bundle-remove telemetrics + +Next steps +========== + +* :ref:`telemetry-config` + +.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html diff --git a/source/clear-linux/guides/telemetry/telemetry-journal.rst b/source/clear-linux/guides/telemetry/telemetry-journal.rst new file mode 100644 index 00000000..1cc38b68 --- /dev/null +++ b/source/clear-linux/guides/telemetry/telemetry-journal.rst @@ -0,0 +1,108 @@ +.. _telemetry-journal: + +Using telemctl journal +###################### + +The telemctl ``journal`` command gives you access to features and options of +the telemetry journal to assist with system analytics and debug. The +:command:`sudo telemctl journal` has a number of options to help filter +records. Use :command:`-h` or :command:`--help` to view usage options. + +.. code-block:: bash + + sudo telemctl journal -h + +:: + + -r, --record_id Print record with specific record_id + -e, --event_id Print records with specific event_id + -c, --classification Print records with specific classification + -b, --boot_id Print records with specific boot_id + -i, --include_record Include record content + -V, --verbose Verbose output + -h, --help Display this help message + +Journal Output +************** + +To see the listing of records in the journal, run the command: + +.. code-block:: bash + + sudo telemctl journal -V + +This will produce output like the following: + +.. list-table:: **Table 1. Using -V for Verbose output** + :widths: 10 30 20 20 20 + :header-rows: 1 + + * - Classification + - Time stamp + - Record ID + - Event ID + - Boot ID + + * - org.clearlinux/heartbeat/ping + - Fri 2018-09-21 00:00:57 UTC + - 269e8e4026e6aa440c4d2ed71e38efcd + - b3a51b5e62a008ed0b56d1740be67d48 + - 853a75aa-da3b-4356-a085-079abab3ffe1 + + * - org.clearlinux/hello/world + - Fri 2018-09-21 17:53:21 UTC + - b06c8d31adf5ccc7d5d3f8959d8d3e72 + - 57c64c79a9b911d68f4dab10a00267d7 + - 853a75aa-da3b-4356-a085-079abab3ffe1 + + * - org.clearlinux/crash/clr + - Fri 2018-09-21 17:57:59 UTC + - b62cd4278672ae3331cf121bc7a8e1c6 + - b6adb5751382c48eebb7ee007fe1790a + - 853a75aa-da3b-4356-a085-079abab3ffe1 + +Each line gives information about a distinct record. The :command:`-V` or +:command:`--verbose` option adds the header to identify the Classification, +Time Stamp, Record ID, Event ID and Boot ID for each record. The journal +feature can filter records according to the Classification, Record ID, Event +ID and Boot ID by using the :command:`-c`,:command:`-r`, :command:`-e` and +:command:`-b` options accordingly. + +Payload Information +******************** + +From the previous output, you may want to get more information about the +record with the "org.clearlinux/crash/clr" classification to help debug a +crash. You can use the :command:`-c` and :command:`-i` options to see the payload of the record, like this: + +.. code-block:: bash + + sudo telemctl journal -c org.clearlinux/crash/clr -i + +.. code-block:: console + + org.clearlinux/crash/clr Tue 2018-09-25 18:43:50 UTC 07ae583edbd13829965d67e9ba97d70c 69c600470769c841649266178375d67e d32c13d1-fda0-49c6-8431-e6c5b29cbefa + Process: /usr/bin/bash + PID: 685 + Signal: 11 + + Backtrace (TID 685): + #0 kill() - [libc.so.6] + #1 bash_tilde_expand() - [/usr/bin/bash] + #2 maybe_execute_file() - [/usr/bin/bash] + #3 main() - [/usr/bin/bash] + #4 __libc_start_main() - [libc.so.6] + #5 _start() - [/usr/bin/bash] + +If you have records of multiple crashes, you can use the :command:'-r' +option to specify the record more precisely, rather than going by +classification. You can also specify a classification of record and use the +:command:'-i' option to see the payload of each record with that +classification. + +Next steps +========== + +Learn to read records: + +* :ref:`telemetry-api` diff --git a/source/clear-linux/guides/telemetry/telemetry.rst b/source/clear-linux/guides/telemetry/telemetry.rst new file mode 100644 index 00000000..ef2ac5d9 --- /dev/null +++ b/source/clear-linux/guides/telemetry/telemetry.rst @@ -0,0 +1,19 @@ +.. _telemetry: + +Telemetrics in |CL-ATTR| +######################## + +The |CL-ATTR| telemetrics solution collects data from running |CL| systems +and helps to quickly identify and fix bugs in the OS. These guides will walk +you through setup, configuration, and customization of the telemetry client. The data collected from the client system is analyzed and presented by the +telemetry backend solution. For more details, learn how to +:ref:`telemetry-backend`. + +.. toctree:: + :maxdepth: 1 + + telemetry-enable + telemetry-config + telemctl + telemetry-journal + telemetry-api