merged with upstream

This commit is contained in:
Kevin Putnam
2018-11-08 13:28:50 -08:00
31 changed files with 742 additions and 283 deletions
+1
View File
@@ -21,6 +21,7 @@ Clear Linux Tooling
:glob:
clearlinux/*
telemetrics/telemetrics
Maintenance
===========
@@ -0,0 +1,64 @@
.. _telemctl:
telemctl options
################
The |CL-ATTR| telemetry client provides an admin tool called telemctl for
managing the telemetry services and probes. The tool is located in
:file:` /usr/bin`. Running it with no argument results in the following:
.. code-block:: bash
sudo telemctl
.. code-block:: console
/usr/bin/telemctl - Control actions for telemetry services
stop Stops all running telemetry services
start Starts all telemetry services
restart Restarts all telemetry services
is-active Checks if telemprobd and telempostd are active
opt-in Opts in to telemetry, and starts telemetry services
opt-out Opts out of telemetry, and stops telemetry services
journal Prints telemetry journal contents. Use -h argument for more
options
telemctl commands:
******************
start/stop/restart
==================
The commands to start, stop and restart the telemetry services manage all
required services and probes on the system. There is no need to separately
start/stop/restart the two client daemons **telemprobd** and **telempostd**.
The **restart** command option will call **telemctl stop** followed by
**telemctl start** .
is-active
=========
The `is-active` option reports whether the two client daemons are active.
This is useful to verify that the **opt-in** and **opt-out** options have
taken effect, or to ensure that telemetry is functioning on the system.
Note that both daemons are verified.
.. code-block:: bash
sudo telemctl is-active
.. code-block:: console
telemprobd : active
telempostd : active
.. include:: ./telemetry-enable.rst
:start-after: incl-opt-in-out-telemetry:
:end-before: incl-opt-in-out-telemetry-end:
Next steps
==========
Learn to read records:
* :ref:`telemetry-journal`
@@ -0,0 +1,19 @@
.. _telemetrics:
Telemetry
#########
The |CL-ATTR| telemetrics solution collects data from running |CL| systems
and helps to quickly identify and fix bugs in the OS. These guides will walk
you through setup, configuration, and customization of the telemetry client. The data collected from the client system is analyzed and presented by the
telemetry backend solution. For more details, learn how to
:ref:`telemetry-backend`.
.. toctree::
:maxdepth: 1
telemetry-enable
telemetry-config
telemctl
telemetry-journal
telemetry-api
@@ -0,0 +1,170 @@
.. _telemetry-api:
Telemetry API
#############
Installing the ``telemetrics`` bundle includes the libtelemetry C library,
which exposes an API used by the telemprobd and telempostd daemons. You
can use these in your applications as well. The API documentation is found
in the :file:`telemetry.h` file in `Telemetrics client`_ repository.
Creating records with telem-record-gen
**************************************
The telemetrics bundle also provides a record generator tool called
``telem-record-gen``. This tool can be used to create records from shell
scripts, etc., when writing a probe in C is not desirable. Records are sent
to the backend server, and can also be echoed to stdout.
telem-record-gen usage
======================
.. code-block:: bash
telem-record-gen [OPTIONS] - create and send a custom telemetry record
.. code-block:: console
Help Options:
-h, --help Show help options
Application Options:
-V, --version Print the program version
-s, --severity Severity level (1-4) - (default 1)
-c, --class Classification level_1/level_2/level_3 (required)
-p, --payload Record body (max size = 8k) (required)
-P, --payload-file File to read payload from
-R, --record-version Version number for format of payload (default 1)
-e, --event-id Event id to use in the record
-o, --echo Echo record to stdout
-n, --no-post Do not post record just print
The :command:`-c` and :command:`-p` options are required; defaults are
supplied for most other options. The maximum payload size is 8k
(8192 bytes). Excess is ignored, regardless of source (file/commandline/
stdin). An empty payload is allowed, but even an empty payload must be
specified in one of the three ways shown below.
telem-record-gen examples
=========================
There are three ways to supply the payload to the record.
#. On the command line, use the :command:`-p <string>` option:
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -p 'payload goes here'
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023189
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2236710e4fc11e4a646ce956c7802788
payload goes here
#. Specify a file that contains the payload with the option
:command:'-P path/to/file'.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -P ./payload_file.txt
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: d73d6040afd7693cccdfece479df9795
payload read from file
#. If the :command:`-p` or :command:`-P` options are absent, the tool reads
from stdin so you can use it in a :file:`heredoc` in scripts.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o << HEOF
payload read from stdin
HEOF
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2f070e8e71679f2b1f28794e3a6c42ee
payload read from stdin
.. note::
Although only the classification and payload are specified, the tool supplies values for the remaining values.
Telemetry records and the REST API
==================================
If you have not configured the telemetry client to keep records locally, you
can view them using the Web UI of the server, or you can query them from the
server using the REST API provided by |CL| telemetrics. The API is
available at :file:`<server>/api/records`, and when queried, returns a JSON
response that contains a list of records. There are several parameters for
filtering queries, similar to the filters available through the telemetryui Records view.
* classification: The classification of the record
* severity: The severity of the record. Restricted to integer value
* machine_id: The id of the machine where this record was generated on
* build: The build on which the record was generated. Restricted to 256
characters.
* created_in_days: causes the query to return records created after the last
given days
* created_in_sec: returns the records created after the last given seconds
* limit: The maximum number of records to be returned.
Next Steps
==========
* :ref:`telemetry-backend`
* `Telemetrics client`_
Related topics
==============
* :ref:`telemetry-about`
.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/
@@ -0,0 +1,101 @@
.. _telemetry-config:
Telemetry client configuration
##############################
The telemetry client will look for the configuration file located at
:file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the
file does not exist, the client will use the default configuration located
at :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. To modify or
customize the configuration, copy the file from
:file:`/usr/share/defaults/telemetrics` to
:file:`/etc/telemetrics` and edit it.
Configuration Options
*********************
The client uses the following configuration options from the config file:
* **server**: This specifies the web server to which telempostd sends the
telemetry records.
* **socket_path**: This specifies the path of the unix domain socket that
the telemprobd listens on for connections from the probes.
* **spool_dir**: This configuration option is related to spooling. If the
daemon is not able to send the telemetry records to the backend server due
to reasons such as the network availability, then it stores the records in
a spool directory. This option specifies that path of the spool directory.
This directory should be owned by the same user as the daemon.
- mkdir -p /var/spool/telemetry
- chown -R telemetry:telemetry /var/spool/telemetry
- systemctl restart telemprobd.service
* **record_expiry**: This is the time in minutes after which the records in
the spool directory are deleted by the daemon.
* **spool_process_time**: This specifies the time interval in seconds that
the daemon waits for before checking the spool directory for records. The
daemon picks up the records in the order of modification date and tries to
send the record to the server. It sends a maximum of 10 records at a time.
If it was able to send a record successfully, it deletes the record from
the spool. If the daemon finds a record older than the "record_expiry"
time, then it deletes that record. The daemon looks at a maximum of 20
records in a single spool run loop.
* **rate_limit_enabled**: This determines whether rate-limiting is enabled or
disabled. When enabled, there is a threshold on both records sent within a
window of time, and record bytes sent within a window a time.
* **record_burst_limit**: This is the maximum amount of records allowed to be
passed by the daemon within the record_window_length of time. If set to
-1, the rate-limiting for record bursts is disabled.
* **record_window_length**: The time in minutes (0-59) that
establishes the window length for the record_burst_limit. EX: if
record_burst_window=1000 and record_window_length=15, then no more than
1000 records can be passed within any given fifteen minute window.
* **byte_burst_limit**: This is the maximum amount of bytes that can be
passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled.
* **byte_window_length**: This is the time, in minutes (0-59), that
establishes the window length for the byte_burst_limit.
* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting
threshold has been reached. Currently the options are 'drop' or 'spool',
with spool being the default. If spool is chosen, records will be spooled
and sent at a later time.
* **record_retention_enabled**: When this key is enabled (true) the daemon
saves a copy of the payload on disk from all valid records. To avoid the
excessive use of disk space only the latest 100 records are kept. The
default value for this configuration key is false.
* **record_server_delivery_enabled**: This key controls the delivery of
records to server; when enabled (default value), the record will be posted
to the address in the configuration file. If this configuration key is
disabled (false), records will not be spooled or posted to backend. This
configuration key can be used in combination with record_retention_enabled
to keep copies of telemetry records locally only.
.. note::
Configuration options may change as the telemetry client evolves.
Please use the comments in the file itself as the most accurate
reference for configuration.
Setting a static machine id
===========================
The machine id reported by the telemetry client is rotated every 3 days for
privacy reasons. If you wish to have a static machine id for testing
purposes, you can opt in by creating a static machine id file named
"opt-in-static-machine-id" under the directory :file:`/etc/telemetrics/`.
Where "unique machine ID" is your desired static machine ID.
.. code-block:: bash
sudo mkdir -p /etc/telemetrics
.. code-block:: bash
sudo echo "unique machine ID" > /etc/telemetrics/opt-in-static-machine-id
.. note::
The machine id mentioned here is not the same as the system hostname. Learn how to :ref:`hostname`:
Next steps
==========
* :ref:`telemctl`
@@ -1,116 +1,114 @@
.. _telemetry-enable:
Enable and disable telemetry in |CL-ATTR|
#########################################
|CL-ATTR| includes a telemetry solution as part of the OS that records events
of interest and reports them back to the development team via the telemetrics
client daemons, **telempobd** and **telempostmd**. This functionality is
maintained in the **telemetrics** software bundle.
.. note::
The telemetry functionality adheres to `Intel privacy policies`_
regarding the collection and use of :abbr:`PII (Personally Identifiable
Information)` and is open source. Specifically, no intentionally
identifiable information about the user or system owner is collected.
End users may enable or disable the telemetry component of |CL| or even
redirect where the records go if they wish to collect records for themselves.
Install the telemetry software bundle
*************************************
During the initial installation of |CL|, you are requested to join the
stability enhancement program and allow |CL| to collect anonymous reports
to improve system stability. If you choose not to join this program, then the
telemetry software bundle is not added to your system.
To install the telemetry bundle, enter the following command as either the
root user or with :command:`sudo` privileges:
.. code-block:: bash
sudo swupd bundle-add telemetrics
This adds the telemetrics-client to your system and you will automatically
opt-in for the service.
Enable telemetry
*****************
To start telemetry on your system, run the following command:
.. code-block:: bash
sudo telemctl start
This enables and starts the :command:`telemprobd` and :command:`telempostd` daemons and your system will
begin to send telemetry data to the server defined in the file
:file:`/etc/telemetrics/telemetrics.conf`. If this file does not exist, the
:command:`telemprobd` and :command:`telempostd` daemons will use the file
:file:`/usr/share/defaults/telemetrics/telemetrics.conf`.
Disable telemetry
*****************
To disable both of the telemetry daemons, run the following command:
.. code-block:: bash
sudo telemctl stop
Opt-out of telemetry
********************
To stop sending telemetrics data from your system, opt out of the
telemetry service:
.. code-block:: bash
sudo telemctl opt-out
This creates the file :file:`/etc/telemetrics/opt-out` and stops the
telemetry services.
Opt-in to telemetry
*******************
Conversely, to opt-in to the telemetry services, simply enter the opt-in
command and start the service:
.. code-block:: bash
sudo telemctl opt-in
This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists,
and starts the telemetry services.
.. note::
To opt-in but not immediately start telemetry services, you will need to
run the command :command:`sudo telemctl stop` after the :command:`opt-in`
command is entered. Once you are ready to start the service, enter the
command :command:`sudo telemctl start`.
Remove the telemetry software bundle
************************************
To completely remove telemetrics from your system, use the command
:command:`swupd` to remove the telemetry software bundle:
.. code-block:: bash
sudo swupd bundle-remove telemetrics
Additional resources
********************
* `Telemetry feature description`_
* :ref:`Telemetry architecture<telemetry-about>`
* :ref:`telemetry-backend`
* https://github.com/clearlinux/telemetrics-client
.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
.. _`Telemetry feature description`:
https://clearlinux.org/features/telemetry
.. _telemetry-enable:
Enable telemetry
################
Telemetry enables developers to observe and proactively address issues on
|CL-ATTR| before end users are impacted. The telemetry functionality
is maintained in the ``telemetrics`` software bundle.
.. note::
The telemetry functionality adheres to `Intel privacy policies`_
regarding the collection and use of :abbr:`PII (Personally Identifiable
Information)` and is open source. Specifically, no intentionally
identifiable information about the user or system owner is collected.
End users may enable or disable the telemetry component of |CL| or even
redirect where the records go if they wish to collect records for themselves.
Install the telemetry software bundle
*************************************
During the initial installation of |CL|, you are requested to join the
stability enhancement program and allow |CL| to collect anonymous reports
to improve system stability. If you choose not to join this program, then the
telemetry software bundle is not added to your system.
To install the telemetry bundle, enter the following command as either the
root user or with :command:`sudo` privileges:
.. code-block:: bash
sudo swupd bundle-add telemetrics
This adds the telemetrics-client to your system, and you will automatically
opt-in for the service.
Enable telemetry
================
To start telemetry on your system, run the following command:
.. code-block:: bash
sudo telemctl start
This enables and starts the :command:`telemprobd` and :command:`telempostd`
daemons. Your system will begin to send telemetry data to the server defined
in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not
exist, the :command:`telemprobd` and :command:`telempostd` daemons will use
the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`.
Disable telemetry
=================
To disable both of the telemetry daemons, run the following command:
.. code-block:: bash
sudo telemctl stop
.. _incl-opt-in-out-telemetry:
Opt-in to telemetry
===================
To opt-in to the telemetry services, simply enter the opt-in
command and start the service:
.. code-block:: bash
sudo telemctl opt-in
This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists,
and starts the telemetry services.
.. note::
To opt-in but not immediately start telemetry services, you will need to
run the command :command:`sudo telemctl stop` after the :command:`opt-in`
command is entered. Once you are ready to start the service, enter the
command :command:`sudo telemctl start`.
Opt-out of telemetry
====================
To stop sending telemetrics data from your system, opt out of the
telemetry service:
.. code-block:: bash
sudo telemctl opt-out
This creates the file :file:`/etc/telemetrics/opt-out` and stops the
telemetry services.
.. _incl-opt-in-out-telemetry-end:
Remove the telemetry software bundle
====================================
To completely remove telemetrics from your system, use the :command:`swupd`
command to remove the telemetry software bundle:
.. code-block:: bash
sudo swupd bundle-remove telemetrics
Next steps
==========
* :ref:`telemetry-config`
.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
@@ -0,0 +1,108 @@
.. _telemetry-journal:
telemctl journal
################
The telemctl ``journal`` command gives you access to features and options of
the telemetry journal to assist with system analytics and debug. The
:command:`sudo telemctl journal` has a number of options to help filter
records. Use :command:`-h` or :command:`--help` to view usage options.
.. code-block:: bash
sudo telemctl journal -h
::
-r, --record_id Print record with specific record_id
-e, --event_id Print records with specific event_id
-c, --classification Print records with specific classification
-b, --boot_id Print records with specific boot_id
-i, --include_record Include record content
-V, --verbose Verbose output
-h, --help Display this help message
Journal Output
**************
To see the listing of records in the journal, run the command:
.. code-block:: bash
sudo telemctl journal -V
This will produce output like the following:
.. list-table:: **Table 1. Using -V for Verbose output**
:widths: 10 30 20 20 20
:header-rows: 1
* - Classification
- Time stamp
- Record ID
- Event ID
- Boot ID
* - org.clearlinux/heartbeat/ping
- Fri 2018-09-21 00:00:57 UTC
- 269e8e4026e6aa440c4d2ed71e38efcd
- b3a51b5e62a008ed0b56d1740be67d48
- 853a75aa-da3b-4356-a085-079abab3ffe1
* - org.clearlinux/hello/world
- Fri 2018-09-21 17:53:21 UTC
- b06c8d31adf5ccc7d5d3f8959d8d3e72
- 57c64c79a9b911d68f4dab10a00267d7
- 853a75aa-da3b-4356-a085-079abab3ffe1
* - org.clearlinux/crash/clr
- Fri 2018-09-21 17:57:59 UTC
- b62cd4278672ae3331cf121bc7a8e1c6
- b6adb5751382c48eebb7ee007fe1790a
- 853a75aa-da3b-4356-a085-079abab3ffe1
Each line gives information about a distinct record. The :command:`-V` or
:command:`--verbose` option adds the header to identify the Classification,
Time Stamp, Record ID, Event ID and Boot ID for each record. The journal
feature can filter records according to the Classification, Record ID, Event
ID and Boot ID by using the :command:`-c`,:command:`-r`, :command:`-e` and
:command:`-b` options accordingly.
Payload Information
********************
From the previous output, you may want to get more information about the
record with the "org.clearlinux/crash/clr" classification to help debug a
crash. You can use the :command:`-c` and :command:`-i` options to see the payload of the record, like this:
.. code-block:: bash
sudo telemctl journal -c org.clearlinux/crash/clr -i
.. code-block:: console
org.clearlinux/crash/clr Tue 2018-09-25 18:43:50 UTC 07ae583edbd13829965d67e9ba97d70c 69c600470769c841649266178375d67e d32c13d1-fda0-49c6-8431-e6c5b29cbefa
Process: /usr/bin/bash
PID: 685
Signal: 11
Backtrace (TID 685):
#0 kill() - [libc.so.6]
#1 bash_tilde_expand() - [/usr/bin/bash]
#2 maybe_execute_file() - [/usr/bin/bash]
#3 main() - [/usr/bin/bash]
#4 __libc_start_main() - [libc.so.6]
#5 _start() - [/usr/bin/bash]
If you have records of multiple crashes, you can use the :command:'-r'
option to specify the record more precisely, rather than going by
classification. You can also specify a classification of record and use the
:command:'-i' option to see the payload of each record with that
classification.
Next steps
==========
Adding telemetry to your applications:
* :ref:`telemetry-api`
@@ -22,7 +22,7 @@ To search for bundles and their contents, enter:
.. code-block:: bash
sudo swupd search [bundle name]
sudo swupd search [bundle name]
To add a bundle, enter:
@@ -30,14 +30,14 @@ To add a bundle, enter:
sudo swupd bundle-add [bundle name]
Additional information
Additional information
======================
For additional :command:`swupd` commands, enter:
.. code-block:: bash
swupd --help
swupd --help
To reference the :command:`swupd` man page, enter:
@@ -3,8 +3,8 @@
Available bundles
#################
This document provides a current list of `available bundles`_ as
of ``12 September 2018``.
This document provides a current list of `available bundles`_ as
of ``12 September 2018``.
Bundle list
===========
@@ -3,7 +3,7 @@
Collaboration guidelines
########################
Thank you for your interest in collaborating with the |CLOSIA|. This guide
Thank you for your interest in collaborating with the |CL-ATTR|. This guide
details the best ways to collaborate with the |CL| team.
Additionally, this guide provides the guidelines our documentation follows.
Thus, you can help improve our documents with your use case tutorials,
@@ -3,7 +3,7 @@
Code blocks
###########
Collaborating to the |CLOSIA| is all about code. Therefore, your
Contributing to the |CL-ATTR| is all about code. Therefore, your
documentation must include as many code examples as possible. You can write
code examples directly in the documentation or include them from a source
file. Use these guidelines to insert code blocks to your documentation:
@@ -4,7 +4,7 @@
Contents directive
##################
For |CL| documentation that has three or more sections, use the `contents::`
For |CL-ATTR| documentation that has three or more sections, use the `contents::`
directive as shown in the example below. This directive automatically captures the headings (and
subheadings if used) as specified in the value given after `:depth:`. Adding this directive to
longer documents allows users to quickly navigate to the desired section.
@@ -24,8 +24,8 @@ longer documents allows users to quickly navigate to the desired section.
EXAMPLE:
Clear Linux Guide Example
*************************
Guide Example
*************
Introduction
============
@@ -12,7 +12,7 @@ consistency of the documents.
Internal cross-references
*************************
An internal cross-reference is a reference to a location within the |CLOSIA|
An internal cross-reference is a reference to a location within the |CL-ATTR|
documentation. Use explicit markup labels and the ``:ref:`` role to create
cross references to headings, figures, and code examples as needed. Every
file must have a label before the title, which is identical to the file's
@@ -127,7 +127,7 @@ Use this template to add a hyperlink with a separated definition:
The include directive
*********************
Clear Linux documentation also uses the ``.. include::``
|CL| documentation also uses the ``.. include::``
directive to include a portion of another reST file.
Use the ``.. include::`` directive to show a select portion of a file.
@@ -3,7 +3,7 @@
Documentation contribution guidelines
#####################################
The |CLOSIA| documentation contribution guidelines provide detailed information
The |CL-ATTR| documentation contribution guidelines provide detailed information
about the scope and purpose of the documentation, the accepted writing style,
and the markup used.
@@ -4,7 +4,7 @@ Grammar guide
#############
This guide provides valuable insight into the correct grammar for the
|CLOSIA| documentation. It covers subjects such as capitalization, verbs,
|CL-ATTR| documentation. It covers subjects such as capitalization, verbs,
hyphenation, possessives, and contractions.
Capitalization
@@ -6,7 +6,7 @@ Headings
Descriptive and brief headings are crucial to the quality of the
documentation. Sphinx uses the headings within the
:abbr:`ReST (RestructuredText)` files to generate the navigation of the HTML
output and the outlines of the PDF files. The |CLOSIA| publishes the
output and the outlines of the PDF files. The |CL-ATTR| publishes the
documentation as HTML making consistent heading levels extremely important.
In addition to the title of the file, only three levels of headings are
@@ -8,7 +8,7 @@ sometimes is difficult to explain using words alone. Well-planned graphics
reduce the amount of text required to explain information. Non-native English
readers rely heavily on graphics because graphics enhance their understanding of the text.
Follow these guidelines when creating graphics for the |CLOSIA|:
Follow these guidelines when creating graphics for the |CL-ATTR|:
* Save the image files in a :file:`figures` folder. The folder must be found
at the same level as the file containing the text.
@@ -4,7 +4,7 @@ Inline Markup
*************
Sphinx supports a large number of inline markup elements called roles. The
|CLOSIA| documentation encourages the use of as many roles as
|CL-ATTR| documentation encourages the use of as many roles as
possible. Thus, you can use any additional roles supported by Sphinx
not listed here. Please refer to the `Sphinx reStructuredText Markup`_
documentation for the full list of supported roles.
@@ -4,13 +4,13 @@ Language reference guide
########################
This section describes acceptable usage of the English language in the
|CLOSIA| documentation. It includes information about words use,
|CL-ATTR| documentation. It includes information about words use,
punctuation, and grammar. This guide does not replace a professional
writer's review but is intended to help collaborators submit consistent
contributions.
To make translations easier and to make the content accessible to non-native
speakers, |CLOSIA| uses Simple English. However, we have not implemented any
speakers, |CL-ATTR| uses Simple English. However, we have not implemented any
controlled language standards.
.. toctree::
@@ -4,7 +4,7 @@ Punctuation guide
#################
This section contains all the information regarding the correct use of
punctuation for the |CLOSIA| documentation.
punctuation for the |CL-ATTR| documentation.
Commas, Semicolons, and Colons
******************************
@@ -8,7 +8,7 @@ RestructuredText guide
Overview
********
The |CLOSIA| uses Sphinx and RestructuredText as authoring tools for its
The |CL-ATTR| uses Sphinx and RestructuredText as authoring tools for its
documentation. This section contains the preferred methods for using the
:abbr:`ReST (RestructuredText)` markup on your documents. Please refer to the
`Sphinx documentation`_ for the complete list of available markup and use
@@ -12,7 +12,7 @@ Simple English improves the clarity of procedural technical writing,
makes translation easier, and improves comprehension for people whose
first language is not English.
|CLOSIA| does not use controlled language, which restricts the writer's
|CL-ATTR| does not use controlled language, which restricts the writer's
vocabulary to a list of approved words. However, we do strongly recommend
using the language principles described below.
@@ -4,7 +4,7 @@ Consistent content structures guide
###################################
This section guides you through the different content structures used in the
|CLOSIA| documentation. This section serves as an example of the correct use
|CL-ATTR| documentation. This section serves as an example of the correct use
of markup. Refer to our :ref:`rest` to learn more about using
restructuredText to author your content.
@@ -5,7 +5,7 @@ Tables
Tables must only be used for information that is either too numerous or too
related for a list to be appropriate. The smallest acceptable table is 2x2
not counting the table header. The |CLOSIA| uses special ReStructuredText
not counting the table header. The |CL-ATTR| uses special ReStructuredText
markup to make including tables easier. If you plan on adding a table
consider transforming it into a list before you embark on creating a table.
Follow these general guidelines:
@@ -4,10 +4,10 @@ Compatible Hardware
###################
This document describes hardware that has been tested and confirmed as
compatible with |CLOSIA|. This list is not comprehensive and will continue to
compatible with |CL-ATTR|. This list is not comprehensive and will continue to
grow.
.. list-table:: **Table 1. Clear Linux Compatible Hardware**
.. list-table:: **Table 1. Compatible Hardware**
:widths: 20, 20
:header-rows: 1
@@ -1,9 +1,9 @@
.. _compatible-kernels:
Compatible Clear Linux\* kernels
################################
Compatible kernels
##################
The |CLOSIA| provides the following Linux kernels with a respective bundle.
The |CL-ATTR| provides the following Linux kernels with a respective bundle.
This document describes the specific use cases these `bundles`_ serve
and provides links to their source code.
@@ -44,10 +44,10 @@ The *kernel-kvm* bundle focuses on the Linux
:abbr:`KVM (Kernel-based Virtual Machine)`. It is optimized for fast booting
and performs best on Virtual Machines running on the Intel® architectures
described on the :ref:`supported hardware list<system-requirements>`.
Use this kernel when running |CL| as the guest OS on top of *qemu/kvm*. Use
this kernel with **cloud orchestrators** using *qemu/kvm* internally as
their **hypervisor**. This kernel can be used as a standalone |CL| VM, see
our :ref:`instructions on using KVM<kvm>` for more information. The
Use this kernel when running |CL| as the guest OS on top of *qemu/kvm*. Use
this kernel with **cloud orchestrators** using *qemu/kvm* internally as
their **hypervisor**. This kernel can be used as a standalone |CL| VM, see
our :ref:`instructions on using KVM<kvm>` for more information. The
optimization patches are found in our `Linux-KVM`_ GitHub repo.
Kernel Hyper-V\*
@@ -58,7 +58,7 @@ Hyper-V. It is optimized for fast booting and performs best on Virtual
Machines running on the Intel® architectures described on the
:ref:`supported hardware list<system-requirements>`.
Use this kernel when running |CL| as the guest OS of **Cloud Instances** in
projects such as Microsoft `Azure`_\*. This kernel can be used in a
projects such as Microsoft `Azure`_\*. This kernel can be used in a
standalone |CL| VM, see our :ref:`instructions on using Hyper-V<hyper-v>` for
more information. The optimization patches are found in our `Linux-HyperV`_
GitHub repo.
@@ -6,7 +6,7 @@ How to Clear training overview
The existing Linux\* ecosystem can be challenging to users because of long
intervals between :abbr:`OS (operating system)` updates, which leads to large
update downloads. Users can also be frustrated when managing complicated
component dependencies. |CLOSIA| solves these problems by:
component dependencies. |CL-ATTR| solves these problems by:
* Allowing you to update frequently, even multiple times per day
* Preventing you from combining incompatible components
@@ -31,7 +31,7 @@ overview of these |CL| mechanisms:
* Tools: `mixer-tools` software suite generates the update server content using |CL|
official software update content, local bundle definitions, and local RPM files.
The training helps you create a customized OS that is based on |CLOSIA|. The
The training helps you create a customized OS that is based on |CL|. The
training content is self-contained and hosted on GitHub. You need a clean
|CL| installation and a functional network connection to complete the
training. For convenience, the project includes the training files you need
+4 -4
View File
@@ -35,14 +35,14 @@ Table 2 lists the currently available images that are platform specific.
* - Image Type
- Description
* - installer.img
- Preferred image of |CL| with interactive installer.
* - installer.img
- Preferred image of |CL| with interactive installer.
* - installer.iso
- ISO of |CL| with interactive installer. Only for special cases where ISO image format is required (not for use with a USB key)
* - live.img
- Image for live booting into memory, without requiring installaton.
- Image for live booting into memory, without requiring installaton.
.. list-table:: Table 2: Types of platform-specific |CL| images
:widths: 15, 85
@@ -70,7 +70,7 @@ Table 2 lists the currently available images that are platform specific.
- Virtual Hard Disk for use with Microsoft Hyper-V\* hypervisor. Includes `optimized kernel`_ for Hyper-V.
* - kvm.img
- Image for booting in a simple VM with start_qemu.sh. Includes
- Image for booting in a simple VM with start_qemu.sh. Includes
`optimized kernel`_ for KVM.
* - vmware.vmdk
+2 -2
View File
@@ -3,11 +3,11 @@
Reference
#########
This section provides additional information on the |CL| project and |CL|
This section provides additional information on the |CL| project and
features.
.. toctree::
:maxdepth: 2
:maxdepth: 1
compatible-hardware
bundle-commands
@@ -3,7 +3,7 @@
Recommended minimum system requirements
#######################################
|CL-ATTR| can run on very minimal hardware. For example, it can run on a
|CL-ATTR| can run on very minimal hardware. For example, it can run on a
system with a single core CPU, 128MB of memory, and 600MB of disk space.
Different use cases and applications will require different configurations.
@@ -12,7 +12,7 @@ minimum requirements include:
* Processors:
|CL-ATTR| can run on any Intel® 64bit processors which support UEFI\*
|CL-ATTR| can run on any Intel® 64bit processors which support UEFI\*
and SSE\* v4.1 streaming SIMD\* instructions.
The following processor families have been verified to run |CL|:
+121 -123
View File
@@ -3,25 +3,25 @@
Enable AWS Greengrass* and OpenVINO™ on |CL-ATTR|
#################################################
Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers
to deploy inference functionalities [1] on Intel® IoT edge devices with
Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers
to deploy inference functionalities [1] on Intel® IoT edge devices with
accelerators (Integrated GPU, Intel® FPGA, and Intel® Movidius™). These
functions provide a great developer experience and seamless migration of
visual analytics from cloud to edge in a secure manner using a containerized
environment. Hardware-accelerated FaaS provides the best-in-class
performance by accessing optimized deep learning libraries on Intel® IoT
functions provide a great developer experience and seamless migration of
visual analytics from cloud to edge in a secure manner using a containerized
environment. Hardware-accelerated FaaS provides the best-in-class
performance by accessing optimized deep learning libraries on Intel® IoT
edge devices with accelerators.
This tutorial shows how to:
This tutorial shows how to:
* Set up the Intel® edge device with |CL-ATTR|
* Install the OpenVINO™ and AWS Greengrass* software stacks
* Use AWS Greengrass and lambdas to deploy the FaaS samples from the cloud
* Use AWS Greengrass and lambdas to deploy the FaaS samples from the cloud
Supported Platforms
*******************
* Operating System: |CL-ATTR| latest release
* Operating System: |CL-ATTR| latest release
* Hardware: Intel® core platforms (Tutorial supports inference on CPU only)
Description of Samples
@@ -32,7 +32,7 @@ The AWS Greengrass samples are located at the `Edge-Analytics-FaaS`_.
We provide the following AWS Greengrass samples:
* :file:`greengrass_classification_sample.py`
This AWS Greengrass sample classifies a video stream using classification
networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS*
IoT Cloud every second.
@@ -41,7 +41,7 @@ We provide the following AWS Greengrass samples:
This AWS Greengrass sample detects objects in a video stream and
classifies them using single-shot multi-box detection (SSD) networks such
as SSD Squeezenet, SSD Mobilenet, and SSD300. This sample publishes
as SSD Squeezenet, SSD Mobilenet, and SSD300. This sample publishes
detection outputs such as class label, class confidence, and bounding box
coordinates on AWS IoT Cloud every second.
@@ -51,13 +51,13 @@ Converting Deep Learning Models
Sample Models
=============
For classification, `download the BVLC Alexnet model`_ as an example.
Any custom pre-trained classification models can be used with the
For classification, `download the BVLC Alexnet model`_ as an example.
Any custom pre-trained classification models can be used with the
classification sample.
For object detection, the sample models optimized for Intel® edge platforms
are present at :file:`/usr/share/openvino/models`. These models are provided
as an example; however, any custom pre-trained SSD models can be used with
For object detection, the sample models optimized for Intel® edge platforms
are present at :file:`/usr/share/openvino/models`. These models are provided
as an example; however, any custom pre-trained SSD models can be used with
the object detection sample.
Running Model Optimizer
@@ -79,42 +79,42 @@ For object detection using SqueezeNetSSD-5Class model:
.. code-block:: bash
python3 mo.py --framework caffe --input_model
python3 mo.py --framework caffe --input_model
SqueezeNetSSD-5Class.caffemodel --input_proto
SqueezeNetSSD-5Class.prototxt
SqueezeNetSSD-5Class.prototxt
--data_type <data_type> --output_dir <output_dir>
In these examples:
In these examples:
* ``<model_location>`` is :file:`/usr/share/openvino/models`
* ``<model_location>`` is :file:`/usr/share/openvino/models`
* ``<data_type>`` is FP32 or FP16, depending on target device.
* ``<data_type>`` is FP32 or FP16, depending on target device.
* ``<output_dir>`` is the directory where the user wants to store the
Intermediate Representation (IR). IR contains .xml format corresponding
to the network structure and .bin format corresponding to weights. This
.xml file should be passed to <PARAM_MODEL_XML>.
* ``<output_dir>`` is the directory where the user wants to store the
Intermediate Representation (IR). IR contains .xml format corresponding
to the network structure and .bin format corresponding to weights. This
.xml file should be passed to <PARAM_MODEL_XML>.
* In the BVLC Alexnet model, the prototxt defines the input shape with
batch size 10 by default. In order to use any other batch size, the
entire input shape needs to be provided as an argument to the model
optimizer. For example, to use batch size 1, you can provide
batch size 10 by default. In order to use any other batch size, the
entire input shape needs to be provided as an argument to the model
optimizer. For example, to use batch size 1, you can provide
“--input_shape [1,3,227,227]”.
Installing |CL| on the edge device
**********************************
Start with a clean installation of |CL| on a new system, using the
Start with a clean installation of |CL| on a new system, using the
:ref:`bare-metal-install`, found in :ref:`get-started`.
Create user accounts
====================
After |CL| is installed, create two user accounts. Create an administrative
After |CL| is installed, create two user accounts. Create an administrative
user in |CL|. You will also create a user account for the Greengrass
services to use (see Greengrass user below).
services to use (see Greengrass user below).
#. Create a new user and set a password for that user. Enter the following
#. Create a new user and set a password for that user. Enter the following
commands as ``root``:
.. code-block:: bash
@@ -122,7 +122,7 @@ services to use (see Greengrass user below).
useradd <userid>
passwd <userid>
#. Next, enable the :command:`sudo` command for your new ``<userid>``. Add
#. Next, enable the :command:`sudo` command for your new ``<userid>``. Add
``<userid>`` to the ``wheel`` group:
.. code-block:: bash
@@ -136,18 +136,18 @@ services to use (see Greengrass user below).
useradd ggc_user
groupadd ggc_group
#. Create a :file:`/etc/fstab` file.
#. Create a :file:`/etc/fstab` file.
.. code-block:: bash
touch /etc/fstab
.. note::
By default |CL| does not create an :file:`/etc/fstab` file.
The Greengrass service needs to have the file created before
.. note::
By default |CL| does not create an :file:`/etc/fstab` file.
The Greengrass service needs to have the file created before
it will run.
Add required bundles
====================
@@ -156,67 +156,67 @@ enable the OpenVINO software stack:
.. code-block:: bash
swupd bundle-add os-clr-on-clear desktop-autostart computer-vision-basic
swupd bundle-add os-clr-on-clr desktop-autostart computer-vision-basic
.. note::
Learn more about how to :ref:`swupd-guide`.
Learn more about how to :ref:`swupd-guide`.
The ``computer-vision-basic`` bundle will install the OpenVINO software,
The ``computer-vision-basic`` bundle will install the OpenVINO software,
along with the edge device models needed.
Configuring an AWS Greengrass group
===================================
For each Intel® edge platform, we need to create a new AWS Greengrass group
and install AWS Greengrass core software to establish the connection between
For each Intel® edge platform, we need to create a new AWS Greengrass group
and install AWS Greengrass core software to establish the connection between
cloud and edge.
#. To create an AWS Greengrass group, follow the
`AWS Greengrass developer guide`_
#. To install and configure AWS Greengrass core on edge platform, follow
the instructions at `Start AWS Greengrass on the Core Device`_.
the instructions at `Start AWS Greengrass on the Core Device`_.
.. note::
You will not need to run the ``cgroupfs-mount.sh`` script in step #6
of Module 1 of the `AWS Greengrass developer guide`_ because this is
enabled already in |CL|.
of Module 1 of the `AWS Greengrass developer guide`_ because this is
enabled already in |CL|.
Creating and Packaging Lambda Functions
=======================================
#. Complete the tutorial at `Configure AWS Greengrass on AWS IoT`_ .
.. note::
#. Complete the tutorial at `Configure AWS Greengrass on AWS IoT`_ .
This creates the tarball needed to create the AWS Greengrass
environment on the edge device.
.. note::
#. Assure to download both the security resources and the AWS Greengrass
core software.
This creates the tarball needed to create the AWS Greengrass
environment on the edge device.
.. note::
#. Assure to download both the security resources and the AWS Greengrass
core software.
Security certificates are linked to your AWS* account.
.. note::
#. Replace greengrassHelloWorld.py with Greengrass samples:
Security certificates are linked to your AWS* account.
#. Replace greengrassHelloWorld.py with Greengrass samples:
* greengrass_classification_sample.py
* greengrass_object_detection_sample_ssd.py
* greengrass_object_detection_sample_ssd.py
#. Zip these files with extracted Greengrass SDK folders from the previous
step into :file:`greengrass_sample_python_lambda.zip`.
#. Zip these files with extracted Greengrass SDK folders from the previous
step into :file:`greengrass_sample_python_lambda.zip`.
The zip should contain:
* greengrasssdk
* greengrass sample
For the sample, choose one of these:
* greengrass sample
For the sample, choose one of these:
- greengrass_classification_sample.py
@@ -229,15 +229,15 @@ Creating and Packaging Lambda Functions
zip -r greengrass_lambda.zip greengrasssdk
greengrass_object_detection_sample_ssd.py
#. Follow steps 6-11 to `complete creating lambdas`_.
.. note::
#. Follow steps 6-11 to `complete creating lambdas`_.
In the AWS documentation, step 9(a), while uploading the zip file,
make sure to name the handler as below depending on the AWS Greengrass
.. note::
In the AWS documentation, step 9(a), while uploading the zip file,
make sure to name the handler as below depending on the AWS Greengrass
sample you are using:
* greengrass_object_detection_sample_ssd.function_handler (or)
* greengrass_object_detection_sample_ssd.function_handler (or)
* greengrass_classification_sample.function_handler
Deploying Lambdas
@@ -246,18 +246,18 @@ Deploying Lambdas
Configuring the Lambda function
-------------------------------
After creating the Greengrass group and the lambda function, start
configuring the lambda function for AWS Greengrass.
After creating the Greengrass group and the lambda function, start
configuring the lambda function for AWS Greengrass.
#. Follow steps 1-8 in `Configure the Lambda Function`_ of the AWS
documentation.
documentation.
#. In addition to the details mentioned in step 8, change the Memory limit
to 2048MB to accommodate large input video streams.
#. Add the following environment variables as key-value pairs when editing
the lambda configuration and click on update:
.. list-table:: **Table 1. Environment Variables: Lambda Configuration**
:widths: 20 80
:header-rows: 1
@@ -265,7 +265,7 @@ configuring the lambda function for AWS Greengrass.
* - Key
- Value
* - PARAM_MODEL_XML
- <MODEL_DIR>/<IR.xml>, where <MODEL_DIR> is user specified and
- <MODEL_DIR>/<IR.xml>, where <MODEL_DIR> is user specified and
contains IR.xml, the Intermediate Representation file from Intel® Model Optimizer
* - PARAM_INPUT_SOURCE
- <DATA_DIR>/input.webm to be specified by user. Holds both input and
@@ -281,19 +281,19 @@ configuring the lambda function for AWS Greengrass.
- User specified for classification sample.
(e.g. 1 for top-1 result, 5 for top-5 results)
#. Add subscription to subscribe, or publish messages from AWS Greengrass
lambda function by following the steps 10-14 in `Configure the Lambda Function`_
#. Add subscription to subscribe, or publish messages from AWS Greengrass
lambda function by following the steps 10-14 in `Configure the Lambda Function`_
.. note::
The “Optional topic filter” field should be the topic
.. note::
The “Optional topic filter” field should be the topic
mentioned inside the lambda function.
For example, openvino/ssd or openvino/classification
Local Resources
---------------
#. Select `this link to add local resources and access privileges`_.
#. Select `this link to add local resources and access privileges`_.
Following are the local resources needed for the CPU:
@@ -301,71 +301,71 @@ Local Resources
:widths: 20, 20, 20, 20
:header-rows: 1
* - Name
- Resource type
- Local path
* - Name
- Resource type
- Local path
- Access
* - ModelDir
- Volume
- <MODEL_DIR> to be specified by user
* - ModelDir
- Volume
- <MODEL_DIR> to be specified by user
- Read-Only
* - Webcam
- Device
* - Webcam
- Device
- /dev/video0
- Read-Only
* - DataDir
- Volume
- <DATA_DIR> to be specified by user. Holds both input and output
* - DataDir
- Volume
- <DATA_DIR> to be specified by user. Holds both input and output
data.
- Read and Write
Deploy
------
To `deploy the lambda function to AWS Greengrass core device`_, select
“Deployments” on group page and follow the instructions.
To `deploy the lambda function to AWS Greengrass core device`_, select
“Deployments” on group page and follow the instructions.
Output Consumption
------------------
There are four options available for output consumption. These options are
used to report, stream, upload, or store inference output at an interval
There are four options available for output consumption. These options are
used to report, stream, upload, or store inference output at an interval
defined by the variable ``reporting_interval`` in the AWS Greengrass samples.
a. IoT Cloud Output:
This option is enabled by default in the AWS Greengrass samples using a
variable ``enable_iot_cloud_output``. We can use it to verify the lambda
running on the edge device. It enables publishing messages to IoT cloud
using the subscription topic specified in the lambda (For example,
openvino/classification for classification and openvino/ssd for
object detection samples). For classification, top-1 result with class
label are published to IoT cloud. For SSD object detection, detection
results such as bounding box co-ordinates of objects, class label, and
class confidence are published.
This option is enabled by default in the AWS Greengrass samples using a
variable ``enable_iot_cloud_output``. We can use it to verify the lambda
running on the edge device. It enables publishing messages to IoT cloud
using the subscription topic specified in the lambda (For example,
openvino/classification for classification and openvino/ssd for
object detection samples). For classification, top-1 result with class
label are published to IoT cloud. For SSD object detection, detection
results such as bounding box co-ordinates of objects, class label, and
class confidence are published.
Follow the instructions here to `view the output on IoT cloud`_
b. Kinesis Streaming:
This option enables inference output to be streamed from the edge device
to cloud using Kinesis [3] streams when enable_kinesis_output is set
to True. The edge devices act as data producers and continually push
processed data to the cloud. The users need to set up and specify
Kinesis stream name, Kinesis shard, and AWS region in the AWS Greengrass
This option enables inference output to be streamed from the edge device
to cloud using Kinesis [3] streams when enable_kinesis_output is set
to True. The edge devices act as data producers and continually push
processed data to the cloud. The users need to set up and specify
Kinesis stream name, Kinesis shard, and AWS region in the AWS Greengrass
samples.
c. Cloud Storage using AWS S3 Bucket:
When the enable_s3_jpeg_output variable is set to True, it enables uploading and storing processed frames (in JPEG format) in an AWS S3 bucket. The users need to set up and specify the S3 bucket name in the
When the enable_s3_jpeg_output variable is set to True, it enables uploading and storing processed frames (in JPEG format) in an AWS S3 bucket. The users need to set up and specify the S3 bucket name in the
AWS Greengrass samples to store the JPEG images. The images are named using the timestamp and uploaded to S3.
d. Local Storage:
When the enable_s3_jpeg_output variable is set to True, it enables storing processed frames (in JPEG format) on the edge device. The
images are named using the timestamp and stored in a directory specified
When the enable_s3_jpeg_output variable is set to True, it enables storing processed frames (in JPEG format) on the edge device. The
images are named using the timestamp and stored in a directory specified
by PARAM_OUTPUT_DIRECTORY.
References
@@ -391,7 +391,7 @@ References
.. _Configure the Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html
.. _Add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _Add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _deploy the lambda function to AWS Greengrass core device: https://docs.aws.amazon.com/greengrass/latest/developerguide/configs-core.html
@@ -402,5 +402,3 @@ References
.. _this link to add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _Configure AWS Greengrass on AWS IoT: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html