clearlinux/clear-linux-documentation
2
0
Fork 0
mirror of https://github.com/clearlinux/clear-linux-documentation.git synced 2026-10-04 16:08:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

fixed merge of latest

Browse Source
This commit is contained in:
Kevin Putnam
2019-03-26 13:53:18 -07:00
parent 7e86cab634 1e0175b195
commit b2b6e9e527
102 changed files with 8281 additions and 3440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/clear-linux/concepts/swupd-about.rst
+73 -104
View File
@@ -1,132 +1,101 @@
.. _swupd-about:
Software update
###############
swupd: software updater
#######################
|CL-ATTR| does software updates differently than traditional Linux-based
operating systems. Where traditional distributions rely on packages for
software deployment, |CL| uses the concept of a "bundle" for
deployment. Traditional Linux packages provide a particular utility or
library; |CL| bundles provide all necessary packages to enable a
specific function.
With |CL|, updating equates to an entirely new OS version with a
specific set of bundles, as compared to a package-based distribution in
which packages may be updated individually. |CL| updates are
efficient, updating only changed files instead of entire packages.
System administrators can customize or add bundles to the OS, while still
taking advantage of a controlled update stream. This enables system
administrators to focus on the pieces that make their deployment unique.
Bundles
=======
While we use packages to manage compiling source code into installable
binaries, we do not deploy software through packages. Instead, we use bundles
to deploy software, where each bundle encapsulates a particular functionality
-- functionality that is enabled by composing all the required upstream
open-source projects and packages into one logical unit: a bundle. This
simplifies installing features on |CL|.
For additional resources regarding available bundles, useful bundle commands,
and compatible |CL| kernels, visit our :ref:`bundles-about`
page.
:command:`swupd` is an operating system software manager and update program
that operates at a file-level to enable verifiable integrity and update
efficiency.
Visit the `swupd man page`_ for more details.
Versioning
==========
In a traditional distribution, the process of describing current software
versioning usually involves:
Using package managers to keep track of software version compatibility or compare multiple systems on many Linux distributions can be cumbersome.
- Listing and keeping track of the current OS release (generally
uninformative about any singular packages or functionality).
With |CL| :command:`swupd`, versioning happens at the individual
file-level. This means |CL| generates an entirely new OS version with any set
of software changes to the system (including software downgrades or removals). This rolling release versioning model is similar to
:command:`git` internal version tracking, where any of the individual file
commits are tracked and move the pointer forward when changed.
- Keeping track of packages and repositories being used, and updating them
individually.
- Listing and tracking every package available and installed on the
system, none of which are directly tied to the current OS release.
This can be done effectively, but given the nearly endless combinations of
packages and versions of packages a server may have, it quickly becomes
non-trivial to define what "version" the system is and what software it
is running without explicitly going through each system and inspecting
every package.
With |CL|, we need only track:
- One single number
A number representing the **current** release of the OS is sufficient to
describe the versions of all the software on the OS. Each build is
composed of a specific set of bundles made from a particular version of
packages. This matters on a daily basis to system administrators, who
need to determine which of their systems do not have the latest security
fixes, or which combinations of software have been tested. Every release
of the same number is guaranteed to contain the same versions of software,
so there's no ambiguity between two systems running the same version of |CL|.
While administrators can pick and choose which `bundles`_ a system has
installed, a single |CL| version number strictly represents one combination
of all software versions that can be installed onto a system of that |CL|
version. This method of whole OS versioning offers unique advantages.
Namely, system administrators can quickly compare multiple |CL| systems that share the same version for important software and security fixes.
Updating
========
Another notable difference between package-based distributions and |CL|
is how updates are managed. On a package-based OS, system administrators update
each individual package or piece of software to a newer (or older!) version. With
|CL|, an update translates to an entirely new OS version, containing one
or many updates. It is not possible to update a piece of the system while
remaining on the same version of |CL|.
|CL| promotes regular and automated updating of software to ensure
integration of new enhancements and security fixes. Refer to :ref:`security`
documentation for more information.
How is this useful? Although it seems, at first, like a huge restriction
or limitation, this method has many non-obvious benefits. Imagine a
cloud environment composed of numerous machines. Here, a homogeneous set of
software makes sense -- from the system administrator's level down to the
user level. Homogeneous systems allow users to focus on their contributions
and/or code, rather than configuring environments or worrying about
synchronizing versions and updates. At the system admin level, it ensures
security is tighter and makes it far easier to monitor and update patches.
Learn how to update your system using :ref:`swupd <swupd-guide>`.
|CL| promotes regular updating of the OS and will automatically check
for updates and apply them by default.
Update efficiency
-----------------
To learn how to run an update of your system, visit our :ref:`swupd-guide` page.
Because :command:`swupd` operates at the individual file-level instead of a
package-level, |CL| updates are small and fast.
On many Linux\* distributions, updates to a particular software package
require the whole software package to be downloaded and replaced
--even for one line of code.
Update speed
============
In |CL|, updates are generated using the :ref:`mixer <mixer-about>` tool. Mixer calculates the difference between two |CL| versions and makes available
*binary deltas*, which contain only the changed portion of files. This
*binary delta technology* [1]_ means :command:`swupd` on |CL| systems only
needs to download and apply a small fraction of a package in order to
receive an update.
Software updates with |CL| are also efficient. Bundles simply describe
a set of files, and the update technology updates *only* files that actually
changed by using binary-delta technology for efficiency [1]_. Operating systems
that use packages as the unit of deployment require full package updates (thus
hogging resources), even when one small file in that package has changed.
The :ref:`mixer <mixer-about>` tool additionally computes updates files in
multiple compression formats, allowing :command:`swupd` to utilize the most
efficiently compressed format for a |CL| system to minimize the cost
to update.
It is quite common for a full OS update fixing a security hole to be
only 15 kilobytes in total update size. If only several kilobytes need
to be changed, it does not make sense to re-download and reinstall an
entire package or suite of programs just to incorporate a minuscule (yet
important) update. Through binary deltas, the OS is able to update only
those bits that changed, yielding very small update content (deltas)
that can be applied exceedingly fast. As a result, major security patches
and core update take merely seconds.
Update integrity
----------------
:command:`swupd` operates against a published manifest of files for a
particular |CL| version that contains the unique hash of each file. This is
the basis of the :command:`swupd verify` subcommand, which allows a |CL|
system to check for and remediate any discrepancies to system files. As
necessary, :command:`swupd verify` provides a useful way for software
developers to return to a known filesystem state.
Customize the OS
================
Bundles
=======
While we realize our definition of bundles makes sense to us, data center
operators may have special needs and ideas. Therefore, we provide a
:ref:`mixer tool <mixer>`. This tool allows users to customize and add bundles
or even add their own software, while keeping the operating
system and its updates as the basis. Using this tool, system administrators
can focus on the customization their deployments require while staying on
a controlled update stream.
|CL-ATTR| approaches software management differently than many other
Linux-based operating systems.
To learn more about mixing, visit our :ref:`mixer-about` page.
Instead of deploying granular software packages, |CL| uses the concept of
bundles with pre-associated software. Each bundle encapsulates a particular
use-case, which is enabled by composing all the required upstream open-source
projects and packages into one logical unit.
This bundle-based approach offers some unique advantages:
* Bundles provide a particular functionality, or stack, which
include all associated runtime dependencies.
* Software package dependencies are resolved on the server, so file-level
conflicts do not occur on the target system after an update.
* All combinations of bundles are able to co-exist on a |CL| system.
For more information on bundles, visit:
* :ref:`bundles`
* :ref:`bundles-about`
* :ref:`bundle-commands`
* :ref:`compatible-kernels`
.. [1] The software update technology for |CL-ATTR| was first presented at the Linux Plumbers conference in 2012.
.. _swupd man page: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst
.. [1] The software update technology for |CL-ATTR| was first presented at the
Linux Plumbers conference in 2012.
source/clear-linux/concepts/telemetry-about.rst
+3 -4
View File
@@ -38,7 +38,7 @@ Architecture
If you want to capture your own records for analysis, you must set up
your own backend server.
.. figure:: figures/telemetry-about-1.png
.. figure:: ../guides/telemetrics/figures/telemetry-e2e.png
:scale: 75%
:alt: Clear Linux Telemetry Architecture.
@@ -48,7 +48,7 @@ The telemetry client provides the front end of a complete telemetrics solution
and includes the following components:
* **telemprobd**, a daemon that prepares the telemetry records and spools them on disk prior to delivery
* **telempostd**, a daemon that sends the records to the telemetry backend server or leaves them on disk until deleting after the record expires.
* **telempostd**, a daemon that sends the records to the telemetry backend server or leaves them on disk until deleting after the record expires.
* **probes**, that collect specific types of data from the operating system.
* **libtelemetry**, that telemetrics probes use to create telemetrics records and
@@ -76,8 +76,7 @@ Next steps
To put this concept into practice, see the following resources:
* :ref:`telemetry-enable`
* :ref:`telemetry-backend`
* :ref:`telem-guide`
* `Telemetry feature description`_
.. _`Telemetry feature description`:
source/clear-linux/get-started/bare-metal-install-beta/bare-metal-install-beta.rst
+49 -22
View File
@@ -5,8 +5,11 @@ Install |CL-ATTR| from the live desktop beta image
The live desktop beta image allows you to boot |CL-ATTR| into a GNOME
desktop without modifying the host system. Using the live image, you can
explore the possibilities of developing with |CL|. You can also launch the
new installer and install |CL| on your target system.
explore the possibilities of developing with |CL|.
You can also launch the installer to install |CL| on your target system.
If you proceed with installation, this document assumes you follow the
`Recommended options`_.
.. contents:: :local:
:depth: 1
@@ -32,9 +35,7 @@ Preliminary steps
#. Follow your OS instructions to create a bootable USB drive.
* :ref:`bootable-usb-linux-all`
* :ref:`bootable-usb-mac-all`
* :ref:`bootable-usb-windows-all`
* :ref:`bootable-usb`
Install from live image
***********************
@@ -86,18 +87,18 @@ Minimum installation requirements
*********************************
To fulfill minimum installation requirements, complete the
`Required options`_. `Advanced options`_ are optional.
`Required options`_. We also encourage you to install `Recommended options`_ for a full desktop experience. `Advanced options`_ are optional.
.. note::
* The :kbd:`Install` button is only highlighted **after** you complete the
`Required options`_, and after you enter required values in submenus.
* The :kbd:`Install` button is **only highlighted after** you complete
`Required options`_.
* You must choose whether or not to participate in :ref:`telemetrics`
* You must choose whether or not to participate in :ref:`telem-guide`
before you can finish installation.
* You may wish to `Test Network Settings`_ before you
`Configure Network Interfaces`_. Assure that a ``Success`` message is received before installation.
* You may `Test Network Settings`_ before installation
`Configure Network Interfaces`_. Assure a *Success* message appears before installation.
Main Menu
*********
@@ -242,7 +243,7 @@ Configure Media
#. Select :kbd:`Enter` to :kbd:`Confirm`.
#. Choose one partitioning method and continue below:
#. Select one partitioning method and continue:
* `Auto Partition`_
* `Add Partition`_
@@ -438,7 +439,7 @@ Telemetry
=========
To fulfill the :guilabel:`Required options`, choose whether or not to
participate in `telemetry`_. :ref:`telemetrics` is a |CL| feature that
participate in `telemetry`_. :ref:`telem-guide` is a |CL| feature that
reports failures and crashes to the |CL| development team for improvements.
For more detailed information, visit our :ref:`telemetry-about` page.
@@ -455,6 +456,22 @@ For more detailed information, visit our :ref:`telemetry-about` page.
Figure 14: Enable Telemetry
Recommended options
*******************
After you complete the `Required options`_, we highly recommend completing
a few `Advanced options`_ at minimum:
* `Bundle Selection`_ Add basic utlities and tools:
* :file:`desktop-autostart`
* :file:`user-basic`
* `User Manager`_ Assign a new user with administrative rights
* `Assign Hostname`_ Simplify your development environment
This document assumes you follow these additional steps.
Skip to finish installation
===========================
@@ -462,10 +479,9 @@ After selecting values for all :guilabel:`Required options`, you may skip
to `Finish installation`_.
Otherwise, continue below. In the Main Menu, select
:guilabel:`Advanced options` to configure network interfaces or proxy
settings, add bundles, add/manage users, add kernel arguments, and more.
:guilabel:`Advanced options` for additional configuration.
Advanced Options
Advanced options
****************
Configure Network Interfaces
@@ -588,12 +604,20 @@ Bundle Selection
#. Select :kbd:`Spacebar` to select the checkbox for each desired bundle.
#. We recommend adding :file:`desktop-autostart` and :file:`user-basic`.
.. figure:: figures/bare-metal-install-beta-19.png
:scale: 100%
:alt: Bundle Selection
Figure 19: Bundle Selection
.. note::
The default bundle choices and selections differ between the
:file:`clear-<XXXXX>.live-desktop-beta.img` and the
:file:`clear-<XXXXX>.installer.img`.
#. Select :kbd:`Confirm` or :kbd:`Cancel`.
You are returned to the :guilabel:`Advanced options` menu.
@@ -815,23 +839,26 @@ Finish installation
#. When you are satisfied with your installation configuration, navigate to
:guilabel:`Install` and select :kbd:`Enter`.
#. Select :guilabel:`reboot`.
.. note::
When installation is finished, a ``reboot`` button appears.
#. Select ``reboot``.
If you do not perform `Recommended options`_, upon rebooting, a
:file:`login:` prompt will appear. At the prompt, enter `root` and change your password immediately.
#. When the system reboots, remove any installation media present.
**Congratulations!**
.. note::
You have successfully installed |CL| on bare metal using the new installer.
Allow time for the graphical login to appear. This shows the administrative user that you created in `Recommended options`_.
#. Log in as the adminstrative user.
Next steps
**********
:ref:`enable-user-space`
.. _Navigate to the image directory: https://download.clearlinux.org/image/
.. _Navigate to the image directory: https://cdn.download.clearlinux.org/image/
.. _Autoproxy: https://clearlinux.org/features/autoproxy
.. _telemetry: https://clearlinux.org/features/telemetry
source/clear-linux/get-started/bare-metal-install-beta/figures/bare-metal-install-beta-19.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 170 KiB

After

Width:  |  Height:  |  Size: 147 KiB

source/clear-linux/get-started/bare-metal-install/bare-metal-install.rst
+2 -2
View File
@@ -34,11 +34,11 @@ Optionally, you can use this command:
.. code-block:: bash
curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep "installer")
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep "installer")
#. Follow your OS instructions to create a bootable USB drive.
* :ref:`bootable-usb-beta-all`
* :ref:`bootable-usb`
#. After downloading the image, verify and decompress the file per your OS.
source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-19.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 170 KiB

After

Width:  |  Height:  |  Size: 147 KiB

source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-28.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 96 KiB

After

Width:  |  Height:  |  Size: 96 KiB

source/clear-linux/get-started/bootable-usb/bootable-usb-beta-all.rst
-203
View File
@@ -1,203 +0,0 @@
.. _bootable-usb-beta-all:
Create a bootable USB on your OS
################################
Follow these instructions to create a bootable |CL-ATTR| USB drive based on
your OS.
* :ref:`bootable-usb-linux-all`
* :ref:`bootable-usb-mac-all`
* :ref:`bootable-usb-windows-all`
Return to :ref:`get-started`
Requirements:
*************
* Use a **16GB** or larger USB drive.
.. _bootable-usb-linux-all:
Create a bootable USB drive on Linux
************************************
.. include:: ../../guides/maintenance/download-verify-decompress-linux.rst
:Start-after: verify-linux:
Burn the |CL| image onto a USB drive
====================================
.. caution::
|CAUTION-BACKUP-USB|
#. Open a terminal emulator and get root privilege.
.. code-block:: bash
sudo -s
#. Go to the directory with the decompressed image.
#. Plug in the USB drive.
#. Identify the USB drive using the :command:`lsblk` command. This shows all
drives attached to the system, including the primary hard disk. In the
example output below, there are 4 drives
(`/dev/sda`, `/dev/sdb`, `/dev/sdc`, and `/dev/sdd`) attached, where
`/dev/sda` is primary drive in this case. The remaining are 3 USB drives.
The output also shows the mounted partitions (under the `MOUNTPOINT`
column) for each drive.
.. code-block:: bash
lsblk
Example output:
.. code-block:: console
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdd 8:48 1 15G 0 disk
├─sdd2 8:50 1 5G 0 part /run/media/user1/960c184f-3bb7-42b7-bcaf-0c1282
├─sdd3 8:51 1 8G 0 part /run/media/user1/704f3382-b26d-4f34-af1b-cb9aab
└─sdd1 8:49 1 2G 0 part
sdb 8:16 1 14.8G 0 disk
└─sdb1 8:17 1 14.8G 0 part /run/media/user1/PATRIOT_USB
sdc 8:32 1 7.3G 0 disk
└─sdc1 8:33 1 7.3G 0 part /run/media/user1/LINUX MINT
sda 8:0 0 335.4G 0 disk
├─sda4 8:4 0 28G 0 part
├─sda2 8:2 0 3.7G 0 part [SWAP]
├─sda7 8:7 0 6G 0 part /home
├─sda5 8:5 0 1G 0 part /boot
├─sda3 8:3 0 954M 0 part /boot/efi
├─sda1 8:1 0 28G 0 part
├─sda8 8:8 0 30G 0 part /
└─sda6 8:6 0 7.9G 0 part [SWAP]
#. Before an image can be burned onto a USB drive, it should be un-mounted.
Some Linux* distros may automatically mount a USB drive when it is plugged
in. To unmount, use the :command:`umount` command followed by the device
identifier/partition. For example: From the above :command:`lsblk` output,
`/dev/sdd` has 2 mounted partitions. To unmount them, enter:
.. code-block:: bash
umount /dev/sdd2
umount /dev/sdd3
#. Burn the image onto the USB drive. The command-line example below burns an
uncompressed image onto `/dev/sdd`:
.. code-block:: bash
dd if=./clear-[version number]-[image type] of=/dev/sdd bs=4M status=progress
.. _bootable-usb-mac-all:
Create a bootable USB drive on macOS*
*************************************
.. include:: ../../guides/maintenance/download-verify-decompress-mac.rst
:start-after: verify-mac:
Burn the |CL| image onto a USB drive
====================================
.. caution::
|CAUTION-BACKUP-USB|
#. Launch the Terminal app.
#. Go to the directory with the decompressed image.
#. Plug in a USB drive and get its identifier by entering the command
:command:`diskutil list`. See Figure 1.
.. code-block:: console
diskutil list
.. figure:: figures/bootable-usb-mac-1.png
:scale: 100 %
:alt: Get USB drive identifier
Figure 1: macOS* - Get USB drive identifier
#. Unmount the USB drive identified in the previous step. The command-line
example below umounts `/dev/disk2`:
.. code-block:: console
diskutil umountDisk /dev/disk2
#. Burn the image onto the drive using the :command:`dd` command. The
command-line example below burns an uncompressed image onto `/dev/disk2`:
.. code-block:: console
sudo dd if=./clear-[version number]-[image type] of=/dev/rdisk2 bs=4m
Adding an ‘r’ in front of the disk identifier should help speed up the
imaging process.
You can press :kbd:`<CTL>-T` to check imaging progress.
#. Eject the USB drive.
.. code-block:: console
diskutil eject /dev/disk2
.. _bootable-usb-windows-all:
Create a bootable USB drive on Windows\*
****************************************
.. include:: ../../guides/maintenance/download-verify-decompress-windows.rst
:Start-after: verify-windows:
Burn the |CL| image onto a USB drive
====================================
.. caution::
|CAUTION-BACKUP-USB|
#. Download the `Rufus`_ utility to burn the image onto a USB drive.
#. Plug in the USB drive and open Rufus.
#. Click the :guilabel:`SELECT` button. See Figure 1.
.. figure:: figures/bootable-usb-windows-1.png
:scale: 80 %
:alt: Rufus utility - Click the SELECT button
Figure 1: Rufus utility - Click the SELECT button
#. Find and select the previously extracted |CL| image file.
Then, click the :guilabel:`Open` button. See Figure 2.
.. figure:: figures/bootable-usb-windows-2.png
:scale: 80 %
:alt: Rufus utility - Show and select |CL| image file
Figure 2: Rufus utility - Show and select |CL| image file
#. Click the :guilabel:`START` button. See Figure 3.
.. figure:: figures/bootable-usb-windows-3.png
:scale: 80 %
:alt: Rufus utility - Click the START button
Figure 3: Rufus utility - Click START button
Return to install from live image
*********************************
Return to :ref:`get-started`
.. _Rufus: https://rufus.ie/
source/clear-linux/get-started/bootable-usb/bootable-usb-linux.rst
-105
View File
@@ -1,105 +0,0 @@
.. _bootable-usb-linux:
Create a bootable USB drive on Linux\*
######################################
Follow these instructions to create a bootable |CL-ATTR| USB drive.
Use an **8GB** or larger USB drive. Download either a live image,
``clear-<version>-live.img.xz`` or an installer image,
``clear-<version>-installer.img.xz``, from our `image`_ download page.
Instructions are also available for other operating systems:
* :ref:`bootable-usb-mac`
* :ref:`bootable-usb-windows`
.. include:: ../../reference/image-types.rst
:start-after: incl-image-filename:
:end-before: incl-image-filename-end:
.. include:: ../../guides/maintenance/download-verify-decompress-linux.rst
:Start-after: verify-linux:
.. _copy-usb-linux:
Burn the |CL| image onto a USB drive
************************************
.. caution::
|CAUTION-BACKUP-USB|
#. Open a terminal emulator and get root privilege.
.. code-block:: bash
sudo -s
#. Go to the directory with the decompressed image.
#. Plug in the USB drive.
#. Identify the USB drive using the :command:`lsblk` command. This shows all
drives attached to the system, including the primary hard disk. In the
example output below, there are 4 drives
(`/dev/sda`, `/dev/sdb`, `/dev/sdc`, and `/dev/sdd`) attached, where
`/dev/sda` is primary drive in this case. The remaining are 3 USB drives.
The output also shows the mounted partitions (under the `MOUNTPOINT`
column) for each drive.
.. code-block:: bash
lsblk
Example output:
.. code-block:: console
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdd 8:48 1 15G 0 disk
├─sdd2 8:50 1 5G 0 part /run/media/user1/960c184f-3bb7-42b7-bcaf-0c1282
├─sdd3 8:51 1 8G 0 part /run/media/user1/704f3382-b26d-4f34-af1b-cb9aab
└─sdd1 8:49 1 2G 0 part
sdb 8:16 1 14.8G 0 disk
└─sdb1 8:17 1 14.8G 0 part /run/media/user1/PATRIOT_USB
sdc 8:32 1 7.3G 0 disk
└─sdc1 8:33 1 7.3G 0 part /run/media/user1/LINUX MINT
sda 8:0 0 335.4G 0 disk
├─sda4 8:4 0 28G 0 part
├─sda2 8:2 0 3.7G 0 part [SWAP]
├─sda7 8:7 0 6G 0 part /home
├─sda5 8:5 0 1G 0 part /boot
├─sda3 8:3 0 954M 0 part /boot/efi
├─sda1 8:1 0 28G 0 part
├─sda8 8:8 0 30G 0 part /
└─sda6 8:6 0 7.9G 0 part [SWAP]
#. Before an image can be burned onto a USB drive, it should be un-mounted.
Some Linux distros may automatically mount a USB drive when it is plugged
in. To unmount, use the :command:`umount` command followed by the device
identifier/partition. For example: From the above :command:`lsblk` output,
`/dev/sdd` has 2 mounted partitions. To unmount them, enter:
.. code-block:: bash
umount /dev/sdd2
umount /dev/sdd3
#. Burn the image onto the USB drive. The command-line example below burns an
uncompressed image onto `/dev/sdd`:
.. code-block:: bash
dd if=./clear-[version number]-[image type] of=/dev/sdd bs=4M status=progress
.. _usb-next:
Next steps
**********
With a bootable |CL| USB drive, you can:
* :ref:`bare-metal-install`
* :ref:`boot-live-image`
* :ref:`multi-boot`
.. _image: https://download.clearlinux.org/image
.. _releases: https://download.clearlinux.org/releases
source/clear-linux/get-started/bootable-usb/bootable-usb-mac.rst
-81
View File
@@ -1,81 +0,0 @@
.. _bootable-usb-mac:
Create a bootable USB drive on macOS
####################################
Follow these instructions to create a bootable |CL-ATTR| USB drive.
Use an **8GB** or larger USB drive. Download either a live image,
``clear-<version>-live.img.xz`` or an installer image,
``clear-<version>-installer.img.xz``, from our `image`_ download page.
Instructions are also available for other operating systems:
* :ref:`bootable-usb-linux`
* :ref:`bootable-usb-windows`
.. include:: ../../reference/image-types.rst
:start-after: incl-image-filename:
:end-before: incl-image-filename-end:
.. include:: ../../guides/maintenance/download-verify-decompress-mac.rst
:start-after: verify-mac:
Burn the |CL| image onto a USB drive
************************************
.. caution::
|CAUTION-BACKUP-USB|
#. Launch the Terminal app.
#. Go to the directory with the decompressed image.
#. Plug in a USB drive and get its identifier by entering the command
:command:`diskutil list`. See Figure 1.
.. code-block:: console
diskutil list
.. figure:: figures/bootable-usb-mac-1.png
:scale: 100 %
:alt: Get USB drive identifier
Figure 1: macOS - Get USB drive identifier
#. Unmount the USB drive identified in the previous step. The command-line
example below umounts `/dev/disk2`:
.. code-block:: console
diskutil umountDisk /dev/disk2
#. Burn the image onto the drive using the :command:`dd` command. The
command-line example below burns an uncompressed image onto `/dev/disk2`:
.. code-block:: console
sudo dd if=./clear-[version number]-[image type] of=/dev/rdisk2 bs=4m
Adding an ‘r’ in front of the disk identifier should help speed up the
imaging process.
You can press :kbd:`<CTL>-T` to check imaging progress.
#. Eject the USB drive.
.. code-block:: console
diskutil eject /dev/disk2
Next steps
**********
With a bootable |CL| USB drive, you can:
* :ref:`bare-metal-install`
* :ref:`boot-live-image`
* :ref:`multi-boot`
.. _image: https://download.clearlinux.org/image
source/clear-linux/get-started/bootable-usb/bootable-usb-windows.rst
-69
View File
@@ -1,69 +0,0 @@
.. _bootable-usb-windows:
Create a bootable USB drive on Windows\*
########################################
Follow these instructions to create a bootable |CL-ATTR| USB drive.
Use an **8GB** or larger USB drive. Download either a live image,
``clear-<version>-live.img.xz`` or an installer image,
``clear-<version>-installer.img.xz``, from our `image`_ download page.
Instructions are also available for other operating systems:
* :ref:`bootable-usb-mac`
* :ref:`bootable-usb-linux`
.. include:: ../../reference/image-types.rst
:start-after: incl-image-filename:
:end-before: incl-image-filename-end:
.. include:: ../../guides/maintenance/download-verify-decompress-windows.rst
:Start-after: verify-windows:
Burn the |CL| image onto a USB drive
************************************
.. caution::
|CAUTION-BACKUP-USB|
#. Download the `Rufus`_ utility to burn the image onto a USB drive.
#. Plug in the USB drive and open Rufus.
#. Click the :guilabel:`SELECT` button. See Figure 1.
.. figure:: figures/bootable-usb-windows-1.png
:scale: 80 %
:alt: Rufus utility - Click the SELECT button
Figure 1: Rufus utility - Click the SELECT button
#. Find and select the previously extracted |CL| image file.
Then, click the :guilabel:`Open` button. See Figure 2.
.. figure:: figures/bootable-usb-windows-2.png
:scale: 80 %
:alt: Rufus utility - Show and select |CL| image file
Figure 2: Rufus utility - Show and select |CL| image file
#. Click the :guilabel:`START` button. See Figure 3.
.. figure:: figures/bootable-usb-windows-3.png
:scale: 80 %
:alt: Rufus utility - Click the START button
Figure 3: Rufus utility - Click START button
Next steps
**********
With a bootable |CL| USB drive, you can:
* :ref:`bare-metal-install`
* :ref:`boot-live-image`
* :ref:`multi-boot`
.. _Rufus: http://rufus.akeo.ie/
.. _image: https://download.clearlinux.org/image
source/clear-linux/get-started/bootable-usb/bootable-usb.rst
+179 -19
View File
@@ -1,32 +1,192 @@
.. _bootable-usb:
Create a bootable |CL-ATTR| USB drive
#####################################
Create a bootable USB drive
###########################
Instructions to create a |CL-ATTR| USB drive vary depending on your operating
system.
system. Follow the instructions applicable to your system:
.. _download-usb-image:
* :ref:`bootable-usb-linux`
* :ref:`bootable-usb-mac`
* :ref:`bootable-usb-windows`
Download the latest |CL| image
******************************
Prerequisites
*************
There are 2 types of |CL| images suitable for burning onto and running
off a USB drive:
* Use an **8GB** or larger USB drive.
* `Download`_ the |CL| live boot image or interactive installer image.
* Live image: :file:`clear-[version number]-live.img.xz`
* Installer image: :file:`clear-[version number]-installer.img.xz`
.. _bootable-usb-linux:
Go to the |CL| `image`_ repository and download the desired type.
Create a bootable USB drive on Linux\*
**************************************
With the appropriate image downloaded, choose the step-by-step instructions
applicable to your system:
Make sure you have have completed all `Prerequisites`_.
.. toctree::
:maxdepth: 1
Before burning the image onto your USB drive,
:ref:`verify and decompress your image <verify-linux>`.
bootable-usb-linux
bootable-usb-windows
bootable-usb-mac
Burn the |CL| image onto a USB drive
====================================
.. _image: https://download.clearlinux.org/image
.. caution::
|CAUTION-BACKUP-USB|
#. Open a terminal emulator and get root privilege.
.. code-block:: bash
sudo -s
#. Go to the directory with the decompressed image.
#. Plug in the USB drive.
#. Identify the USB drive using the :command:`lsblk` command. This shows all
drives attached to the system, including the primary hard disk. In the
example output below, there are 4 drives
(`/dev/sda`, `/dev/sdb`, `/dev/sdc`, and `/dev/sdd`) attached, where
`/dev/sda` is primary drive. The remaining are three USB drives. The output
also shows the mounted partitions (under the `MOUNTPOINT` column) for each
drive.
.. code-block:: bash
lsblk
Example output:
.. code-block:: console
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdd 8:48 1 15G 0 disk
├─sdd2 8:50 1 5G 0 part /run/media/user1/960c184f-3bb7-42b7-bcaf-0c1282
├─sdd3 8:51 1 8G 0 part /run/media/user1/704f3382-b26d-4f34-af1b-cb9aab
└─sdd1 8:49 1 2G 0 part
sdb 8:16 1 14.8G 0 disk
└─sdb1 8:17 1 14.8G 0 part /run/media/user1/PATRIOT_USB
sdc 8:32 1 7.3G 0 disk
└─sdc1 8:33 1 7.3G 0 part /run/media/user1/LINUX MINT
sda 8:0 0 335.4G 0 disk
├─sda4 8:4 0 28G 0 part
├─sda2 8:2 0 3.7G 0 part [SWAP]
├─sda7 8:7 0 6G 0 part /home
├─sda5 8:5 0 1G 0 part /boot
├─sda3 8:3 0 954M 0 part /boot/efi
├─sda1 8:1 0 28G 0 part
├─sda8 8:8 0 30G 0 part /
└─sda6 8:6 0 7.9G 0 part [SWAP]
#. You must unmount a USB drive before you can burn an image onto it. Note that
some Linux distros automatically mount a USB drive when it is plugged in.
Unmount a USB drive with the :command:`umount` command followed by the device
identifier/partition. For example:
.. code-block:: bash
umount /dev/sdd2
umount /dev/sdd3
#. Burn the image onto the USB drive. The example below burns an uncompressed
image onto `<your USB device>`:
.. code-block:: bash
dd if=./clear-[version number]-[image type] of=<your USB device> bs=4M status=progress && sync
.. _bootable-usb-mac:
Create a bootable USB drive on macOS\*
**************************************
Make sure you have have completed all `Prerequisites`_.
Before burning the image onto your USB drive,
:ref:`verify and decompress your image <verify-mac>`.
Burn the |CL| image onto a USB drive
====================================
.. caution::
|CAUTION-BACKUP-USB|
#. Launch the Terminal app.
#. Go to the directory with the decompressed image.
#. Plug in a USB drive and get its identifier:
.. code-block:: bash
diskutil list
This will list available disks and their partitions, as shown in Figure 1.
.. figure:: figures/bootable-usb-mac-1.png
:scale: 100 %
:alt: Get USB drive identifier
Figure 1: macOS - Get USB drive identifier
#. Unmount the USB drive identified in the previous step. For example:
.. code-block:: bash
diskutil umountDisk /dev/disk2
#. Burn the image onto the drive using the :command:`dd` command. The example
below burns an uncompressed image onto `<your USB device>`:
.. code-block:: bash
sudo dd if=./clear-[version number]-[image type] of=<your USB device> bs=4m
To speed up the imaging process, add an ‘r’ in front of the disk identifier.
For example `/dev/rdisk2`.
Press :kbd:`<CTL>-T` to check imaging progress.
#. Eject the USB drive.
.. code-block:: bash
diskutil eject /dev/disk2
.. _bootable-usb-windows:
Create a bootable USB drive on Windows\*
****************************************
Make sure you have have completed all `Prerequisites`_.
Before burning the image onto your USB drive,
:ref:`verify and decompress your image <verify-windows>`.
Burn the |CL| image onto a USB drive
====================================
.. caution::
|CAUTION-BACKUP-USB|
#. Download the `Rufus`_ utility to burn the image onto a USB drive.
#. Plug in the USB drive and open Rufus.
#. Under `Boot selection`, click the :guilabel:`SELECT` button.
#. Find and select the previously extracted |CL| image file.
#. Click the :guilabel:`START` button. See Figure 2.
.. figure:: figures/bootable-usb-windows-3.png
:scale: 80 %
:alt: Rufus utility
Figure 2: Rufus utility
.. _Rufus: https://rufus.ie/
.. _Download: https://clearlinux.org/downloads
source/clear-linux/get-started/compatibility-check.rst
+2 -2
View File
@@ -19,7 +19,7 @@ below.
.. code-block:: console
curl -O https://download.clearlinux.org/current/clear-linux-check-config.sh
curl -O https://cdn.download.clearlinux.org/current/clear-linux-check-config.sh
#. Make the script executable.
@@ -58,4 +58,4 @@ below.
SUCCESS: Carry-less Multiplication extensions (pclmulqdq)
SUCCESS: EFI Firmware
.. _clear-linux-check-config.sh: https://download.clearlinux.org/current/clear-linux-check-config.sh
.. _clear-linux-check-config.sh: https://cdn.download.clearlinux.org/current/clear-linux-check-config.sh
source/clear-linux/get-started/get-started.rst
-1
View File
@@ -17,7 +17,6 @@ Pre-install
compatibility-check
bootable-usb/bootable-usb
bootable-usb/bootable-usb-beta-all
Install |CL|
************
source/clear-linux/get-started/live-image.rst
+1 -1
View File
@@ -29,7 +29,7 @@ Boot the |CL| live image
.. _create and enable user space: https://clearlinux.org/documentation/clear-linux/guides/maintenance/enable-user-space
.. _`image`: https://download.clearlinux.org/image
.. _`image`: https://cdn.download.clearlinux.org/image
.. _`Intel® Virtualization Technology (Intel® VT)`: http://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html
source/clear-linux/get-started/virtual-machine-install/figures/7zipwin.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 28 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-cd.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 20 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-convert-image.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-create-vm.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 14 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-efi.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 43 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-hdisk.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 11 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-memory-size.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.6 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-x.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 304 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-first-login.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.7 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-remove-non-lts-kernels.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-convert-raw-to-VDI.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-disk.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-choose-disk.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-existing-disk.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-new-disk.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-no-disk.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-IMG.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-ISO.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-new-vm.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox-no-vtx.png → source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-no-vtx.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 13 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-shutdown-vm.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.8 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-start-vm.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-created.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-insert-ga-cd.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-EFI.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-browse-ISO.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-mount-ISO.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-unmount-ISO.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB

source/clear-linux/get-started/virtual-machine-install/hyper-v.rst
+1 -1
View File
@@ -57,4 +57,4 @@ Your virtual machine running |CL| is ready!
.. _Windows Server Virtualization: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/
.. _Microsoft documentation: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v
.. _Create A Virtual Network documentation: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/connect-to-network
.. _downloads: https://download.clearlinux.org/image/
.. _downloads: https://cdn.download.clearlinux.org/image/
source/clear-linux/get-started/virtual-machine-install/kvm.rst
+19 -8
View File
@@ -50,12 +50,12 @@ Download and launch the virtual machine
***************************************
#. Download the latest pre-built |CL| KVM image file from
the `image <https://download.clearlinux.org/image/>`_ directory. Look for
the `image <https://cdn.download.clearlinux.org/image/>`_ directory. Look for
``clear-<version>-kvm.img.xz``. You can also use this command:
.. code-block:: bash
curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm)
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm)
#. Uncompress the downloaded image:
@@ -64,7 +64,7 @@ Download and launch the virtual machine
unxz clear-<version>-kvm.img.xz
#. Download the `OVMF file`_ file that provides UEFI support for
virtual machines from the `image <https://download.clearlinux.org/image/>`_ directory.
virtual machines from the `image <https://cdn.download.clearlinux.org/image/>`_ directory.
#. Copy :file:`OVMF.fd` to the working directory, as shown below.
@@ -79,9 +79,14 @@ Download and launch the virtual machine
For non-Clear Linux hosts, the preferred approach is to download it from https://cdn.download.clearlinux.org/image/OVMF.fd
#. Download the sample `QEMU-KVM launcher`_ script from the
`image <https://download.clearlinux.org/image/>`_ directory. This script
`image <https://cdn.download.clearlinux.org/image/>`_ directory. This script
will launch the |CL| VM and provide console interaction within the same
terminal emulator window.
.. code-block:: bash
curl -O https://cdn.download.clearlinux.org/image/start_qemu.sh
#. Make the script executable:
@@ -103,7 +108,7 @@ SSH access into the virtual machine
To interact with the |CL| VM through SSH instead of the console it was
launched from, follow these steps.
#. Enable SSH in the |CL| VM:
#. Configure SSH in the |CL| VM to allow root login:
.. code-block:: bash
@@ -111,6 +116,12 @@ launched from, follow these steps.
PermitRootLogin yes
EOF
#. Start SSH server in the |CL| VM:
.. code-block:: bash
systemctl start sshd
#. From the host, SSH into the |CL| VM. The port number ``10022`` is defined
in the ``start_qemu.sh`` script.
@@ -135,7 +146,7 @@ To add :abbr:`GDM (GNOME Display Manager)` to the |CL| VM, follow these steps:
.. code-block:: bash
swupd bundle-add desktop-apps-extras
swupd bundle-add tigervnc
* On Ubuntu\* 16.04 LTS Desktop:
@@ -232,5 +243,5 @@ To add :abbr:`GDM (GNOME Display Manager)` to the |CL| VM, follow these steps:
.. _Intel® Virtualization Technology: https://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html
.. _Intel®Virtualization Technology for Directed I/O: https://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices
.. _QEMU-KVM launcher: https://download.clearlinux.org/image/start_qemu.sh
.. _OVMF file: https://download.clearlinux.org/image/OVMF.fd
.. _QEMU-KVM launcher: https://cdn.download.clearlinux.org/image/start_qemu.sh
.. _OVMF file: https://cdn.download.clearlinux.org/image/OVMF.fd
source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst
+1
View File
@@ -35,5 +35,6 @@ appropriate set of step-by-step instructions to proceed.
vmw-player
vmw-player-preconf
hyper-v
virtualbox-cl-installer
../../guides/maintenance/increase-virtual-disk-size.rst
source/clear-linux/get-started/virtual-machine-install/virtualbox-cl-installer.rst
+277
View File
@@ -0,0 +1,277 @@
.. _virtualbox-cl-installer:
Install |CL-ATTR| as a VirtualBox\* Virtual Machine
###################################################
VirtualBox\* is a type 2 hypervisor from Oracle. This document explains how
to create a virtual machine on the `VirtualBox hypervisor`_ with |CL-ATTR|
as the guest operating system.
These instructions make use of the |CL| installer to create a brand new |CL|
installation. A preinstalled disk image is also available to get started
with |CL| faster. See: :ref:`virtualbox`
.. contents:: :local:
:depth: 2
.. include:: virtualbox.rst
:start-after: vbox-prereqs-begin:
:end-before: vbox-prereqs-end:
Download and extract the |CL| installer ISO image
*************************************************
The appropriate |CL| installer image needs to be downloaded and extracted.
.. note::
The :file:`installer.iso` is for limited use on special cases where ISO
image format is required, such as |VB|. The preferred installer for |CL|
for UEFI systems is the :file:`-installer.img`.
#. Download the **installer ISO** image (:file:`clear-<VERSION>-installer.iso.xz`) of
|CL|. On the `downloads page`_, this is listed as
**Clear Linux OS for Virtual Provisioning**.
You can also use this command to download from a terminal:
.. code-block:: bash
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep installer.iso)
#. Validate the integrity of the downloaded image by checking the file hash
and signatures. Refer to the document on :ref:`validate-signatures` for
detailed steps.
#. Decompress the downloaded image. Uncompressed image size is ~ **5GB**.
- On Windows you can use `7zip`_ to extract the file by right-clicking the
file to *Extract Here* (in the same directory)
.. image:: ./figures/vbox/vbox-extract-cl-ISO.png
:alt: 7zip extract here command
- On Linux :
.. code-block:: bash
xz -d clear-<VERSION>-installer.iso.xz
#. The originally downloaded compressed archive file
(:file:`clear-<VERSION>-installer.iso.xz`) can now be deleted.
Create a new |VB| virtual machine
*********************************
A new :abbr:`VM (Virtual Machine)` needs to be created in |VBM| for |CL| to be installed onto.
General instructions for creating a virtual machine and details about using
different settings are available on the
`VirtualBox manual section on Creating a VM`_.
#. Launch the |VBM| from your host system.
#. Click the *New* button to create a new VM.
.. image:: ./figures/vbox/vbox-new-vm.png
:alt: Create a new VM in VirtualBox
#. A *Create Virtual Machine* window will appear.
Select the following settings:
- Type: **Linux**
- Version: **Linux 2.6 / 3.x / 4.x (64-bit)**
- Memory size: **1024 MB** (this can be adjusted appropriately)
- Hard disk: **Create a virtual hard disk now**
.. image:: ./figures/vbox/vbox-create-vm-new-disk.png
:alt: Create a new VM in VirtualBox with a new disk
#. Click the *Create* button.
#. A *Create Virtual Hard Disk* window will appear.
Select the following settings:
- File size: **8.00 GB** (this can be adjusted appropriately)
- Hard disk file type: **VDI (Virtual Box Disk Image)**
- Storage on physical hard disk: **Dynamically allocated**
.. image:: ./figures/vbox/vbox-create-disk.png
:alt: Create a new virtual hard disk in VirtualBox
#. Click the *Create* button.
#. A new virtual machine will be created and appear in the |VBM|. Click
*Settings* to configure the |CL| VM.
.. image:: ./figures/vbox/vbox-vm-created.png
:alt: A VM selected in VirtualBox Manager
#. A *VM - Settings* window will appear. Navigate to the *System* pane from
the left-hand and select the following setting:
- **Enable I/O APIC**
- **Enable EFI (special OSes only)**
.. image:: ./figures/vbox/vbox-vm-settings-EFI.png
:alt: Enable EFI on a VirtualBox VM settings
.. note::
By default, only 1 virtual CPU is allocated to the new VM. Consider
increasing the number of virtual processors allocated to the virtual
machine under Settings --> System --> Processor for increased
performance.
Install |CL| on the |VB| VM
***************************
|CL| is ready to be installed.
Mount the installation ISO
==========================
At this point, the newly created VM has a blank virtual hard disk with no
operating system.The |CL| installer ISO needs to be mounted as a virtual
CD-ROM on the VM before powering the VM on.
#. From the *VM - Settings* window, navigate to the *Storage* pane from the
left-hand side.
#. From the middle *Storage Devices* column, click the blue CD disk labeled
*Empty* under the *Controller: IDE* from.
#. From the right-hand *Attributes* column, click the blue CD disk next to
the *Optical Drive* drop down menu and click *Choose Virtual Optical
Disk File...*
.. image:: ./figures/vbox/vbox-vm-settings-mount-ISO.png
:alt: Mounting an ISO in VirtualBox VM Settings
#. A *choose a virtual optical disk file* browser window will appear.
Navigate to the extracted ISO file, select it, and click *Open*.
.. image:: ./figures/vbox/vbox-vm-settings-browse-ISO.png
:alt: Mounting an ISO in VirtualBox VM Settings
#. Click *OK* to exit the *VM Settings* menu and return to the main
|VBM|.
Install |CL| using the installer
================================
#. Start the VM from the |VBM| by selecting the |CL| VM and
clicking *Start*
.. image:: ./figures/vbox/vbox-start-vm.png
:alt: Starting a VirtualBox VM
#. A new window of the VM console will appear and boot into the |CL|
installer. Follow the steps in the `Install Clear Linux OS`_ to
install |CL| onto the VM virtual disk.
.. note::
Do not choose a different kernel from the installer. **kernel-lts**, the
Long Term Support (LTS) kernel is required for |VB| driver compatibility.
#. After |CL| installation is complete, the VM will reboot and return to the
|CL| installer.
.. note::
To release the mouse cursor from the VM console window, press the right Ctrl key on the keyboard.
Unmount the installation ISO
============================
The |CL| installer ISO needs to be unmounted to allow the VM to boot from the
virtual hard disk, which |CL| has been installed to.
#. Power off the |CL| VM.
.. image:: ./figures/vbox/vbox-shutdown-vm.png
:alt: Powering off a VirtualBox VM
#. Click *Settings* to configure the |CL| VM.
.. image:: ./figures/vbox/vbox-vm-created.png
:alt: A VM selected in VirtualBox Manager
#. From the *VM - Settings* window, navigate to the *Storage* pane from the
left-hand side.
#. From the middle *Storage Devices* column, click the blue CD disk labeled
*clear-<VERSION>-installer.iso* under the *Controller: IDE* from.
#. From the right-hand *Attributes* column, click the blue CD disk next to
the *Optical Drive* drop down menu and click *Remove Disk from Virtual
Drive*
.. image:: ./figures/vbox/vbox-vm-settings-unmount-ISO.png
:alt: Unmounting an ISO in VirtualBox VM Settings
#. Click *OK* to exit the *VM Settings* menu and return to the main
|VBM|.
.. include:: virtualbox.rst
:start-after: vbox-start-vm-and-lga-begin:
:end-before: vbox-start-vm-and-lga-end:
.. include:: virtualbox.rst
:start-after: vbox-troubleshooting-begin:
:end-before: vbox-troubleshooting-end
.. |VB| replace:: VirtualBox
.. |VBM| replace:: VirtualBox Manager
.. _appropriate instructions: https://www.virtualbox.org/manual/ch02.html
.. _official VirtualBox website: https://www.virtualbox.org/wiki/Downloads
.. _VirtualBox hypervisor: https://www.virtualbox.org/
.. _downloads page: https://clearlinux.org/downloads
.. _`VirtualBox manual section on Creating a VM`: https://www.virtualbox.org/manual/UserManual.html#gui-createvm
.. _`VirtualBox manual section on Running a VM`: https://www.virtualbox.org/manual/ch01.html#intro-starting-vm-first-time
.. _`Virtual Media Manager`: https://www.virtualbox.org/manual/ch05.html#vdis
.. _`Install Clear Linux OS`: https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install#install-cl-on-your-target-system\
.. _7zip: http://www.7-zip.org/
.. _Virtualization Technology: https://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html
.. _`VirtualBox manual section on VBoxManage`: https://www.virtualbox.org/manual/ch08.html#vboxmanage-convertfromraw
source/clear-linux/get-started/virtual-machine-install/virtualbox.rst
+308 -134
View File
@@ -1,196 +1,370 @@
.. _virtualbox:
Run pre-configured |CL-ATTR| as a VirtualBox\* guest OS
###########################################################
Run pre-configured |CL-ATTR| as a VirtualBox\* Virtual Machine
##############################################################
This instruction explains how to deploy a pre-configured |CL-ATTR| image as a guest on the `VirtualBox hypervisor`_ .
VirtualBox\* is a type 2 hypervisor from Oracle. This document explains how
to create a virtual machine on the `VirtualBox hypervisor`_ with |CL-ATTR|
as the guest operating system.
Download VirtualBox
*******************
These instructions make use of a preinstalled |CL| disk image to setup a |CL|
virtual machine without manual installation. |CL| can also be installed from
scratch on a |VB| using the |CL| installed.
See: :ref:`virtualbox-cl-installer`
VirtualBox\* is a type 2 hypervisor from Oracle. Download and use **version 5.0 or greater** from the `official VirtualBox website`_.
.. contents:: :local:
:depth: 2
.. _create_vm_vbox:
.. _vbox-prereqs-begin:
Prerequisites
*************
The instruction assumes that you have:
Before continuing make sure that you have:
#. Enabled virtualization technology in the host machine's BIOS.
#. Enabled virtualization, such as Intel®
`Virtualization Technology`_ (Intel® VT), on the host system from
EFI/BIOS.
#. Downloaded and installed |VB| **version 6.0 or greater** from
the `official VirtualBox website`_ per the `appropriate instructions`_
for your platform.
.. _vbox-prereqs-end:
Download and extract |CL|
*************************
The |CL| live image needs to be downloaded and extracted. The live image will
be used to created a |VB| virtual disk image that can be used with a
:abbr:`VM (Virtual Machine)`.
#. Download the **live image** (:file:`clear-<VERSION-live.img.xz`) of
|CL|. On the `downloads page`_, this is listed as
**Clear Linux OS live boot image**.
.. note::
For help, see: Intel® `Virtualization Technology`_ (Intel® VT).
#. Installed VirtualBox on your host machine per the
`appropriate instructions`_ for your platform.
If you have not completed the above steps, do so before continuing.
Create a virtual machine in VirtualBox
**************************************
#. Log in to your host and open a terminal emulator.
#. Download the `latest`_ **live** version (clear-XXXX-live.img.xz) of
|CL|. You can also use this command:
You can also use this command to download from a terminal:
.. code-block:: bash
curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep live)
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep live.img)
#. Validate the integrity of the downloaded image by checking the file hash
and signatures. Refer to the document on :ref:`validate-signatures` for
detailed steps.
#. Decompress the downloaded image. Uncompressed image size is ~ **5GB**.
+ On Linux ::
- On Windows you can use `7zip`_ to extract the file by right-clicking the
file and selecting *Extract Here* (in the same directory)
xz -d clear-XXXX-live.img.xz
+ On Windows you can use `7zip`_.
- Right-click the file to *extract in the same directory*.
.. image:: ./figures/7zipwin.png
.. image:: ./figures/vbox/vbox-extract-cl-IMG.png
:alt: 7zip extract here command
#. To convert a raw image to :abbr:`VDI (VirtualBox Disk Image)`
format, you can use one of the following commands::
- On Linux :
VBoxManage convertfromraw clear-XXXX-live.img clear-XXXX-live.vdi --format VDI
.. code-block:: bash
or::
xz -d clear-<VERSION>-live.img.xz
vbox-img convert --srcfilename clear-XXXX-live.img --dstfilename clear-XXXX-live.vdi --srcformat raw --dstformat vdi
#. There originally downloaded compressed archive file
(:file:`clear-<VERSION>-live.img.xz`) can now be deleted.
.. note:: Be sure you have VirtualBox directory in your PATH (i.e., on
Windows :file:`C:\\Program Files\\Oracle\\VirtualBox`).
+ On windows: launch a **Command Prompt** program and type
Convert |CL| live image to a |VB| Disk Image
********************************************
.. code-block:: console
The |CL| live image is in a RAW disk image. The live image needs to be
converted to a :abbr:`VDI (VirtualBox Disk Image)` format which |VB|
can utilize.
set PATH=%PATH%;"C:\Program Files\Oracle\VirtualBox"
#. Launch a terminal and navigate to the directory containing the
extracted live image.
.. image:: ./figures/vbox-convert-image.png
:alt: Convert image in Windows command propt
#. Create a virtual machine using the VirtualBox assistant:
#. Convert RAW live image to a :abbr:`VDI (VirtualBox Disk Image)`
format using the command-line VirtualBox Disk Utility.
a. Type: **Linux**
.. code-block:: bash
VBoxManage convertfromraw clear-<VERSION>-live.img clear-VM.vdi --format VDI
.. note::
The :command:`PATH` environment variable may need to be updated to make the
:command:`VBoxManage` command easily accessible from the terminal.
For example, using Windows PowerShell:
.. code-block:: bash
$env:PATH += ";C:\Program Files\Oracle\VirtualBox"
.. image:: ./figures/vbox/vbox-convert-raw-to-VDI.png
:alt: Convert image in Windows command prompt
For more information on the :command:`VBoxManage` command,
see the `VirtualBox manual section on VBoxManage`_.
#. The originally extracted live image file
(:file:`clear-<VERSION>-live.img`) can now be deleted.
#. Move the converted :file:`clear-VM.vdi` disk image file to a permanent
location. The VDI will be attached to the |VB| VM and should not be
deleted.
Create a new |VB| virtual machine
*********************************
A new VM needs to be created in |VBM| to attach the VDI with |CL| installed.
General instructions for creating a virtual machine and details about using
different settings are available on the
`VirtualBox manual section on Creating a VM`_.
#. Launch the |VBM| from your host system.
#. Click the *New* button to create a new VM.
.. image:: ./figures/vbox/vbox-new-vm.png
:alt: Create a new VM in VirtualBox
#. A *Create Virtual Machine* window will appear.
Select the following settings:
b. Version: **Linux 2.6 / 3.x / 4.x (64-bit)**
- Type: **Linux**
- Version: **Linux 2.6 / 3.x / 4.x (64-bit)**
- Memory size: **1024 MB** (this can be adjusted appropriately)
- Hard disk: **Use an existing virtual hard disk file**
.. image:: ./figures/vbox-create-vm.png
:alt: Create a new image in VirtualBox
Click the folder icon next to the drop down menu:
c. Select default memory size.
.. image:: ./figures/vbox-memory-size.png
d. Attach the virtual disk created in step number 3 as a virtual hard
disk file. Click the folder icon (lower right) to browse to find the
VDI file.
.. image:: ./figures/vbox-hdisk.png
#. After it is created, go to settings to enable **EFI support**
* System -> Enable EFI (special OSes only)
.. image:: ./figures/vbox-efi.png
:alt: Enable EFI on VirtualBox
.. image:: ./figures/vbox/vbox-create-vm-existing-disk.png
:alt: Create a new VM in VirtualBox with an existing disk
Run your new VM
***************
#. A new window will appear for choosing an existing disk. Click the *Add*
button, browse to the saved VDI file, and click *Choose*.
|CL| supports VirtualBox kernel modules used
by the Linux kernel 4.14 :abbr:`LTS (Long Term Support)`
(*kernel-lts bundle*).This kernel was selected because |CL| OS's main kernel
(``kernel-native``) bundle keeps up-to-date with the upstream Linux kernel,
and sometimes VirtualBox kernel modules aren't compatible with pre-kernel
releases.
.. image:: ./figures/vbox/vbox-create-vm-choose-disk.png
:alt: Create a new VM in VirtualBox with an existing disk
On the first boot, |CL| requests a user login.
#. Type **root**.
#. Enter a new password when prompted.
To install the VirtualBox kernel modules, here are the steps:
#. Install the bundle that supports VirtualBox modules::
swupd bundle-add kernel-lts
#. Set a timeout in the bootmanager to shows a menu at boot time::
clr-boot-manager set-timeout 10
#. Update the bootloader entries with::
clr-boot-manager update
#. Reboot your system with::
reboot
and choose **clear-linux-lts-4.14.XX-YYY** kernel version.
#. (*Optional*) Unset timeout to boot directly to LTS version::
clr-boot-manager set-timeout 0
#. (*Mandatory*) Update bootmanger to always use LTS version::
clr-boot-manager update
Install Guest Additions
=======================
The kernel modules are shipped with the ``kernel-lts`` bundle. Insert Guest
Additions CD image using *Devices* menu you'll need to install the *user*
Linux Guest Additions. To install the VirtualBox Guest Additions,
follow these steps:
#. Click the *Create* button.
#. Insert Guest Additions CD image using *Devices* menu
#. A new virtual machine will be created and appear in the |VBM|. Click
*Settings* to configure the |CL| VM.
.. image:: ./figures/vbox-cd.png
.. image:: ./figures/vbox/vbox-vm-created.png
:alt: A VM selected in VirtualBox Manager
#. A *VM - Settings* window will appear. Navigate to the *System* pane from
the left-hand and select the following setting:
- **Enable I/O APIC**
- **Enable EFI (special OSes only)**
.. image:: ./figures/vbox/vbox-vm-settings-EFI.png
:alt: Enable EFI on a VirtualBox VM settings
.. note::
By default, only 1 virtual CPU is allocated to the new VM. Consider
increasing the number of virtual processors allocated to the virtual
machine under Settings --> System --> Processor for increased
performance.
.. _vbox-start-vm-and-lga-begin:
Start the |CL| VM
*****************
The |CL| VM can now be powered on and setup.
General instructions for using a |VB| virtual machine are available on the
`VirtualBox manual section on Running a VM`_.
#. Start the VM from the |VBM| by selecting the |CL| VM and clicking *Start*
.. image:: ./figures/vbox/vbox-start-vm.png
:alt: Starting a VirtualBox VM
#. |CL| will boot and prompt for login.
- Enter **root** for the username.
#. You will be immediately prompted to set a new password for the **root**
user. Reference :ref:`security` for more information about |CL| security
concepts.
.. image:: ./figures/vbox/vbox-cl-first-login.png
:alt: Initial login to Clear Linux OS on a VirtualBox VM
Install |VB| Linux Guest Additions
==================================
The |VB| Linux Guest Additions provide drivers for full compatibility and
functionality.
|CL| provides |VB| guest drivers and an install script in the **kernel-lts**
(Long Term Support) bundle by |CL|.
#. Validate the installed kernel is **kernel-lts** by checking the output
of the :command:`uname -r` command. It should end in **.lts**.
.. code-block:: bash
uname -r
4.<VERSION>.lts
If the running kernel is not **lts**: install the LTS kernel manually,
update the bootloader, and check again:
.. code-block:: bash
swupd bundle-add kernel-lts
clr-boot-manager set-kernel $(basename $(realpath /usr/lib/kernel/default-lts))
clr-boot-manager update
reboot
#. Remove any kernel bundles that are not *kernel-lts* or *kernel-install*
to simplify and avoid conflicts:
.. code-block:: bash
swupd bundle-list | grep kernel
swupd bundle-remove <NON-LTS-KERNEL>
.. image:: ./figures/vbox/vbox-cl-remove-non-lts-kernels.png
:alt: Initial login to Clear Linux OS on a VirtualBox VM
#. From the VM Console window, click *Devices* on the top menu bar, and
select *Insert Guest Additions CD image...* to mount the |VB| driver
installation to the |CL| VM.
.. image:: ./figures/vbox/vbox-vm-insert-ga-cd.png
:alt: VirtualBox CD
#. Install Linux users Guest Additions::
.. note::
To release the mouse cursor from the VM console window, press the right Ctrl key on the keyboard.
install-vbox-lga
#. Reboot your system::
reboot
#. |CL| provides a script called :command:`install-vbox-lga` to help patch
and install |VB| drivers for |CL|. Inside |CL| VM run this command:
.. code-block:: bash
install-vbox-lga
#. After the script completes successfully, reboot the |CL| VM.
.. code-block:: bash
reboot
#. After the VM reboot, login and verify the |VB| drivers are loaded:
.. code-block:: bash
lsmod | grep ^vbox
You should see drivers loaded with names beginning with **vbox**: (vboxguest, vboxsf, vboxvideo).
The |CL| VM running on |VB| is ready to be used.
.. _vbox-start-vm-and-lga-end:
.. _vbox-troubleshooting-begin:
Troubleshooting
===============
***************
On Windows OS, *VirtualBox* cannot do a **Hardware Virtualization** when
*Hyper-V* is enabled.
#. **Problem:** Out of disk space inside of |CL| and not be able to install
additional bundles.
.. image:: ./figures/vbox-no-vtx.png
:alt: VirtualBox hardware acceleration error
**Solution:** The |CL| images are small to minimize download time and
initial disk space .
To disable *Hyper-V* you should execute::
Power off the VM and resize the virtual disk for the |CL| VM using the |VB|
`Virtual Media Manager`_ found under the File menu. Afterwards, power the
|CL| VM on and follow the instructions here to have |CL| detect the resized
disk. :ref:`increase-virtual-disk-size`
bcdedit /set {current} hypervisorlaunchtype off
#. **Problem:** On a Microsoft Windows OS, |VB| encounters an error when
trying to start a VM indicating *VT-X/AMD-v hardware acceleration is not
available on your system.*
in an **Administrator: Command Prompt**, then reboot your system.
To enable Hyper-V again, you should execute::
.. image:: ./figures/vbox/vbox-no-vtx.png
:alt: VirtualBox hardware acceleration error
bcdedit /set {current} hypervisorlaunchtype Auto
**Solution:** First, double check the `Prerequisites`_ section to make sure
*Hardware accelerated virtualization* extensions have been enabled in the
host system's EFI/BIOS.
*Hardware accelerated virtualization*, may get disabled for |VB| when another
hypervisor, such as *Hyper-V* is enabled.
To disable *Hyper-V* execute this command in an
**Administrator: Command Prompt or Powershell**, and reboot the system:
.. code-block:: bash
bcdedit /set {current} hypervisorlaunchtype off
To enable Hyper-V again, execute this command in an
**Administrator: Command Prompt or Powershell**, and reboot the system:
.. code-block:: bash
bcdedit /set {current} hypervisorlaunchtype Auto
.. _vbox-troubleshooting-end:
.. |VB| replace:: VirtualBox
.. |VBM| replace:: VirtualBox Manager
.. _appropriate instructions: https://www.virtualbox.org/manual/ch02.html
.. _official VirtualBox website: https://www.virtualbox.org/wiki/Downloads
.. _VirtualBox hypervisor: https://www.virtualbox.org/
.. _latest: https://download.clearlinux.org/image/
.. _downloads page: https://clearlinux.org/downloads
.. _`VirtualBox manual section on Creating a VM`: https://www.virtualbox.org/manual/UserManual.html#gui-createvm
.. _`VirtualBox manual section on Running a VM`: https://www.virtualbox.org/manual/ch01.html#intro-starting-vm-first-time
.. _`Virtual Media Manager`: https://www.virtualbox.org/manual/ch05.html#vdis
.. _`Install Clear Linux OS`: https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install#install-cl-on-your-target-system\
.. _7zip: http://www.7-zip.org/
.. _Virtualization Technology: https://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html
.. _`VirtualBox manual section on VBoxManage`: https://www.virtualbox.org/manual/ch08.html#vboxmanage-convertfromraw
source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst
+3 -3
View File
@@ -64,7 +64,7 @@ this command:
.. code-block:: bash
curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep vmware)
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep vmware)
Visit :ref:`image-types` for additional information about all available |CL| images.
@@ -306,8 +306,8 @@ For other guides on using the VMWare Player and ESXi, see:
.. _VMware ESXi: https://www.vmware.com/products/esxi-and-esx.html
.. _VMware Workstation 14 Player: https://www.vmware.com/products/workstation-player.html
.. _VMware Workstation Player guide: https://docs.vmware.com/en/VMware-Workstation-Player/index.html
.. _latest: https://download.clearlinux.org/image/
.. _image: https://download.clearlinux.org/image
.. _latest: https://cdn.download.clearlinux.org/image/
.. _image: https://cdn.download.clearlinux.org/image
source/clear-linux/get-started/virtual-machine-install/vmw-player.rst
+2 -2
View File
@@ -297,6 +297,6 @@ For other guides on using the VMWare Player and ESXi, see:
.. _VMware Workstation Player guide:
https://docs.vmware.com/en/VMware-Workstation-Player/index.html
.. _latest: https://download.clearlinux.org/image/
.. _latest: https://cdn.download.clearlinux.org/image/
.. _image: https://download.clearlinux.org/image
.. _image: https://cdn.download.clearlinux.org/image
source/clear-linux/get-started/virtual-machine-install/vmware-esxi-install-cl.rst
+1 -1
View File
@@ -291,4 +291,4 @@ Related topics
.. _VMware ESXi: https://www.vmware.com/products/esxi-and-esx.html
.. _VMware Workstation Player: https://www.vmware.com/products/workstation-player.html
.. _image: https://download.clearlinux.org/image
.. _image: https://cdn.download.clearlinux.org/image
source/clear-linux/get-started/virtual-machine-install/vmware-esxi-preconfigured-cl-image.rst
+2 -2
View File
@@ -36,7 +36,7 @@ this command:
.. code-block:: bash
curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep vmware)
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep vmware)
Visit :ref:`image-types` for additional information about all available |CL| images.
@@ -290,4 +290,4 @@ Related topics
.. _VMware documentation on Enable the Secure Shell (SSH): https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.html.hostclient.doc/GUID-B649CB74-832F-467B-B6A4-8BA67AD5C1F0.html
.. _VMware documentation on General ESXi Security Recommendations: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-B39474AF-6778-499A-B8AB-E973BE6D4899.html
.. _VMware Workstation Player: https://www.vmware.com/products/workstation-player.html
.. _image: https://download.clearlinux.org/image
.. _image: https://cdn.download.clearlinux.org/image
source/clear-linux/guides/clearlinux/autoproxy.rst
+145 -6
View File
@@ -3,12 +3,151 @@
Autoproxy
#########
About
=====
Autoproxy is provided to enable |CL-ATTR| to work smoothly behind a
corporate proxy.
.. include:: autoproxy_about.rst.txt
.. contents::
:local:
:depth: 1
Guide
=====
Description
***********
.. include:: autoproxy_guide.rst.txt
Autoproxy tries to detect a Proxy Auto-Config (PAC) script and use it to
automatically resolve the proxy needed for a given connection. With
Autoproxy, you can use |CL| inside any proxy environment without having to
manually configure the proxies.
Corporate and private networks can be very complex, needing to restrict and
control network connections for security reasons. The typical side effects
are limited or blocked connectivity and requiring manual configuration of
proxies to perform the most mundane tasks such as cloning a repo or checking
for updates. With |CL|, all of the work is done behind the scenes to
effortlessly use your network and have connections “just work”.
This feature removes massive complications in network connectivity due to
proxy issues. You can automate tasks like unit testing without worrying
about the proxy not being set and you can remove unset proxies from the
equation when dealing with network unavailability across systems.
How it works
************
We designed Autoproxy around tools provided by most Linux
distributions with a few minor additions and modifications. We leveraged the
DHCP and network information provided from systemd and created a
PAC-discovery daemon. The daemon uses the information to resolve a URL for a
PAC file. The daemon then passes the URL into PACrunner\*. PACrunner
downloads the PAC file and uses the newly implemented Duktape\* engine to
parse it.
.. figure:: figures/autoproxy_0.png
:width: 400px
Figure 1: Autoproxy Flow
From that point on, any cURL\* or network requests query PACrunner for the
correct proxy to use. We modified the cURL library to communicate with
PACrunner over DBus. However, cURL will ignore PACrunner and run normally if
no PAC file is loaded or if you set any proxies manually. Thus, your
environment settings are respected and no time is wasted trying to resolve a
proxy. All these steps happen in the background with no user interaction.
Troubleshooting
===============
Autoproxy allows |CL| to operate seamlessly behind a proxy
because :ref:`swupd <swupd-guide>` and other |CL| tools are implemented on
top of libcurl. Tools that do not use libcurl, like git, must
be configured independently.
If you are familiar with PAC files and WPAD, you can use
:command:`pacdiscovery` and :command:`FindProxyForURL` to
troubleshoot problems with autproxy.
.. note::
Learn more about WPAD, PAC files, and PAC functions at `findproxyforurl`_.
.. _findproxyforurl: http://findproxyforurl.com/
Run :command:`pacdiscovery` with no arguments to indicate
1. if there is a problem resolving the :command:`WPAD` host name resolution:
.. code-block:: bash
pacdiscovery
.. code-block:: console
failed getaddrinfo: No address associated with hostname
Unable to find wpad host
2. or if the :command:`pacrunner` service is disabled (masked).
.. code-block:: bash
pacdiscovery
.. code-block:: console
PAC url: http://autoproxy.your.domain.com/wpad.dat
Failed to create proxy config: Unit pacrunner.service is masked.
Unmask the :command:`pacrunner` service by running:
.. code-block:: bash
systemctl unmask pacrunner.service
:command:`FindProxyForURL` with :command:`busctl` can also indicate if the
:command:`pacrunner.service` is masked.
.. code-block:: bash
busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client
.. code-block:: console
FindProxyForURL ss "http://www.google.com" "google.com"
Unit pacrunner.service is masked.
dig wpad, dig wpad.<domain>
:command:`FindProxyForURL` returns the URL and port of the proxy server when
an external URL and host are provided as arguments.
.. code-block:: bash
busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client
.. code-block:: console
FindProxyForURL ss "http://www.google.com" "google.com"
s "PROXY proxy.your.domain.com:<port>"
If a proxy server is not avialable, or if :command:`pacrunner` is running
without a PAC file, :command:`FindProxyForURL` will return "DIRECT".
.. code-block:: bash
busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client
.. code-block:: console
FindProxyForURL ss "http://www.google.com" "google.com"
s "DIRECT"
Once :command:`pacdiscovery` is able to look up :command:`WPAD`, restart the
:command:`pacrunner` service:
.. code-block:: bash
systemctl stop pacrunner
systemctl restart pacdiscovery
.. note::
A "domain" or "search" entry in :file:`/etc/resolv.conf` is required
for short name lookups to resolve. The :file:`resolv.conf` man page has
additional details.
source/clear-linux/guides/clearlinux/autoproxy_about.rst.txt
-46
View File
@@ -1,46 +0,0 @@
The |CL-ATTR| is the first Linux distribution to support autoproxy. The OS
can discover a Proxy Auto-Config (PAC) script and use it to automatically
resolve the proxy needed for a given connection. With Autoproxy, you can use
|CL| inside any proxy environment without having to manually
configure the proxies:
* Automate unit testing without worrying about the proxy not being set
* Remove unset proxies from the equation when dealing with network
unavailability across systems.
* Automate unit testing without worrying about the proxy not being set
* Remove unset proxies from the equation when dealing with network
unavailability across systems.
Corporate and private networks can be very complex, needing to restrict and
control network connections for security reasons. The typical side effects
are limited or blocked connectivity and requiring manual configuration of
proxies to perform the most mundane tasks such as cloning a repo or checking
for updates. With |CL|, all of the work is done behind the scenes to
effortlessly use your network and have connections “just work”.
How Autoproxy works
-------------------
We designed autoproxy around general tools provided by nearly any Linux
distribution with a few minor additions and modifications. We leveraged the
DHCP and network information provided from systemd and created a
PAC-discovery daemon. The daemon uses the information to resolve a URL for a
PAC file. The daemon then passes the URL into PACrunner*. PACrunner
downloads the PAC file and uses the newly implemented Duktape* engine to
parse it.
.. figure:: figures/autoproxy_0.png
:width: 400px
Figure 1: Autoproxy Flow
From that point on, any cURL* or network requests query PACrunner for the
correct proxy to use. We modified the cURL library to communicate with
PACrunner over DBus. However, cURL will ignore PACrunner and run normally if
no PAC file is loaded or if you set any proxies manually. Thus, your
environment settings are respected and no time is wasted trying to resolve a
proxy.
More importantly: all these steps happen in the background, very quickly, and
with no user interaction.
source/clear-linux/guides/clearlinux/autoproxy_guide.rst.txt
-79
View File
@@ -1,79 +0,0 @@
Autoproxy allows |CL| to operate seamlessly behind a proxy
because :ref:`software update <swupd-guide>` and other |CL| tools are
implemented on top of libcurl. Tools that do not use libcurl, like git, must
be configured independently.
If you encounter problems with autoproxy functioning, use
:command:`sudo pacdiscovery` and :command:`FindProxyForURL` to
help troubleshoot assuming a familiarity with PAC files and WPAD.
.. note::
Learn more about WPAD, PAC files, and PAC functions at `findproxyforurl`_.
.. _findproxyforurl: http://findproxyforurl.com/
Running :command:`pacdiscovery` with no arguments will immediately indicate
one of the following:
1. If there is a problem resolving the :command:`WPAD` host name resolution:
.. code:: console
$ pacdiscovery
failed getaddrinfo: No address associated with hostname
Unable to find wpad host
2. Or if the :command:`pacrunner` service is disabled (masked):
.. code:: console
$ pacdiscovery
PAC url: http://autoproxy.your.domain.com/wpad.dat
Failed to create proxy config: Unit pacrunner.service is masked.
Unmask the :command:`pacrunner` service by running:
.. code:: console
$ systemctl unmask pacrunner.service
:command:`FindProxyForURL` with :command:`busctl` can also indicate if the
:command:`pacrunner.service` is masked.
.. code:: console
$ busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client FindProxyForURL ss "http://www.google.com" "google.com"
Unit pacrunner.service is masked.
dig wpad, dig wpad.<domain>
:command:`FindProxyForURL` returns the URL and port of the proxy server when
an external URL and host are provided as arguments.
.. code:: console
$ busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client FindProxyForURL ss "http://www.google.com" "google.com"
s "PROXY proxy.your.domain.com:<port>"
If a proxy server is not available, or if :command:`pacrunner` is running
without a PAC file, :command:`FindProxyForURL` will return "DIRECT". Otherwise,
it should return your local proxy settings as shown above.
.. code:: console
$ busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client FindProxyForURL ss "http://www.google.com" "google.com"
s "DIRECT"
Once :command:`pacdiscovery` is able to look up :command:`WPAD`, restart the
:command:`pacrunner` service:
.. code:: console
$ systemctl stop pacrunner
$ systemctl restart pacdiscovery
.. note::
A "domain" or "search" entry in :file:`/etc/resolv.conf` is required for short
name lookups to resolve. The :file:`resolv.conf` man page has additional
details.
source/clear-linux/guides/deploy-at-scale.rst
+1 -1
View File
@@ -269,7 +269,7 @@ challenges your monitoring systems, and business continuity plans.
.. _`mixin process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixin
.. _`mixer process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixer
.. _`downloads page`: https://download.clearlinux.org/image/
.. _`downloads page`: https://cdn.download.clearlinux.org/image/
.. _`containers page`: https://clearlinux.org/containers
.. _`systemd journal-remote service`: https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html
.. _`native telemetry solution`: https://clearlinux.org/features/telemetry
source/clear-linux/guides/guides.rst
-3
View File
@@ -20,9 +20,6 @@ Clear Linux Tooling
:maxdepth: 1
:glob:
clearlinux/*
telemetrics/telemetrics
Maintenance
===========
source/clear-linux/guides/maintenance/architect-lifecycle.rst
+76
View File
@@ -0,0 +1,76 @@
.. _architect-lifecycle:
Architect the life-cycle of |CL-ATTR|
#####################################
This guide provides DevOps with a model to architect the life-cycle of a |CL|
derivative that integrates custom software and content using distinct
workflows.
Maintaining a |CL| derivative requires:
* Monitoring upstream |CL| for new releases
* Building software packages and staging
* Employing CI/CD automation for building releases
* Integrating Quality Assurance for testing and validation
This guide provides the foundation of the recommended infrastructure.
.. contents::
:local:
:depth: 1
Prerequisites
*************
* A repository with software RPM artifacts and a CI/CD system with a |CL|
machine for building `mixes`
* Experience using :ref:`mixer <mixer>` to create a |CL|-based distro
* Experience using :ref:`swupd <swupd-guide>` for maintaining the |CL|
build environment
* Familiarity with |CL| architecture and reuse of its content in releases
Description
***********
Coordinated infrastructure is deployed to automate the life-cycle
of your |CL| derivative. We divide deployment of this infrastucture in two
parts: *Content Workflow*; and *Release Workflow*, shown in Figure 1. Distro Factory manages the *Release Workflow* while capturing the requirements for
maintaining a long-term release cadence.
.. figure:: figures/architect-lifecycle-1.png
:scale: 100%
:alt: Architect the life-cycle
Figure 1: Architect the life-cycle
Content workflow
****************
The *Content Workflow* (Figure 1) orchestrates the processes used to manage
the creation of content for the distribution. This includes everything from detecting a new release in a custom software repository to generating RPM package files. The RPM files serve as intermediary artifacts that track software dependencies and provide file-level data consumed in a *Release Workflow*. The `Watcher Pipeline`_ checks |CL| and a content provider, such as Koji, to determine if a new release is necessary.
Release workflow
****************
The *Release Workflow* (Figure 1) gathers the content of the RPMs and
ensures it can be consumed by :ref:`mixer <mixer>`. A content web server
hosts the |CL| derivative, to which targets connect for updating their OSes.
As an integral part of this toolchain, the *Release Pipeline* enables these
derivatives to incorporate |CL| content into their own custom
content. The *Watcher Pipeline* triggers the `Release Pipeline`_ to create
new releases.
Implementation
**************
Distro factory implements the *Release workflow*. To get started on a full implementation, visit |CL| `Distro factory documentation`_.
.. _Distro factory documentation: https://github.com/clearlinux/clr-distro-factory/wiki#clear-linux-distro-factory
.. _Release Pipeline: https://github.com/clearlinux/clr-distro-factory/wiki/Release
.. _Watcher Pipeline: https://github.com/clearlinux/clr-distro-factory/wiki/Watcher
source/clear-linux/guides/maintenance/autospec.rst
+293 -142
View File
@@ -1,178 +1,167 @@
.. _autospec:
Build RPMs with autospec
########################
autospec
########
This guide shows you how to create RPMs with :ref:`autospec <autospec-about>`,
a tool that assists in automated creation and maintenance of RPM packaging
on |CL-ATTR|.
**autospec** is a tool to assist in the automated creation and maintenance of
RPM packaging in |CL-ATTR|. Where a standard RPM build process using
:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec
requires only a tarball and package name to start.
See our :ref:`autospec concept page <autospec-about>` for a detailed explaination
of how ``autospec`` works on |CL|. For a general understanding of how RPMs work,
we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ .
.. contents::
:local:
:depth: 1
Description
***********
The autospec tool attempts to infer the requirements of the :file:`.spec` file
by analyzing the source code and :file:`Makefile` information. It will
continuously run updated builds based on new information discovered from build
failures until it has a complete and valid :file:`.spec` file. If needed, you
can influence the behavior of autospec and customize the build by providing
optional `control files`_ to the autospec tool.
autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for
additional information on using mock.
For a general understanding of how RPMs work, visit the `rpm website`_ or the
`RPM Packaging Guide`_ .
How it works
************
Learn the autospec tool set up and process.
.. contents::
:local:
:depth: 1
Prerequisites
*************
=============
This guide requires that you:
The setup for building source in |CL| must be completed before using the
autospec tool.
* Have installed |CL| on a host machine or virtual environment. For detailed
instructions on installing |CL|, visit the :ref:`get-started` section.
Refer to `Setup environment to build source`_ for instructions on completing
setup.
* :ref:`install-tooling`
Create an RPM
=============
.. _install-tooling:
The basic autospec process is described in the following steps:
Install the |CL| tooling framework
==================================
#. The :command:`make autospec` command generates a :file:`.spec` file based on
analysis of code and existing control files.
#. Install the `os-clr-on-clr` developer bundle on your host system.
Any control files should be located in the same directory as the resulting
:file:`.spec` file.
.. code-block:: bash
View the `autospec README`_ for more information on `control files`_.
sudo swupd bundle-add os-clr-on-clr
#. autospec creates a build root with mock config.
#. Download the :file:`user-setup.sh` script.
#. autospec attempts to build an RPM from the generated :file:`.spec`.
.. code-block:: bash
#. autospec detects any missed declarations in the :file:`.spec`.
curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh
#. If build errors occur, autospec will scan the build log to try and detect
the root cause.
#. Make :file:`user-setup.sh` executable.
#. If autospec detects the root cause and knows how to continue, it will restart
the build automatically at step 1 with updated build instructions.
.. code-block:: bash
#. Otherwise, autospec will stop the build for user inspection to resolve the
errors. Respond to the build process output by fixing source code issues
and/or editing control files to resolve issues, which may include
dependencies or exclusions. See `autospec README`_ for more information on
control files.
chmod +x user-setup.sh
The user resumes the process at step 1 after errors are resolved.
#. Run the script as an unprivileged user.
If a binary dependency doesn't exist in |CL|, you will need to build it
before running autospec again.
.. code-block:: bash
Following these steps, autospec continues to rebuild the package, based on
new information discovered from build failures, until it has a valid
:file:`.spec`. If no build errors occur, RPM packages are successfully built.
./user-setup.sh
Examples
********
#. After the script completes, log out and log in again to complete
the setup process.
Complete `Setup environment to build source`_ before using these examples.
The `user-setup script`_ creates a folder called :file:`clearlinux`, which
contains the :file:`Makefile`, :file:`packages`, and :file:`projects`
subfolders.
.. contents::
:local:
:depth: 1
The :file:`projects` folder contains the main tools, `autospec`
and `common`, used for making packages in |CL|.
Example 1: Build RPM with existing spec file
============================================
Create a RPM with autospec
**************************
This example shows how to build a RPM from a pre-packaged upstream package, with
an existing spec file. The example uses the ``dmidecode`` package.
Choose one of the following options to build RPMs and manage source
code:
* :ref:`build-a-new-rpm` and spec file using ``make autospecnew``.
* :ref:`build-source-code-with-existing-spec-file` using ``make build``, without changing the
spec file.
* :ref:`generate-a-new-spec-file` using ``make autospec``, based on changes in the control files.
.. _build-a-new-rpm:
Option 1: Build a new RPM
=========================
Use this method to build a new RPM with no spec file. In this example,
we build a new helloclear RPM.
#. Navigate to the autospec workspace.
#. Navigate to the autospec workspace and clone the ``dmidecode`` package:
.. code-block:: bash
cd ~/clearlinux
#. Enter the command:
.. code-block:: bash
make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz"
NAME="helloclear"
make clone_dmidecode
.. note::
For a local tarball, use this type of *URL*: \file://<absolute-path-to-tarball>
#. If build failures or dependency issues occur, continue below.
Otherwise, skip directly to `Next steps`_.
#. Navigate to the specific package.
You can clone all package repos at once using:
.. code-block:: bash
cd ~/clearlinux/packages/[package-name]
make [-j NUM] clone-packages
#. Respond to the build process output by editing control files to resolve
issues, which may include dependencies or exclusions.
See `autospec readme`_
The optional NUM is the number of threads to use.
#. Run this command:
For a list of available packages, view the
:file:`~/clearlinux/projects/common/packages` file.
.. code-block:: bash
make autospec
Repeat the last two steps above until all errors are resolved and you
complete a successful build.
**Congratulations!**
You've successfully created a RPM.
Skip to `Next steps`_.
.. _build-source-code-with-existing-spec-file:
Option 2: Build source code with an existing spec file
======================================================
Use this method if you only want to build the RPM using the spec file. This
method assumes that a spec file already exists. In this example, we run a
``make build`` on the ``dmidecode`` package.
#. Navigate to the ``dmidecode`` package in clearlinux:
#. Navigate to the local copy of the ``dmidecode`` package and build it:
.. code-block:: bash
cd ~/clearlinux/packages/dmidecode/
#. To download the tarball and build, run the command:
.. code-block:: bash
make build
**Congratulations!**
#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are
in :file:`./results`.
You've successfully created a RPM.
Example 2: Build a new RPM
==========================
Skip to `Next steps`_.
This example shows how to build a new RPM with no spec file. The example will
create a simple helloclear RPM.
.. _generate-a-new-spec-file:
Option 3: Generate a new spec file with a pre-defined package
=============================================================
Use this method to modify an existing package. In this example, you will
modify an existing |CL| package called ``dmidecode`` to create a custom
RPM. You will make a simple change to this package, change the revision to
a new number that is higher than the |CL| OS version, and rebuild the package.
#. Navigate to clearlinux:
#. Navigate to the autospec workspace and build the helloclear RPM. The
:file:`Makefile` provides a :command:`make autospecnew` that can
automatically generate an RPM package using the autospec tool. You must pass
the URL to the source tarball and the NAME of the RPM you wish to create:
.. code-block:: bash
cd ~/clearlinux
make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear"
#. Copy the ``dmidecode`` package.
The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and
additional RPMs are in :file:`./packages/helloclear/results`.
Example 3: Generate a new spec file with a pre-defined package
==============================================================
This example shows how to modify an existing package to create a custom RPM. In
this example you will make a simple change to the ``dmidecode`` package and
rebuild the package.
#. Navigate to the autospec workspace and clone the ``dmidecode`` package:
.. code-block:: bash
cd ~/clearlinux
make clone_dmidecode
#. Navigate into the *dmidecode* directory:
@@ -181,9 +170,9 @@ a new number that is higher than the |CL| OS version, and rebuild the package.
cd packages/dmidecode
#. With an editor, open the :file:`excludes` file and add these lines:
#. Open the :file:`excludes` file with an editor and add these lines:
.. code-block:: bash
.. code-block:: console
/usr/bin/biosdecode
/usr/bin/ownership
@@ -197,49 +186,211 @@ a new number that is higher than the |CL| OS version, and rebuild the package.
These files aren't needed by dmidecode, so we can remove them without
any issues.
#. Save the file and exit.
#. At :file:`~/clearlinux/packages/dmidecode`, build the modified
``dmidecode`` package:
#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package:
.. code-block:: bash
make autospec
When the process completes, you will see new RPM packages in the
:file:`results/` folder.
#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`.
#. To view the new RPM packages, enter:
Example 4: Provide control files to autospec
============================================
This example shows how to modify control files to correct build failures that
autospec is unable to resolve. In this example you will add a missing license
and dependencies in order for autospec to complete a successful build.
#. Navigate to the autospec workspace:
.. code-block:: bash
ls /clearlinux/packages/dmidecode/results/
cd ~/clearlinux
**Congratulations!**
#. If you have not already, clone all upstream package repos:
You've successfully created a RPM.
.. code-block:: bash
Next steps
make [-j NUM] clone-packages
The optional NUM is the number of threads to use.
.. note::
In a later step of this example, we will search the cloned package repos
for a missing dependency.
#. Build the opae-sdk RPM:
.. code-block:: bash
make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk"
This will give an error for a missing license file:
.. code-block:: console
[FATAL] Cannot find any license or opae-sdk.license file!
#. Navigate to the package with build failures:
.. code-block:: bash
cd packages/opae-sdk
#. Add a license:
.. code-block:: bash
echo "Intel Corporation" > opae-sdk.license
#. Run autospec again:
.. code-block:: bash
make autospec
This will result in a generic error:
.. code-block:: console
[FATAL] Build failed, aborting
#. Open the build log to view the error details:
.. code-block:: bash
cat ./results/build.log
In the build log, you will find details for the specific failures. In this
instance, there are missing dependencies:
.. code-block:: console
CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files:
CJSON_LIBRARY
linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib
json-c_LIBRARIES
linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
libuuid_LIBRARIES
linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
#. Search the spec files of upstream |CL| packages to see if the json-c library
is availabe. In this case, it does exist and we'll add the json-c 'dev'
package into the buildreq_add:
.. code-block:: bash
grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec
echo "json-c-dev" >> buildreq_add
.. note::
This search step works only if the user cloned all of the upstream package
repos. In this example, upstream package repos were cloned in a previous
step.
#. Search the spec files of upstream |CL| packages to see if the libuuid library
is available. In this case, it exists in the util-linux package, so we'll add
util-linux-dev package into the buildreq_add:
.. code-block:: bash
grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec
echo "util-linux-dev" >> buildreq_add
#. Run autospec again and find the successfully-generated RPMs in the rpms
directory:
.. code-block:: bash
make autospec
.. note::
If you need a dependency that does not exist in the |CL| repo, you must first
build it manually (see `Example 2: Build a new RPM`_), then add the repo so
that autospec knows the package exists. For example:
.. code-block:: bash
cd ~/clearlinux/packages/<package-name>
make repoadd
make repostatus
You only need to add the dependency to the :file:`buildreq_add` control file
if autospec is not able to automatically find the correct dependency on its
own.
References
**********
Now you can create a custom bundle with your new RPM and use it with |CL|:
Reference the `autospec README`_ for details regarding autospec commands and
options.
* Use the :ref:`Mixer tool <mixer>` to add a new bundle to your derivative of |CL|.
* Use the :ref:`Mixin tool <mixin>` to customize your upstream |CL| installation with a new bundle.
Setup environment to build source
=================================
.. _install-tooling-after-header:
Setup of the workspace and tooling used for building source in |CL| is mostly
automated for you with a setup script. It uses tools from the
:command:`os-clr-on-clr` bundle.
The setup script creates a workspace in the :file:`clearlinux` folder, with the
subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The
:file:`projects` folder contains the main tools used for making packages in
|CL|: `autospec` and `common`.
Follow these steps to setup the workspace and tooling for building source:
#. Install the :command:`os-clr-on-clr` bundle:
.. code-block:: bash
sudo swupd bundle-add os-clr-on-clr
#. Download the :file:`user-setup.sh` script:
.. code-block:: bash
curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh
#. Make :file:`user-setup.sh` executable:
.. code-block:: bash
chmod +x user-setup.sh
#. Run the script as an unprivileged user:
.. code-block:: bash
./user-setup.sh
#. After the script completes, log out and log in again to complete the setup
process.
#. Set your Git user email and username for the repos on your system:
.. code-block:: bash
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
This global setting is used by |CL| tools that make use of Git.
.. _install-tooling-end:
Related topics
**************
* :ref:`Mixer tool <mixer>`
* :ref:`Mixin tool <mixin>`
* :ref:`autospec <autospec-about>`
* :ref:`Bundles <bundles-about>`
.. _rpm website: http://rpm.org
.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/
.. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh
.. _autospec readme: https://github.com/clearlinux/autospec
.. _autospec README: https://github.com/clearlinux/autospec
.. _control files: https://github.com/clearlinux/autospec#control-files
.. _mock wiki: https://github.com/rpm-software-management/mock/wiki
.. _rpm website: http://rpm.org
.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/
source/clear-linux/guides/maintenance/enable-user-space.rst
+32 -75
View File
@@ -6,22 +6,15 @@ Create and enable a new user space
This section provides steps to complete the following basic setup tasks for
a newly installed |CL-ATTR| system:
* Create a new user.
* Update the OS to its most current version using `swupd`.
* Install the most common applications for system administrators and
developers using bundles.
* Set up a new user and add the new user to the `wheel` group.
* Install a GUI to test `sudo` privileges.
.. note::
Log in as the root user to complete the tasks in this
section.
.. contents::
:local:
:depth: 1
Create a new user
******************
*****************
To create a new user and set a password for that user, enter the following
commands as a root user:
commands as a `root` user:
.. code-block:: bash
@@ -33,29 +26,8 @@ including the password for that user. The :command:`passwd` command prompts
you to enter a new password. Retype the new password for the new user
account just created.
Install and update the OS software to its current version
*********************************************************
|CL| has a unique application and architecture to add and update applications
and to perform system updates called software update utility or
:command:`swupd`. Software applications are installed as bundles using the
sub-command :command:`bundle-add`.
The `sysadmin-basic` bundle installs the vast majority of
applications useful to a system administrator.
Install the `sysadmin-basic` bundle:
.. code-block:: bash
swupd bundle-add sysadmin-basic
We provide the full list of bundles and packages installed with the
`sysadmin-basic`_ bundle. Additionally, we have listed
`all bundles`_ for |CL|, active or deprecated. Click any bundle on the
list to view the manifest of the bundle.
Set up a new user and add the new user to the `wheel` group
***********************************************************
Add the new user to the `wheel` group
*************************************
Before logging off as root and logging into your new user account,
enable the :command:`sudo` command for your new `<userid>`.
@@ -73,65 +45,50 @@ To be able to execute all applications with root privileges, add the
To log off as root, enter :command:`exit`.
The command will bring you back to the `login:` prompt.
#. Enter the new `<userid>` and the password created earlier.
You will now be in the home directory of `<userid>`. The bundle
`sysadmin-basic`_ contains the majority of applications that a system
administrator would want, but it does not include a graphical user
interface. The `desktop` bundle includes the GNOME\* Display Manager and
additional supporting applications.
You will now be in the home directory of `<userid>`.
Install a GUI to test `sudo` privileges
========================================
.. note::
Install and update the OS software to its current version
*********************************************************
If you are following this sequence after just setting up the
pre-configured VMware\* virtual machine from the repo, you must
:ref:`increase virtual disk size<increase-virtual-disk-size>` or the
following step will fail.
The |CL| software utility :ref:`swupd <swupd-guide>` allows you to perform system updates while reaping the benefits of upstream development.
To test the :command:`sudo` command and ensure it is set up correctly,
install the GNOME Display Manager (gdm) and start it.
To update your newly installed OS, run:
#. To install the the GNOME Display Manager using :command:`swupd`, enter
the following command:
.. code-block:: bash
.. code-block:: bash
sudo swupd update
sudo swupd bundle-add desktop
Add a bundle
************
#. To start the GNOME Display Manager, enter the following command:
Software applications are installed as bundles using the command
:command:`swupd bundle-add`. Experienced Linux* users might compare `swupd`
to running :command:`apt-get` or :command:`yum install` for package
management. Yet |CL| manages packages at the level of bundles, which
are integrated stacks of packages.
.. code-block:: bash
For example, the `sysadmin-basic` bundle installs the majority of applications useful to a system administrator. To install it, enter:
systemctl start gdm
.. code-block:: bash
#. The system prompts you to authenticate the user. Enter the password for
`<userid>`, and the GNOME Display Manager starts as shown in Figure
1:
swupd bundle-add sysadmin-basic
.. figure:: figures/gnomedt.png
:scale: 50 %
:alt: Gnome Desktop
View a full list of bundles and packages installed with the `sysadmin-basic`_ bundle. You can also view `all bundles`_ for |CL|, active or deprecated.
Figure 1: :guilabel:`Gnome Desktop`
Expand your knowledge of :command:`swupd` and check out our developer resources:
#. To start the GNOME Display Manager each time you start your system, enter
the following command:
.. code-block:: bash
systemctl enable gdm
* :ref:`swupd-guide`
* :ref:`developer-workstation`
Next steps
***********
**********
With your system now running |CL|, many opportunities exist.
Check out our guides and tutorials.
Visit the :ref:`tutorials <tutorials>` page for examples on using your |CL|
system.
* :ref:`guides`
* :ref:`tutorials`
.. _`sysadmin-basic`:
https://github.com/clearlinux/clr-bundles/blob/master/bundles/sysadmin-basic
source/clear-linux/guides/maintenance/figures/architect-lifecycle-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

source/clear-linux/guides/maintenance/figures/distro-factory-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

source/clear-linux/guides/maintenance/increase-virtual-disk-size.rst
+1 -1
View File
@@ -166,4 +166,4 @@ Resize the filesytem
Congratulations! You have resized the disk, partition, and filesystem. At
this point, the increase in disk capacity is usable.
.. _releases: https://download.clearlinux.org/releases/
.. _releases: https://cdn.download.clearlinux.org/releases/
source/clear-linux/guides/maintenance/ister.rst
+94
View File
@@ -0,0 +1,94 @@
.. _ister:
ister.py image builder
######################
The `ister.py tool`_ is a template based installer used by |CL-ATTR| to produce
images for each release. The same ister tool is available for use in |CL| to
create custom images based on an upstream image.
.. contents::
:local:
:depth: 1
Description
***********
|CL| is a rolling release and produces on average 10 releases per week using the
ister tool. With each release we produce multiple
`image types for different environments`_ and use cases such as installers,
Hyper-V, KVM, or VMWare.
Each image has a JSON configuration file used by ister to generate the image.
These JSON configuration files describe the image type, partitions, version,
and which bundles will be preinstalled by default with the image. For each image
type we produce, the corresponding JSON configuration file for the image is also
published.
The :ref:`mixer<mixer>` tool also uses ister to build images for your custom
mix. Like upstream images, a JSON configuration file is defined for the image,
which ister uses to generate the image. Refer to the :ref:`mixer<mixer>` guide
for instruction on using ister to build an image for a custom mix.
Examples
********
Recreate an upstream image
==========================
The published configuration files for upstream images may be used to recreate an
image, for example when you want to:
* Use an older version of |CL| and the image is no longer available (only after
March 2017).
* Customize the partitions of an image.
* Customize the bundles preinstalled in an image.
* Run your own post installation script.
Follow these steps to recreate an upstream image based on the image's JSON
configuration file:
#. Install the :command:`os-installer` bundle. Refer to `Install a bundle`_ for
more details.
#. Download the `ister.py tool`_ and grant it sudo privileges.
#. Download the JSON configuration file for the desired image:
* `Configuration files for the current release`_
* `Previous releases`_ (only after March 2017)
For a previous release, navigate to `Previous releases`_, select the version
you want, and find the JSON configuration file under
:file:`/clear/config/image`. For example:
``https://cdn.download.clearlinux.org/releases/15700/clear/config/image/``
#. Download “PostNonChroot” script (if applicable).
The JSON configuration file for the image may have an accompanying
“PostNonChroot” script that is executed at the end of the image creation
process. If it does, download the script and make it executable.
#. Edit the JSON configuration file as needed.
#. If your configuration file has an accompanying "PostNonChroot" script, change
the default path of the script to match your path.
#. Generate the new image with the following command:
.. code-block:: bash
sudo ister.py -t [JSON configuration]
Related topics
**************
* :ref:`mixer`
* :ref:`bulk-provision`
.. _ister.py tool: https://github.com/bryteise/ister
.. _image types for different environments: https://cdn.download.clearlinux.org/image/README-IMAGES.html
.. _Configuration files for the current release: https://cdn.download.clearlinux.org/current/config/image/
.. _Previous releases: https://cdn.download.clearlinux.org/releases/
.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle
source/clear-linux/guides/maintenance/kernel-development.rst
+447
View File
@@ -0,0 +1,447 @@
.. _kernel-development:
Kernel development
##################
This document shows how to obtain and compile a Linux* kernel source
using |CL-ATTR| development tooling.
The `kernels available`_ in |CL| aim to be performant and practical. In some
cases, it may be necessary to modify the kernel to suit your specific needs
or test new kernel code as a developer.
.. contents::
:local:
:depth: 1
:backlinks: top
Source RPM files (SRPM) are also available for all |CL| kernels, and can be
used for development instead. Select this link to view the latest `source RPM files`_.
Request changes be included with the |CL| kernel
************************************************
If the kernel modification you need is already open source and likely to be
useful to others, consider submitting a request to include it in the
|CL| kernels.If your change request is accepted, you do not need to maintain your own modified kernel.
Make enhancement requests to the |CL| `distribution on GitHub`_ .
Set up kernel development environment
*************************************
In some cases, it may be necessary to modify the kernel to suit your specific
needs or to test new kernel code.
You can build and install a custom kernel; however you must:
* Disable Secure Boot
* Maintain any updates to the kernel going forward
To create a custom kernel, start with the |CL| development environment.
Then make changes to the kernel, build it, and install it.
Install the |CL| development tooling framework
==============================================
.. include:: autospec.rst
:start-after: install-tooling-after-header:
:end-before: install-tooling-end:
Clone the kernel package
========================
Clone the existing kernel package repository from |CL| as a starting point.
#. Clone the Linux kernel package from |CL|. Using the
:command:`make clone_<PACKAGENAME>` command in the
:file:`clearlinux/` directory clones the package from the
`clearlinux-pkgs GitHub`_.
.. code-block:: bash
cd ~/clearlinux
make clone_linux
#. Navigate into the cloned package directory.
.. code-block:: bash
cd ~/clearlinux/packages/linux
The "linux" package is the kernel that comes with |CL| in the `kernel-native`
bundle. Alternatively, you can use a different kernel variant as the base for
modification. For a list of kernel package names which you can clone instead,
see the `clearlinux-pkgs GitHub`_.
.. note::
The latest version of the |CL| kernel package is pulled as a starting
point. An older version can pulled by switching to different git tag by using :command:`git checkout tag/<TAG_NAME>`.
Change the kernel version
=========================
|CL| tends to use the latest kernel available from `kernel.org`_, the Linux
upstream. The kernel version that will be built can be changed in the
RPM SPEC file. While most packages in Clear Linux are typically packaged
using :ref:`autospec-about`, the kernel is not. This means control files
provided by autospec are not available and changes must be made manually.
#. Open the Linux kernel package RPM SPEC file in an editor.
.. code-block:: bash
$EDITOR linux.spec
#. Modify the Version, Release, and Source0 URL entries at the top of the
file to change the version of Linux kernel that will be compiled.
A list of current and available kernel release can be found on
`kernel.org`_.
.. code-block:: bash
Name: linux
Version: 4.20.8
Release: 696
License: GPL-2.0
Summary: The Linux kernel
Url: http://www.kernel.org/
Group: kernel
Source0: https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.20.8.tar.xz
Source1: config
Source2: cmdline
%define ktarget native
.. note::
- Consider changing the Name from *linux* in the RPM spec file to easily identify a modified kernel.
- Consider changing the ktarget from *native* in the RPM spec file to easily identify a modified kernel.
#. Commit and save the changes to the file.
Pull a copy of the Linux kernel source code
===========================================
Obtain a local copy of the source code to make modifications against.
#. Run make sources to pull the kernel source code specified in the RPM
SPEC file. In the example, it downloads the :file:`linux-4.20.8.tar.xz` file.
.. code-block:: bash
make sources
#. Extract the kernel source code archive. This will create a working copy
of the Linux source that you can modify.
.. code-block:: bash
tar -xvf linux-4.20.8.tar.xz
#. Navigate to the extracted directory. In this example, it has been
extracted into a :file:`linux-4.20.8` directory.
.. code-block:: bash
cd linux-4.20.8/
Customize the Linux kernel source
*********************************
After the kernel sources have been obtained, customizations to the kernel
configuration or source code can be made for inclusion with the kernel
build. These customizations are optional.
Modify kernel configuration
===========================
The kernel source has many configuration options available to pick support for different hardware and software features.
These configuration values must be provided in the :file:`.config` file at
compile time. You will need to make modifications to the :file:`.config`
file, and include it in the kernel package.
#. Make sure you have followed the steps to
`Pull a copy of the Linux kernel source code`_ and are in the kernel
source working directory.
#. If you have an existing :file:`.config` file from an old kernel, copy it
into the working directory as :file:`.config` for comparison.
Otherwise, use the |CL| kernel configuration file as template
.. code-block:: bash
cp ~/clearlinux/packages/linux/config .config
#. Make any desired changes to the :file:`.config` using a kernel
configuration tool. Below are some popular options:
- :command:`$EDITOR .config` - the .config file can be directly edited
for simple changes with names that are already known.
- :command:`make config` - a text-based tool that asks questions
one-by-one to decide configuration options.
- :command:`make menuconfig` - a terminal user interface that provides
menus to decide configuration options.
- :command:`make xconfig` - a graphical user interface that provides
tree views to decide configuration options.
More configuration tools can be found by looking at the make help:
:command:`make help | grep config`
#. Commit and save the changes to the :file:`.config` file.
#. Copy the :file:`.config` file from the kernel source directory into
the kernel package directory as :file:`config` for inclusion in the build.
.. code-block:: bash
cp .config ../config
Modify kernel source code
=========================
Changes to kernel code are applied with patch files. Patch files are
formatted git commits that can be applied to the main source code.
You will need to obtain a copy of the source code,
make modifications, generate patch file(s), and add them to the RPM SPEC
file for inclusion during the kernel build.
If you have a large number of patches or a more complex workflow,
consider using a patch management tool in addition to Git such as
`Quilt`_.
#. Make sure you have followed the steps to
`Pull a copy of the Linux kernel source code`_ and are in the kernel
source working directory.
#. Initialize the kernel source directory as a new git repo and create a
commit with all the existing source files to begin tracking changes.
.. code-block:: bash
git init
git add -A
git commit -m "Initial commit of Linux kernel source"
#. Apply patches provided by the |CL| kernel package to the kernel source
in the working directory.
.. code-block:: bash
git am ../*.patch
#. Make any of your desired code changes to the Linux source code files.
#. Track and commit your changes to the local git repo.
.. code-block:: bash
git add <FILENAME>
git commit -m "My patch for driver A" <FILENAME>
#. Generate a patch file based on your git commits.
<n> represents the number of local commits to create patch file.
See the `git-format-patch Documentation`_ for detailed information
on using :command:`git format-patch`
.. code-block:: bash
git format-patch -<n>
#. Copy the patch files from the patches directory in the linux
source tree to the RPM build directory.
.. code-block:: bash
cp *.patch ~/clearlinux/packages/linux/
#. Navigate back to the RPM build directory.
.. code-block:: bash
cd ~/clearlinux/packages/linux/
#. Open the Linux kernel package RPM SPEC file in an editor.
.. code-block:: bash
$EDITOR linux.spec
#. Locate the section of the SPEC file that contains existing patch
variable definitions and append your patch file name. Ensure the
patch number does not collide with an existing patch.
In this example, the patch file is called
:file:`2001-my-patch-for-driver-A.patch`
.. code-block:: bash
#
# Small Clear Linux Tweaks
#
Patch0501: 0501-zero-extra-registers.patch
Patch0502: 0502-locking-rwsem-spin-faster.patch
#Serie1.name WireGuard
#Serie1.git https://git.zx2c4.com/WireGuard
#Serie1.tag 00bf4f8c8c0ec006633a48fd9ee746b30bb9df17
Patch1001: 1001-WireGuard-fast-modern-secure-kernel-VPN-tunnel.patch
#Serie1.end
Patch2001: 2001-my-patch-for-driver-A.patch
#. Locate the section of the SPEC file further down that contains
patch application and append your patch file number used in the step above.
In this example, patch2001 is added.
.. code-block:: bash
#
# Small tweaks
#
%patch0501 -p1
%patch0502 -p1
#Serie1.patch.start
%patch1001 -p1
#Serie1.patch.end
%patch2001 -p1
#. Commit and save the changes to the RPM SPEC file.
Modify kernel boot parameters
=============================
The kernel boot options are passed from the bootloader to the kernel with
command-line parameters.
While temporary changes can be made to kernel parameters on a running
system or on a during boot, you can also modify the default parameters that
are persistent and distributed with a customized kernel.
#. Open the kernel :file:`cmdline` file in an editor.
.. code-block:: bash
$EDITOR cmdline
#. Make any desired change to the kernel parameters.
For example, you can remove the :command:`quiet` parameter to see more
verbose output of kernel log messages during the boot process.
#. Commit and save the changes to the :file:`cmdline` file.
See the `Kernel parameters documentation`_ for a list of available
parameters.
Build and install the kernel
****************************
After changes have been made to the kernel source and RPM SPEC file,
the kernel is ready to be compiled and packaged into an RPM.
The |CL| development tooling makes use of :command:`mock` environments to
isolate building of packages in a sanitized workspace.
#. Start the compilation process by issuing the :command:`make build`
command. This process is typically resource intensive and will take a while.
.. code-block:: bash
make build
.. note::
The `ccache plugin for mock`_ can be enabled to help speed up any future rebuilds of the kernel package by caching compiler outputs and reusing them.
#. The result will be multiple :file:`.rpm` files in the :file:`rpms`
directory as output.
.. code-block:: bash
ls rpms/
The kernel RPM will be named
:file:`linux<NAME>-<VERSION>-<RELEASE>.x86_64.rpm`
#. The kernel RPM file can be input to the :ref:`mixer` to create a
custom bundle and mix of |CL|.
Alternatively, the kernel RPM bundle can be installed manually on a local
machine for testing. This approach works well for individual development or
testing. For a more scalable and customizable approach, consider using the
:ref:`mixer` to provide a custom kernel with updates.
1. Install the kernel onto the local system by extracting the RPM with the
:command:`rpm2cpio` command.
.. code-block:: bash
rpm2cpio linux<NAME>-<VERSION>-<RELEASE>.x86_64.rpm | (cd /; sudo cpio -i -d -u -v);
#. Update the |CL| boot manager using :command:`clr-boot-manager` and reboot.
.. code-block:: bash
sudo clr-boot-manager list-kernels
sudo clr-boot-manager set-kernel org.clearlinux.<TARGET>.<VERSION>-<RELEASE>
sudo reboot
#. After a reboot, verify the customized kernel is running.
.. code-block:: bash
uname -a
Related topics
**************
* :ref:`kernel-modules`
* :ref:`mixer`
.. _kernels available: https://clearlinux.org/documentation/clear-linux/reference/compatible-kernels
.. _distribution on GitHub: https://github.com/clearlinux/distribution/issues/new/choose
.. _source RPM files: https://cdn.download.clearlinux.org/current/source/SRPMS/
.. _Quilt: http://savannah.nongnu.org/projects/quilt
.. _clearlinux-pkgs GitHub: https://github.com/clearlinux-pkgs
.. _kernel.org: https://www.kernel.org/
.. _Kernel parameters documentation: https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt
.. _ccache plugin for mock: https://fedoraproject.org/wiki/Mock/Plugin/CCache?rd=Subprojects/Mock/Plugin/CCache
.. _git-format-patch Documentation: https://git-scm.com/docs/git-format-patch
.. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh
source/clear-linux/guides/maintenance/kernel-modules-dkms.rst
+319
View File
@@ -0,0 +1,319 @@
.. _kernel-modules-dkms:
Add kernel modules with DKMS
############################
Kernel modules are additional pieces of software capable of being inserted
into the Linux kernel to add functionality, such as a hardware driver.
Kernel modules may already be part of the Linux source tree (in-tree) or may
come from an external source, such as directly from a vendor (out-of-tree).
In cases where drivers beyond those enabled by default in |CL-ATTR| are
needed it may be necessary to manually build out-of-tree modules.
Out-of-tree kernel modules can be can be `manually built and maintained
<kernel-modules>`_. Out-of-tree kernel modules can also be managed with the
`Dynamic Kernel Module System (DKMS)`_ on |CL| using the instructions in this
document.
:abbr:`DKMS (Dynamic Kernel Module System)` is a framework that facilitates
the building and installation of kernel modules. This allows |CL| to provide
hooks that automatically rebuild modules against new kernel versions.
.. contents:: :local:
:depth: 1
:backlinks: top
.. include:: kernel-modules.rst
:start-after: kernel-modules-availability-begin:
:end-before: kernel-modules-availability-end:
Install DKMS on |CL|
====================
.. _kernel-modules-dkms-install-begin:
The *kernel-native-dkms* bundle provides the :command:`dkms` program and
Linux kernel headers, which are required for compiling kernel modules.
The *kernel-native-dkms* bundle also:
* Adds a systemd update trigger
(:file:`/usr/lib/systemd/system/dkms-new-kernel.service`) to automatically
run DKMS to rebuild modules after a kernel upgrade occurs with :ref:`swupd
update <swupd-guide>`.
* Disables kernel modules signature verification by appending a kernel
command-line parameter (:command:`module.sig_unenforce`) from
:file:`/usr/share/kernel/cmdline.d/clr-ignore-mod-sig.conf`.
* Adds a notification to the Message of the Day (MOTD) indicating kernel
modules signature verification is disabled.
.. warning::
#. It is important to always review the output of :command:`swupd update` to
make sure kernel modules rebuilt against the new kernel successfully. This is
especially important for systems where a successful boot relies on a kernel
module.
Install the *kernel-native-dkms* or *kernel-lts-dkms* bundle:
#. Determine which kernel variant is running on |CL|. Only the *native*
and *lts* kernels are enabled to build and load out-of-tree kernel modules
with DKMS.
.. code-block:: bash
$ uname -r
5.XX.YY-ZZZZ.native
Ensure *.native* or *.lts* is in the kernel name.
#. Install the dkms bundle corresponding to the installed kernel.
*kernel-native-dkms* for the native kernel or *kernel-lts-dkms* for the
lts kernel.
.. code-block:: bash
sudo swupd bundle-add kernel-native-dkms
or
.. code-block:: bash
sudo swupd bundle-add kernel-lts-dkms
#. Update the |CL| bootloader and reboot.
.. code-block:: bash
sudo clr-boot-manager update
reboot
.. _kernel-modules-dkms-install-end:
Build, install, and load an out-of-tree module
==============================================
In some cases you may need an out-of-tree kernel module that is not available
through |CL|.
Prerequisites
-------------
You can build and load out-of-tree kernel modules, however you must:
* Disable Secure Boot in UEFI/BIOS. The loading of new out-of-tree modules
modifies the signatures Secure Boot relies on for trust.
* Have a kernel module package in the form of source code and/or precompiled
binaries.
This approach works well for individual use or testing. For a more
scalable and customizable approach, consider using the `mixer tool`_ to
provide a custom kernel and updates.
Obtain kernel module source
---------------------------
A :file:`dkms.conf` file inside of the kernel module's source code directory
is required to inform DKMS how the kernel module should be compiled.
Kernel modules may come packaged as:
- Source code without a dkms.conf
- Source code with a premade dkms.conf
- Source code with a premade dkms.conf and precompiled module binaries
- Precompiled module binaries only without source code
Precompiled kernel module binaries will not work on |CL| because it requires
kernel modules to be built against the same kernel source tree before they can
be loaded.
If you are only able to obtain source code without a dkms.conf, a
:file:`dkms.conf` file will need to be manually created.
#. Download the kernel module's source code.
- Review the available download options. Some kernel modules provide
separate archives which are specifically enabled for DKMS support.
- Review the README documentation because it often provides required
information to build the module with DKMS support.
.. code-block:: bash
curl -O http://<URL-TO-KERNEL-MODULE-SOURCE>.tar.gz
tar -xvf <KERNEL-MODULE-SOURCE>.tar.gz
cd <KERNEL-MODULE-SOURCE>/
cat README
Build kernel module with an existing dkms.conf
----------------------------------------------
If the kernel module maintainer packaged the source archive with the
:command:`dkms mktarball` command, the entire archive can be passed to the
:command:`dkms ldtarball` which will complete many steps for you.
The archive will contain the required :file:`dkms.conf` file, and may contain
a :file:`dkms_source_tree` directory and a :file:`dkms_binaries_only`
directory.
#. Run the :command:`dkms ldtarball` command against the kernel module archive.
.. code-block:: bash
dkms ldtarball <KERNEL-MODULE-SOURCE_WITH_DKMS>.tar.gz
:command:`dkms ldtarball` will place the kernel module source under
:file:`/usr/src/<MODULE-NAME>-<MODULE-VERSION>/`, build if necessary, and
add the module into the dkms tree.
#. Verify the kernel module is detected by checking the output of
:command:`dkms status`.
.. code-block:: bash
dkms status
#. Install the kernel module.
.. code-block:: bash
dkms install -m <MODULE-NAME> -v <MODULE-VERSION>
Build kernel module without an existing dkms.conf
-------------------------------------------------
If the kernel module source does not contain a :file:`dkms.conf` file or the
:command:`dkms ldtarball` command encounters errors, it needs to be manually
created.
Review the kernel module README documentation for guidance on what needs to be
in the :file:`dkms.conf` including special variables that may be required to
build successfully.
Here are some additional resources that can be used for reference:
* The DKMS manual page (:command:`man dkms`) shows detailed syntax in the
DKMS.CONF section
* `<https://help.ubuntu.com/community/Kernel/DkmsDriverPackage#Configure_DKMS>`_
(shows an example where a single package contains multiple modules)
* `<https://github.com/dell/dkms/blob/master/sample.conf>`_
The instructions below show a generic example:
#. Create or modify the :file:`dkms.conf` file inside of the extracted source
code directory.
.. code-block:: bash
$EDITOR dkms.conf
MAKE="make -C src/ KERNELDIR=/lib/modules/${kernelver}/build"
CLEAN="make -C src/ clean"
BUILT_MODULE_NAME=custom_module
BUILT_MODULE_LOCATION=src/
PACKAGE_NAME=custom_module
PACKAGE_VERSION=1.0
DEST_MODULE_LOCATION=/kernel/drivers/other
This example identifies a kernel module named *custom_module* with version
*1.0*.
#. Copy the kernel module source code into the :file:`/usr/src/` directory.
.. code-block:: bash
sudo mkdir /usr/src/<PACKAGE_NAME>-<PACKAGE_VERSION>
sudo cp -Rv . /usr/src/<PACKAGE_NAME>-<PACKAGE_VERSION>
.. note::
*<PACKAGE_NAME>* and *<PACKAGE_VERSION>* should match the entries in :file:`dkms.conf`
#. Add the kernel module to the DKMS tree so that it is tracked by DKMS.
.. code-block:: bash
sudo dkms add -m <MODULE-NAME>
#. Build the kernel module using DKMS. If the build encounters errors, the
:file:`dkms.conf` may need to be adjusted.
.. code-block:: bash
sudo dkms build -m <MODULE-NAME> -v <MODULE-VERSION>
#. Install the kernel module using DKMS.
.. code-block:: bash
sudo dkms install -m <MODULE-NAME> -v <MODULE-VERSION>
Load kernel module
------------------
By default, DKMS installs modules "in-tree" under :file:`/lib/modules` so the
:command:`modprobe` command can be used to load them.
#. Load the installed module with the :command:`modprobe` command.
.. code-block:: bash
sudo modprobe <MODULE-NAME>
#. Validate the kernel module is loaded.
.. code-block:: bash
lsmod | grep <MODULE-NAME>
.. include:: kernel-modules.rst
:start-after: kernel-modules-autoload-begin:
:end-before: kernel-modules-autoload-end:
Additional resources
====================
* `Dynamic Kernel Module System (DKMS) project on GitHub <https://github.com/dell/dkms>`_
* `Dell Linux Engineering Dynamic Kernel Module Support: From Theory to Practice <https://www.kernel.org/doc/ols/2004/ols2004v1-pages-187-202.pdf>`_
* `Linux Journal: Exploring Dynamic Kernel Module Support <https://www.linuxjournal.com/article/6896>`_
.. _`on GitHub`: https://github.com/clearlinux/distribution
.. _`mixer tool`: https://clearlinux.org/features/mixer-tool
.. _`Dynamic Kernel Module System (DKMS)`: https://github.com/dell/dkms
source/clear-linux/guides/maintenance/kernel-modules.rst
+104 -61
View File
@@ -1,88 +1,125 @@
.. _kernel-modules:
Add kernel modules
##################
Add kernel modules manually
###########################
Kernel modules are additional pieces of software capable of being inserted
into the Linux kernel to add functionality, such as a hardware driver.
Kernel modules may already be part of the Linux source tree (in-tree) or may
come from an external source, such as directly from a vendor (out-of-tree).
Kernel modules are additional pieces of software capable of being inserted
into the Linux kernel to add functionality, such as a hardware driver. Kernel
modules may already be part of the Linux source tree (in-tree) or may come
from an external source, such as directly from a vendor (out-of-tree).
In cases where drivers beyond those enabled by default in |CL-ATTR| are
needed it may be necessary to:
needed it may be necessary to manually build out-of-tree modules.
Out-of-tree kernel modules can be managed by `Dynamic Kernel Module System
(DKMS) <kernel-modules-dkms>`_ on |CL| for automatic rebuilding upon kernel
updates. Out-of-tree kernel modules can also be manually built and maintained
using the instructions in this document.
.. contents:: :local:
:depth: 1
:backlinks: top
Check if the module is available through |CL|
=============================================
Using an existing module is significantly easier to maintain and retains
signature verification of the |CL| kernel. For more information on |CL|
.. _kernel-modules-availability-begin:
Kernel module availability in |CL|
==================================
Before continuing, check if the kernel module you're looking for is already
available in |CL| or can be requested.
Check if the module is already available
----------------------------------------
|CL| comes with many upstream kernel modules available for use. If you
require a kernel module, be sure to check whether it is already available in
|CL| first.
Using an existing module is significantly easier to maintain and retains
signature verification of the |CL| kernel. For more information on |CL|
security practices, see the :ref:`security` page.
|CL| comes with many upstream kernel modules available for use. If
you require a kernel module, be sure to check whether it is already available in |CL| first.
You can search for kernel module file names, which end with the :file:`.ko`
file extension, using the :command:`swupd search` command. For example:
:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-search` for
more information.
You can search for kernel module file names, which end with the :file:`.ko`
file extension, using the :command:`swupd search` command. For example:
:command:`sudo swupd search ${module_name}.ko`.
See :ref:`swupd-search` for more information.
Request the module be added to |CL|
===================================
-----------------------------------
If the kernel module you need is already open source
(e.g. in the Linux upstream) and likely to be useful to others,
consider submitting a request to add or enable in the |CL| kernel.
If the kernel module you need is already open source (e.g. in the Linux
upstream) and likely to be useful to others, consider submitting a request to
add or enable in the |CL| kernel.
Make enhancement requests to the |CL| distribution `on GitHub`_ .
Build and load an out-of-tree module
====================================
.. _kernel-modules-availability-end:
In some cases you may need an out-of-tree kernel module that is not
available through |CL|.
Build, install, and load an out-of-tree module
==============================================
In some cases you may need an out-of-tree kernel module that is not available
through |CL|.
Prerequisites
-------------
You can build and load out-of-tree kernel modules, however you must:
* disable secure boot
* disable kernel module integrity checking
* build the module against new versions of the Linux kernel
* Disable Secure Boot.
* Disable kernel module integrity checking.
* Have a kernel module package in the form of source code.
* Rebuild the module against new versions of the Linux kernel.
.. note::
Any time the kernel is upgraded on your Clear Linux system, you will
need to rebuild your out-of-tree modules.
This approach works well for individual development or testing.
For a more scalable and customizable approach, consider using the
`mixer tool`_ to provide a custom kernel and updates.
This approach works well for individual development or testing. For a more
scalable and customizable approach, consider using the `mixer tool`_ to
provide a custom kernel and updates.
Build kernel module
-------------------
#. From a |CL| system, ensure you are running the *native* kernel.
Currently only the native kernel is enabled to build and load
out-of-tree modules.
Build and install kernel module
-------------------------------
#. Determine which kernel variant is running on |CL|. In the example below,
the *native* kernel is in use.
.. code-block:: bash
$ uname -r
4.XX.YY-ZZZZ.native
5.XX.YY-ZZZZ.native
Ensure *.native* is in the kernel name
#. Install the `linux-dev` bundle to obtain the kernel headers, which are
required for compiling kernel modules.
#. Install the kernel dev bundle corresponding to the installed kernel. The
kernel dev bundle contains the kernel headers, which are required for
compiling kernel modules.For example:
* `linux-dev` for developing against the native kernel.
* `linux-lts-dev` for developing against the LTS kernel.
.. code-block:: bash
sudo swupd bundle-add linux-dev
#. Follow instructions from the kernel module source code to compile the
kernel module.
kernel module. For example:
.. code-block:: bash
curl -O http://<URL-TO-KERNEL-MODULE-SOURCE>.tar.gz
tar -xvf <KERNEL-MODULE-SOURCE>.tar.gz
cd <KERNEL-MODULE-SOURCE>/
cat README
Load kernel module
@@ -124,28 +161,30 @@ Load kernel module
.. code-block:: bash
sudo insmod ${path_to_module}
sudo insmod </PATH/TO/MODULE.ko>
Optional: Use `modprobe` to specify module options and aliases
--------------------------------------------------------------
Use :command:`modprobe` to load a module and set options.
.. _kernel-modules-autoload-begin:
Because :command:`modprobe` can add or remove more than one module, due to
modules having dependencies, a method of specifying what options are
to be used with individual modules is useful. This can be done with
configuration files under the :file:`/etc/modprobe.d` directory.
Optional: Specify module options and aliases
============================================
Use the :command:`modprobe` command to load a module and set options.
Because :command:`modprobe` may add or remove more than one module due to
modules having dependencies, a method of specifying what options are to be
used with individual modules is useful. This can be done with configuration
files under the :file:`/etc/modprobe.d` directory.
.. code-block:: bash
sudo mkdir /etc/modprobe.d
All files underneath the :file:`/etc/modprobe.d` directory
that end with the :file:`.conf` extension specify module options to use when
loading. This can also be used to create convenient aliases for modules or
they can override the normal loading behavior altogether for those with
special requirements.
All files underneath the :file:`/etc/modprobe.d` directory that end with the
:file:`.conf` extension specify module options to use when loading. This can
also be used to create convenient aliases for modules or they can override the
normal loading behavior altogether for those with special requirements.
You can find more info on module loading in the modprobe.d manual page:
@@ -154,18 +193,18 @@ You can find more info on module loading in the modprobe.d manual page:
man modprobe.d
Optional: Configure kernel modules to load at boot
--------------------------------------------------
==================================================
Use the :file:`/etc/modules-load.d` configuration directory to
specify kernel modules to load automatically at boot.
Use the :file:`/etc/modules-load.d` configuration directory to specify kernel
modules to load automatically at boot.
.. code-block:: bash
sudo mkdir /etc/modules-load.d
All files underneath the :file:`/etc/modules-load.d` directory
that end with the :file:`.conf` extension contain a list of module names
of aliases (one per line) to load at boot.
All files underneath the :file:`/etc/modules-load.d` directory that end with
the :file:`.conf` extension contain a list of module names of aliases (one per
line) to load at boot.
You can find more info on module loading in the modules-load.d manual page:
@@ -173,5 +212,9 @@ You can find more info on module loading in the modules-load.d manual page:
man modules-load.d
.. _kernel-modules-autoload-end:
.. _`on GitHub`: https://github.com/clearlinux/distribution
.. _`mixer tool`: https://clearlinux.org/features/mixer-tool
source/clear-linux/guides/maintenance/mixin.rst
+172 -148
View File
@@ -1,167 +1,191 @@
.. _mixin:
Create and add custom bundles to your upstream Clear Linux system
#################################################################
mixin
#####
|CL-ATTR| offers many curated bundles that you can install on your system to
create your desired capabilities. If the available upstream bundles do not
meet your needs, you can create and add your own custom bundles to your
system using one of two methods. Note: Upstream refers to the official
version of |CL|.
mixin is a tool provided in the |CL-ATTR| that allows users to add custom
content to their client systems and still receive updates from their upstream OS
vendor.
The first method is to use the :ref:`mixer tool<mixer>` to create your own
|CL| image and add your bundles to it. Mixing your own |CL| image can
give you great control and flexibility; however, you must act as an
:abbr:`OSV (Operating System Vendor)` and maintain your releases and
updates because you have forked from upstream.
.. contents::
:local:
:depth: 1
The second method is to use the :command:`mixin` tool, which also
makes use of mixer to create custom bundles that you can add to your
upstream |CL| system. This simpler method provides a “light” forking from
upstream, which means you can continue to get upstream bundles and updates.
If needed, you can easily revert your system back to the upstream version.
Description
***********
This guide shows you how to accomplish the second method by following these
steps:
mixin uses the mixer tool to generate a local update for client systems. With
the mixin tool, a user can add remote RPM repositories or local RPMs and mix
them into their update stream, while continuing to get upstream bundles and
updates. The metadata generated from the mixin tool is merged with the upstream
metadata to provide a single source of update content, which swupd uses to
perform updates.
#. Set up the workspace.
#. Copy your custom RPM package to the workspace.
#. Create a bundle with your custom RPM package.
#. Migrate your |CL| system to your custom mix.
#. Add your custom bundle to your system.
#. Optional: Revert your system back to 100% upstream.
The mixin tool is included in the :command:`mixer` bundle.
Set up the workspace
********************
How to use
**********
#. Install the mixer bundle to enable mixer.
Learn the mixin tool set up and workflow.
.. contents::
:local:
:depth: 1
Prerequisites
=============
Install the :command:`mixer` bundle to add the mixin tool. Refer to
`Install a bundle`_ for more details.
Workflow
========
The following steps show how to create and add a custom bundle with the mixin
tool:
#. Add or create a new repo(s)
mixin pulls packages to build your custom bundle from locations referred to
as repos. There are two default repos for mixin:
* upstream
* local
Additional repos can be added, such as other locations on your local system
or remote repos.
RPMs must be built specifically for |CL| in order for them to work properly.
Refer to :ref:`autospec` for instruction on creating RPMs for |CL|.
#. Create a custom bundle with desired RPMs
Add the desired packages to your new bundle and build the bundle. By default,
the bundle will be named after its parent repo.
The first time you build the bundle, mixer will create a new OS version by
taking your current upstream |CL| version and multiplying it by 1000. For
example, if your upstream version is 27650, your custom version will be
27650000. For each subsequent call to mixin, mixer will increment the version
by 10.
View the `mixin man page`_ for more information on mixin commands.
#. Update system to make custom bundle available
Update your system using swupd to make your custom bundle accessible.
When you first create your mix, you will have to do a one-time migration to
your custom mix as part of the update. After you migrate, the system version
switches over to your last custom version number as noted in the previous
step. As long as you remain on your custom version of |CL| you can continue
to create and add new bundles to your mix with no extra migration step.
#. Install custom bundles
Install your custom bundle using the normal swupd :command:`bundle-add`
command.
View the `swupd man page`_ for more information on swupd commands.
Examples
********
Complete all `Prerequisites`_ before using these examples.
Example 1: Add custom helloclear bundle
=======================================
This example shows the basic steps of adding a custom bundle from a local repo.
#. Check that :command:`helloclear` does not exist on your system:
.. code-block:: bash
helloclear
.. code-block:: console
$ sudo swupd bundle-add mixer
helloclear: command not found
#. Create the workspace.
#. Follow the "Build a new RPM" example from :ref:`autospec` to create a new
`helloclear` RPM.
The resulting RPMs are in `~/clearlinux/packages/helloclear/rpms`.
#. Create a new repo.
#. Create a local repo folder and copy the new `helloclear` RPM files into
the repo:
.. code-block:: bash
mkdir ~/mixin-repo
cp ~/clearlinux/packages/helloclear/rpms/helloclear-v1.0-1.x86_64.rpm ~/mixin-repo
cp ~/clearlinux/packages/helloclear/rpms/helloclear-bin-v1.0-1.x86_64.rpm ~/mixin-repo
#. Create the repo data:
.. code-block:: bash
cd ~/mixin-repo
createrepo_c .
#. Add the repo name:
.. code-block:: bash
sudo mixin repo add mylocalrepo file://$HOME/mixin-repo/
#. Create custom bundle with the new `helloclear` RPM. Add `helloclear` to the
:command:`helloclear-bundle` bundle and build the bundle:
.. code-block:: bash
sudo mixin package add helloclear --bundle helloclear-bundle
sudo mixin build
#. Migrate your |CL| to your custom mix. Check your version before and after the
update to see the switch to your custom mix:
.. code-block:: bash
sudo swupd check-update
sudo swupd update --migrate
sudo swupd check-update
#. Install your custom bundle. Check that the `helloclear-bundle` is now
available and install it to your system:
.. code-block:: bash
sudo swupd bundle-list -a | grep helloclear-bundle
sudo swupd bundle-add helloclear-bundle
#. Test for `helloclear` again to see that it is installed:
.. code-block:: bash
helloclear
#. Revert your system back to upstream (optional). This example reverts back to
upstream version 27650:
.. code-block:: console
$ sudo mkdir -p /usr/share/mix/local-rpms
sudo swupd verify --fix --picky --force -m 27650 -C /usr/share/clear/update-ca/Swupd_Root.pem
sudo swupd clean --all
sudo swupd check-update
Copy your custom RPM package to the workspace
*********************************************
Related topics
**************
.. note::
* :ref:`About mixer <mixer-about>`
* :ref:`mixer`
* :ref:`autospec-about`
* :ref:`bundles-about`
* :ref:`swupd-about`
You cannot simply use RPMs from other Linux distros on |CL|. You must
build RPMs specifically for |CL| in order for them to work properly.
Follow the instructions on how to build RPMs found at the
`Developer tooling framework for Clear Linux`_.
If you have a local RPM you want to add to your mix you can do so by copying
your RPM package to the workspace.
.. code-block:: console
$ sudo cp [RPM] /usr/share/mix/local-rpms
Alternatively, you can add a remote RPM repository by running the following
command.
.. code-block:: console
$ sudo mixin repo add [repo-name] [repo-url]
Create a bundle with your custom RPM package
********************************************
Use the :command:`mixin` command to create a bundle with the RPM
package.
.. code-block:: console
$ sudo mixin package add [package-name] [--bundle bundle-name] [--build]
This command will add package-name to a bundle that is named after its parent
repository. For example, if the RPM was provided locally, it will be added to
the 'local' bundle. If it came from a repo that was added with
:command:`mixin repo add`, it will be added to a bundle named after the
repo-name. If the `--bundle bundle-name` flag is provided, the package will
be added to `bundle-name` instead. The `--build` flag tells :command:`mixin`
to run a `mixer` build after adding the package.
To add more than one RPM to your previously-created bundle, repeat
the :command:`mixin package add` command and change the package name. Do not
add the `--build` flag until all packages have been added. Once done adding
packages, run the following to create your local mix.
.. code-block:: console
$ sudo mixin build
.. note::
* The first time you run the :command:`mixin build` command, mixer
creates a new OS version by taking your current upstream |CL| version
and multiplying it by 1000. For example, if your upstream version is
21530, your custom version will be 21530000. For each subsequent call
to mixin, mixer will increment the version by 10. For example,
21530010, 21530020, etc.
Migrate your Clear Linux system to your custom mix
**************************************************
Before you can use your custom bundle, you must migrate your |CL| system
to your custom mix to make the bundle accessible.
.. code-block:: console
$ sudo swupd update --migrate
After you migrate, the version of your |CL| system switches over to your
last custom version number as noted in the previous section.
You can continue to create new bundles with :command:`mixin`
while you are in your custom version of |CL|. You do not need to migrate
again. However, you must run :command:`swupd update` again to update your
system in order to make those bundles visible.
Add your custom bundle to your system
*************************************
#. Get a listing of your newly-created bundle.
.. code-block:: console
$ sudo swupd bundle-list -a
The listing includes all upstream bundles.
#. Add your bundle.
.. code-block:: console
$ sudo swupd bundle-add [bundle-name]
.. note::
You can also update your system to the latest upstream version using
this command:
.. code-block:: console
$ sudo swupd update
Optional: Revert your system back to 100% upstream
**************************************************
If you want to revert your |CL| system back to the official upstream
version, use this command:
.. code-block:: console
$ sudo swupd verify --fix --force --picky -m [upstream-version-number] -C /usr/share/clear/update-ca/Swupd_Root.pem
After the command completes, all custom RPMs and bundles are unavailable
because :file:`/usr/share/mix` is deleted as part of the reversion process.
.. _Developer tooling framework for Clear Linux:
https://github.com/clearlinux/common
.. _mixin man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixin.1.rst
.. _swupd man page: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst
.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle
source/clear-linux/guides/maintenance/swupd-guide.rst
+1 -1
View File
@@ -49,7 +49,7 @@ Current OS version and update server info:
.. code-block:: console
Installed version: 23330
Version URL: https://download.clearlinux.org/update/
Version URL: https://cdn.download.clearlinux.org/update/
Content URL: https://cdn.download.clearlinux.org/update/
Enable or disable automatic updates
source/clear-linux/guides/maintenance/validate-signatures.rst
+8 -8
View File
@@ -26,11 +26,11 @@ used for illustrative purposes. You may use any image of |CL| you choose.
.. code-block:: console
# Image
curl -O https://download.clearlinux.org/current/clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz
curl -O https://cdn.download.clearlinux.org/current/clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz
# Signature of SHA512 sum of image
curl -O https://download.clearlinux.org/current/clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig
curl -O https://cdn.download.clearlinux.org/current/clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig
# Certificate
curl -O https://download.clearlinux.org/releases/$(curl https://download.clearlinux.org/latest)/clear/ClearLinuxRoot.pem
curl -O https://cdn.download.clearlinux.org/releases/$(curl https://cdn.download.clearlinux.org/latest)/clear/ClearLinuxRoot.pem
#. Generate the SHA256 sum of the |CL| certificate.
@@ -49,7 +49,7 @@ used for illustrative purposes. You may use any image of |CL| you choose.
.. code-block:: console
sha512sum clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz > sha512sum.out
sha512sum clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz > sha512sum.out
#. Ensure the signature of the SHA512 sum of the image was created using the
|CL| certificate. This validates the image is trusted and it has not
@@ -57,7 +57,7 @@ used for illustrative purposes. You may use any image of |CL| you choose.
.. code-block:: console
openssl smime -verify -purpose any -in clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig -inform der -content sha512sum.out -CAfile ClearLinuxRoot.pem
openssl smime -verify -purpose any -in clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig -inform der -content sha512sum.out -CAfile ClearLinuxRoot.pem
.. note::
@@ -83,11 +83,11 @@ these steps manually when performing a ``swupd update``.
.. code-block:: console
# MoM
curl -O https://download.clearlinux.org/update/$(curl https://download.clearlinux.org/latest)/Manifest.MoM
curl -O https://cdn.download.clearlinux.org/update/$(curl https://cdn.download.clearlinux.org/latest)/Manifest.MoM
# Signature of MoM
curl -O https://download.clearlinux.org/update/$(curl https://download.clearlinux.org/latest)/Manifest.MoM.sig
curl -O https://cdn.download.clearlinux.org/update/$(curl https://cdn.download.clearlinux.org/latest)/Manifest.MoM.sig
# Swupd certificate
curl -O https://download.clearlinux.org/releases/$(curl https://download.clearlinux.org/latest)/clear/Swupd_Root.pem
curl -O https://cdn.download.clearlinux.org/releases/$(curl https://cdn.download.clearlinux.org/latest)/clear/Swupd_Root.pem
#. Generate the SHA256 sum of the Swupd certificate.
source/clear-linux/guides/network/dpdk.rst
+2 -2
View File
@@ -262,7 +262,7 @@ machines control the NICs on the host.
.. code-block:: bash
sudo curl -O https://download.clearlinux.org/image/start_qemu.sh
sudo curl -O https://cdn.download.clearlinux.org/image/start_qemu.sh
#. Download a bare-metal image of |CL| and rename it as :file:`clear.img`.
@@ -338,7 +338,7 @@ machines control the NICs on the host.
#. Run the :file:`start_qemu.sh` script.
.. _13330: https://download.clearlinux.org/releases/13330/
.. _13330: https://cdn.download.clearlinux.org/releases/13330/
.. _DPDK project: http://dpdk.org
.. _dpdk.org NICs: http://dpdk.org/doc/nics
.. _pktgen tar package: http://dpdk.org/browse/apps/pktgen-dpdk/refs
source/clear-linux/guides/network/ipxe-install.rst
+2 -2
View File
@@ -103,8 +103,8 @@ To set up |CL| manually, perform the steps below.
sudo mkdir -p $ipxe_root
sudo curl -o /tmp/clear-pxe.tar.xz \
https://download.clearlinux.org/current/clear-$(curl \
https://download.clearlinux.org/latest)-pxe.tar.xz
https://cdn.download.clearlinux.org/current/clear-$(curl \
https://cdn.download.clearlinux.org/latest)-pxe.tar.xz
sudo tar -xJf /tmp/clear-pxe.tar.xz -C $ipxe_root
sudo ln -sf $(ls $ipxe_root | grep 'org.clearlinux.*') $ipxe_root/linux
source/clear-linux/tutorials/telemetry-backend/figures/telemetry-backend-1.png → source/clear-linux/guides/telemetrics/figures/telemetry-backend-1.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 50 KiB

After

Width:  |  Height:  |  Size: 50 KiB

source/clear-linux/tutorials/telemetry-backend/figures/telemetry-backend-2.png → source/clear-linux/guides/telemetrics/figures/telemetry-backend-2.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 61 KiB

After

Width:  |  Height:  |  Size: 61 KiB

source/clear-linux/guides/telemetrics/figures/telemetry-e2e.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

source/clear-linux/guides/telemetrics/telem-guide.rst
+629
View File
@@ -0,0 +1,629 @@
.. _telem-guide:
Telemetrics
###########
Telemetrics in |CL-ATTR| is a client and server solution used to collect data from running |CL| systems to help quickly identify and fix bugs in the OS. Both client and server are customizable, and an API is available on the client side for instrumenting your code for debug and analysis.
.. contents::
:local:
:depth: 1
Description
*************
Telemetry, one of the key features of |CL|, enables developers to observe and proactively address issues in the OS before end users are impacted.
Telemetrics is a combination word made from:
* Telemetry, which is sensing and reporting data.
* Analytics, which is using visualization and statistical inferencing to make sense of the reported data.
|CL| telemetry reports system-level debug/crash information using specialized probes. The probes monitor system tasks such as swupd, kernel oops, machine error checks, and the BIOS error report table for unhandled hardware failures. Telemetry enables real-time issue reporting to allow system developers to quickly focus on an issue and monitor corrective actions.
|CL| telemetry is fully customizable and can be used during software development for debugging purposes. You can use the libtelemetry library in your code to create custom telemetry records. You can also use the telem-record-gen utility in script files for light touch record creation where instrumenting code files doesn't make sense.
Architecture
============
|CL| telemetry has two fundamental components, which are shown in figure 1:
* Client: generates and delivers records to the backend server via the network
* Backend: receives records sent from the client and displays the cumulative content through a specialized web interface.
.. figure:: figures/telemetry-e2e.png
:alt: Figure 1, Telemetry Architecture
Figure 1: :guilabel:`|CL| Telemetry Architecture`
The telemetry client provides the front end of the telemetrics solution and includes the following components:
* telemprobd, a daemon that receives and prepares telemetry records from probes and spools them to disk.
* telempostd, a daemon that manages spooled telemetry records and delivers these records according to configurable settings.
* probes, that collect specific types of data from the operating system.
* libtelemetry, the API that telemetrics probes use to create records.
The telemetry backend provides the server-side component of the telemetrics solution and consists of:
* Nginx web server.
* Two Flask apps:
* Collector, an ingestion web app for records received from client probes.
* TelemetryUI, a web app that exposes different views to visualize the telemetry data.
* PostgreSQL as the underlying database server.
.. note::
The default telemetry backend server is hosted by the Intel |CL| development team and is not viewable outside the Intel firewall. To collect your own records, you must set up your own telemetry backend server.
.. note::
The telemetry functionality adheres to `Intel privacy policies`_ regarding the collection and use of :abbr:`PII (Personally Identifiable Information)` and is open source.
Specifically, no intentionally identifiable information about the user or system owner is collected.
How To Use
**********
From a workflow perspective, the |CL| telemetrics system is straightforward. On the client side, the main decisions after installation and enabling telemetry concern what to do with the record data generated by the probes. You can send the data to the default or a custom backend server, keep the data local to the system, or both. The backend server has a more complex setup, but once it's running, it is simple to use and configure.
This section walks through some of the possible scenarios for configuring the |CL| telemetrics system, and suggests which make sense according to your needs.
Scenarios
=========
#. Enable telemetry
Before probes can generate records, the telemetry client daemons must be enabled. You can configure the client before enabling by creating a custom :file:`telemetrics.conf` file that you place in the :file:`/etc/telemetrics` directory. If you choose to use the default settings, records will be sent to the telemetrics backend server managed by the |CL| development team at Intel.
#. Save record data locally
You can configure the telemetry client to save records locally. This is convenient when you want instant feedback during a development cycle, or to track system issues if you believe there is a machine specific problem. The client can be set to not send records at all, or to both keep the records locally and send to the backend server.
#. Set up a server to collect data
Whether you are managing a network of |CL| systems or you don't want to send records to the default telemetry server, you can set up a backend server to collect your records. The backend server can be installed on any Linux system and will give you the same dashboard as the default server.
#. Instrument your code with the libtelemetry API
The ``telemetrics`` bundle includes the libtelemetry C library, which exposes an API used by the telemprobd and telempostd daemons. You can use these in your applications as well. The API documentation is found in the :file:`telemetry.h` file in `Telemetrics client`_ repository.
Examples
********
.. contents::
:local:
:depth: 1
Enable or Disable Telemetry
===========================
#. Enabling during installation
During the initial installation of |CL|, you are requested to join the stability enhancement program and allow |CL| to collect anonymous reports to improve system stability. If you choose not to join this program, then the telemetry software bundle is not added to your system. Choosing to join will automatically enable telemetry on your system after installation is commplete.
#. Enabling after install
To start telemetry on your system, run the following command:
.. code-block:: bash
sudo telemctl start
This enables and starts the :command:`telemprobd` and :command:`telempostd` daemons. Your system will begin to send telemetry data to the server defined in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not exist, the :command:`telemprobd` and :command:`telempostd` daemons will use the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`.
#. Disabling after install
To disable both of the telemetry daemons, run the following command:
.. code-block:: bash
sudo telemctl stop
#. Opt in to telemetry
To opt-in to the telemetry services, simply enter the opt-in command and start the service:
.. code-block:: bash
sudo telemctl opt-in
This removes the :file:`/etc/telemetrics/opt-out` file, if it exists, and starts the telemetry services.
.. note::
To opt-in but not immediately start telemetry services, you will need to run the command :command:`sudo telemctl stop` after the :command:`opt-in` command is entered. Once you are ready to start the service, enter the command :command:`sudo telemctl start`.
#. Opt out of telemetry
To stop sending telemetrics data from your system, opt out of the telemetry service:
.. code-block:: bash
sudo telemctl opt-out
This creates the file :file:`/etc/telemetrics/opt-out` and stops the telemetry services.
Saving Data Locally
===================
This example requires |CL| to be installed, and telemetry to be enabled on the system.
To change how records are managed, we will be copying the default :file:`/usr/share/defaults/telemetrics/telemetrics.conf` file to :file:`/etc/telemetrics/telemetrics.conf` and editing it. The changes in the :file:`/etc/telemetrics/telemetrics.conf` file will override the defaults in the :file:`/usr/share/defaults/telemetrics/telemetrics.conf` file. You may need ``root`` permissions to create and edit files in :file:`/etc`. For each example, and for any time you make changes to the configuration file, you will need to restart the client daemons to pick up the changes:
.. code-block:: bash
sudo telemctl restart
The :command:`telemctl journal` command gives you access to features and options of the telemetry journal to assist with system analytics and debug. :command:`telemctl journal` has a number of options to help filter records. Use :command:`-h` or :command:`--help` to view usage options.
#. Keep local copy and send records to backend server
To keep a local copy of the telemetry record and also send it on to the backend server, we will need to change the :guilabel:`record_retention_enabled` configuration key value to :guilabel:`true`.
#. Keep all records -- don't send to backend server
To keep records on the system without sending them to a backend server, set the :guilabel:`record_server_delivery_enabled` key value to :guilabel:`false`. Note that you will also need to ensure the :guilabel:`record_retention_enabled` configuration key value is set to :guilabel:`true` or the system will not keep local copies.
#. Keep and send records to custom server
This assumes you have set up a custom server according to the next example.
The server is identified by the :guilabel:`server` setting, and by default records will be sent to the |CL| server :guilabel:`server=https://clr.telemetry.intel.com/v2/collector`. To change this, you can use an IP address or fully qualified domain name.
Set up a backend server to collect telemetry records
====================================================
For this example, start with a clean installation of |CL| on a new system using the :ref:`bare-metal-install` getting started guide and:
#. Join the :guilabel:`Stability Enhancement Program` to install and
enable the telemetrics components.
#. Select the manual installation method with the following settings:
* Set the hostname to :guilabel:`clr-telem-server`,
* Create an administrative user named :guilabel:`clear` and add this user
to sudoers
#. Log in with your administrative user, from your :file:`$HOME` directory, run :command:`git` to clone the :guilabel:`telemetrics-backend` repository into the :file:`$HOME/telemetrics-backend` directory:
.. code-block:: console
git clone https://github.com/clearlinux/telemetrics-backend
.. note::
You may need to set up the :envvar:`https_proxy` environment variable if you have issues reaching github.com.
#. Change your current working directory to :file:`telemetrics-backend/scripts`.
#. We will install the telemetrics backend with the :file:`deploy.sh` script file. We will set the following options and leave the remainder as default:
* *-a install* to perform an install
* *-d clr* to install to a |CL| distro
* *-H localhost* to set the domain to localhost
.. caution::
The :file:`deploy.sh` shell script has minimal error checking and makes
several changes to your system. Be sure that the options you define on the cmdline are correct before proceeding.
#. Run the shell script from the :file:`$HOME/telemetrics-backend/scripts` directory:
.. code-block:: console
./deploy.sh -H localhost -a install -d clr
The script will start and list all the defined options and prompt you for the :guilabel:`PostgreSQL` database password
.. code-block:: console
Options:
host: localhost
distro: clr
action: install
repo: https://github.com/clearlinux/telemetrics-backend
source: master
type: git
DB password: (default: postgres):
#. For the :guilabel:`DB password:`, press the :kbd:`Enter` key to accept the default password `postgres`.
.. note::
The :file:`deploy.sh` script uses :command:`sudo` to run commands and you may be prompted to enter your user password at any time while the script is executing. If this occurs, enter your user password to execute the :command:`sudo` command.
#. Once all the server components have been installed you are prompted to enter the :guilabel:`PostgreSQL` database password to change it as illustrated below:
.. code-block:: console
Enter password for 'postgres' user:
New password:
Retype new password:
passwd: password updated successfully
Enter `postgres` for the current value of the password and then enter a new password, retype it to verify the new password and the :guilabel:`PostgreSQL` database password will be updated.
#. Once the installation is complete you can use your web browser to view the new server by opening the browser on the system and typing in ``localhost`` in the address bar. You should see a web page similar to the one shown in figure 1:
.. figure:: figures/telemetry-backend-1.png
:alt: Telemetry UI
Figure 1: :guilabel:`Telemetry UI`
Create records with telem-record-gen
====================================
The telemetrics bundle provides a record generator tool called ``telem-record-gen``. This tool can be used to create records from shell scripts or the command line when writing a probe in C is not desirable. Records are sent to the backend server, and can also be echoed to stdout.
There are three ways to supply the payload to the record.
#. On the command line, use the :command:`-p <string>` option:
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -p 'payload goes here'
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023189
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2236710e4fc11e4a646ce956c7802788
payload goes here
#. Specify a file that contains the payload with the option :command:`-P path/to/file`.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -P ./payload_file.txt
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: d73d6040afd7693cccdfece479df9795
payload read from file
#. If the :command:`-p` or :command:`-P` options are absent, the tool reads from stdin so you can use it in a :file:`heredoc` in scripts.
.. code-block:: bash
#telem-record-gen -c a/b/c -n -o << HEOF
payload read from stdin
HEOF
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2f070e8e71679f2b1f28794e3a6c42ee
payload read from stdin
Instrument your code with the libtelemetry API
==============================================
Prerequisites
-------------
Confirm that the telemetrics header file is located on the system at :file:`usr/include/telemetry.h` The `latest version`_ of the file can also be found on github for reference, but installing the `telemetry` bundle will install the header file that matches your |CL| version.
#. Includes and variables
You will need to include the following headers in your code to use the API:
::
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <telemetry.h>
Use the following code to create the variables we need to hold the data for the record we will be creating:
::
uint32_t severity = 1;
uint32_t payload_version = 1;
char classification[30] = "org.clearlinux/hello/world";
struct telem_ref *tm_handle = NULL;
char *payload;
int ret = 0;
Severity:
| Type: uint32_t
| Value: Severity field value. Accepted values are in the range 1-4, with 1 being the lowest severity, and 4 being the highest severity. Values provided outside of this range are clamped to 1 or 4. [low, med, high, crit]
Payload_version:
| Type: uint32_t
| Value: Payload format version. The only supported value right now is 1, which indicates that the payload is a freely-formatted (unstructured) string. Values greater than 1 are reserved for future use.
Classification:
| Type: char array
| Value: It should have the form, DOMAIN/PROBENAME/REST: DOMAIN is the reverse domain to use as a namespace for the probe (e.g. org.clearlinux); PROBENAME is the name of the probe; and REST is an arbitrary value that the probe should use to classify the record. The maximum length for the classification string is 122 bytes. Each sub-category may be no longer than 40 bytes long. Two / delimiters are required.
Tm_handle:
| Type: Telem_ref struct pointer
| Value: Struct pointer declared by the caller, The struct is initialized if the function returns success.
Payload:
| Type: char pointer
| Value: The payload to set
#. For this example, we'll set the payload to “hello” by using ``asprintf()``
::
if (asprintf(&payload, "hello\n") < 0) {
exit(EXIT_FAILURE);
}
The functions ``asprintf()`` and ``vasprintf()`` are analogs of ``sprintf(3)`` and ``vsprintf(3)``, except that they allocate a string large enough to hold the output including the terminating null byte ('\0'), and return a pointer to it via the first argument. This pointer should be passed to ``free(3)`` to release the allocated storage when it is no longer needed.
#. Create the new telemetry record
The function ``tm_create_record()`` initializes a telemetry record and sets the severity and classification of that record, as well as the payload version number. The memory needed to store the telemetry record is allocated and should be freed with ``tm_free_record()`` when no longer needed.
::
if ((ret = tm_create_record(&tm_handle, severity, classification, payload_version)) < 0) {
printf("Failed to create record: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
#. Set the payload field of a telemetrics record
The function ``tm_set_payload()`` attaches the provided telemetry record data to the telemetry record. The current maximum payload size is 8192b.
::
if ((ret = tm_set_payload(tm_handle, payload)) < 0) {
printf("Failed to set record payload: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
free(payload);
The ``free()`` function frees the memory space pointed to by ptr, which must have been returned by a previous call to ``malloc()``, ``calloc()``, or ``realloc()``. Otherwise, or if ``free(ptr)`` has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed.
#. Send a record to the telemetrics daemon
The function ``tm_send_record()`` delivers the record to the local ``telemprobd(1)`` service. Since the telemetry record was allocated by the program it should be freed with ``tm_free_record()`` when it is no longer needed.
::
if ((ret = tm_send_record(tm_handle)) < 0) {
printf("Failed to send record to daemon: %s\n", strerror(-ret));
ret = 1;
goto fail;
} else {
printf("Successfully sent record to daemon.\n");
ret = 0;
}
fail:
tm_free_record(tm_handle);
tm_handle = NULL;
return ret;
#. Full sample application with compiling flags
Create a new file test.c add the following code.
::
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <telemetry.h>
int main(int argc, char **argv)
{
uint32_t severity = 1;
uint32_t payload_version = 1;
char classification[30] = "org.clearlinux/hello/world";
struct telem_ref *tm_handle = NULL;
char *payload;
int ret = 0;
if (asprintf(&payload, "hello\n") < 0) {
exit(EXIT_FAILURE);
}
if ((ret = tm_create_record(&tm_handle, severity, classification, payload_version)) < 0) {
printf("Failed to create record: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
if ((ret = tm_set_payload(tm_handle, payload)) < 0) {
printf("Failed to set record payload: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
free(payload);
if ((ret = tm_send_record(tm_handle)) < 0) {
printf("Failed to send record to daemon: %s\n", strerror(-ret));
ret = 1;
goto fail;
} else {
printf("Successfully sent record to daemon.\n");
ret = 0;
}
fail:
tm_free_record(tm_handle);
tm_handle = NULL;
return ret;
}
Compile with the gcc compiler, using this command:
.. code-block:: bash
gcc test.c -ltelemetry -o test_telem
Test to ensure the program is working:
.. code-block:: bash
./test_telem
Successfully sent record to daemon.
.. note::
A full example of the `heartbeat probe`_ in C is documented in the source code.
Reference
*********
.. contents::
:local:
:depth: 1
The Telemetry API
=================
Installing the ``telemetrics`` bundle includes the libtelemetry C library, which exposes an API used by the telemprobd and telempostd daemons. You can use these in your applications as well. The API documentation is found in the :file:`telemetry.h` file in `Telemetrics client`_ repository.
Client Configuration
====================
The telemetry client will look for the configuration file located at :file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the file does not exist, the client will use the default configuration located at :file:`/usr/share/defaults telemetrics/telemetrics.conf`. To modify or customize the configuration, copy the file from :file:`/usr/share/defaults/telemetrics` to :file:`/etc/telemetrics` and edit it.
Configuration Options
---------------------
The client uses the following configuration options from the config file:
* **server**: This specifies the web server to which telempostd sends the telemetry records.
* **socket_path**: This specifies the path of the unix domain socket that the telemprobd listens on for connections from the probes.
* **spool_dir**: This configuration option is related to spooling. If the daemon is not able to send the telemetry records to the backend server due to reasons such as the network availability, then it stores the records in a spool directory. This option specifies that path of the spool directory. This directory should be owned by the same user as the daemon.
* **record_expiry**: This is the time in minutes after which the records in the spool directory are deleted by the daemon.
* **spool_process_time**: This specifies the time interval in seconds that the daemon waits for before checking the spool directory for records. The daemon picks up the records in the order of modification date and tries to send the record to the server. It sends a maximum of 10 records at a time. If it was able to send a record successfully, it deletes the record from the spool. If the daemon finds a record older than the "record_expiry" time, then it deletes that record. The daemon looks at a maximum of 20 records in a single spool run loop.
* **rate_limit_enabled**: This determines whether rate-limiting is enabled or disabled. When enabled, there is a threshold on both records sent within a window of time, and record bytes sent within a window a time.
* **record_burst_limit**: This is the maximum amount of records allowed to be passed by the daemon within the record_window_length of time. If set to -1, the rate-limiting for record bursts is disabled.
* **record_window_length**: The time in minutes (0-59) that establishes the window length for the record_burst_limit. EX: if record_burst_window=1000 and record_window_length=15, then no more than 1000 records can be passed within any given fifteen minute window.
* **byte_burst_limit**: This is the maximum amount of bytes that can be
passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled.
* **byte_window_length**: This is the time, in minutes (0-59), that establishes the window length for the byte_burst_limit.
* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting threshold has been reached. Currently the options are 'drop' or 'spool', with spool being the default. If spool is chosen, records will be spooled and sent at a later time.
* **record_retention_enabled**: When this key is enabled (true) the daemon saves a copy of the payload on disk from all valid records. To avoid the excessive use of disk space only the latest 100 records are kept. The default value for this configuration key is false.
* **record_server_delivery_enabled**: This key controls the delivery of records to server; when enabled (default value), the record will be posted to the address in the configuration file. If this configuration key is disabled (false), records will not be spooled or posted to backend. This configuration key can be used in combination with record_retention_enabled to keep copies of telemetry records locally only.
.. note::
Configuration options may change as the telemetry client evolves.
Please use the comments in the file itself as the most accurate
reference for configuration.
Client Run-time Options
=======================
The |CL| telemetry client provides an admin tool called :guilabel:`telemctl` for managing the telemetry services and probes. The tool is located in :file:`/usr/bin`. Running it with no argument results in the following:
.. code-block:: bash
sudo telemctl
.. code-block:: console
/usr/bin/telemctl - Control actions for telemetry services
stop Stops all running telemetry services
start Starts all telemetry services
restart Restarts all telemetry services
is-active Checks if telemprobd and telempostd are active
opt-in Opts in to telemetry, and starts telemetry services
opt-out Opts out of telemetry, and stops telemetry services
journal Prints telemetry journal contents. Use -h argument for more
options
start/stop/restart
------------------
The commands to start, stop and restart the telemetry services manage all required services and probes on the system. There is no need to separately start/stop/restart the two client daemons **telemprobd** and **telempostd**. The **restart** command option will call **telemctl stop** followed by **telemctl start** .
is-active
---------
The `is-active` option reports whether the two client daemons are active. This is useful to verify that the **opt-in** and **opt-out** options have taken effect, or to ensure that telemetry is functioning on the system. Note that both daemons are verified.
.. code-block:: bash
sudo telemctl is-active
.. code-block:: console
telemprobd : active
telempostd : active
.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/
.. _latest version: https://github.com/clearlinux/telemetrics-client/tree/master/src
.. _heartbeat probe: https://github.com/clearlinux/telemetrics-client/tree/master/src/probes/hello.c
.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
source/clear-linux/guides/telemetrics/telemctl.rst
-65
View File
@@ -1,65 +0,0 @@
.. _telemctl:
telemctl options
################
The |CL-ATTR| telemetry client provides an admin tool called ``telemctl``
that is used for managing the telemetry services and probes. The tool is
located in :file:`/usr/bin`. Running it with no argument results in the
following:
.. code-block:: bash
sudo telemctl
.. code-block:: console
/usr/bin/telemctl - Control actions for telemetry services
stop Stops all running telemetry services
start Starts all telemetry services
restart Restarts all telemetry services
is-active Checks if telemprobd and telempostd are active
opt-in Opts in to telemetry, and starts telemetry services
opt-out Opts out of telemetry, and stops telemetry services
journal Prints telemetry journal contents. Use -h argument for more
options
telemctl commands:
******************
start/stop/restart
==================
The commands to start, stop and restart the telemetry services manage all
required services and probes on the system. There is no need to separately
start/stop/restart the two client daemons **telemprobd** and **telempostd**.
The **restart** command option will call **telemctl stop** followed by
**telemctl start** .
is-active
=========
The `is-active` option reports whether the two client daemons are active.
This is useful to verify that the **opt-in** and **opt-out** options have
taken effect, or to ensure that telemetry is functioning on the system.
Note that both daemons are verified.
.. code-block:: bash
sudo telemctl is-active
.. code-block:: console
telemprobd : active
telempostd : active
.. include:: ./telemetry-enable.rst
:start-after: incl-opt-in-out-telemetry:
:end-before: incl-opt-in-out-telemetry-end:
Next steps
==========
Learn to read records:
* :ref:`telemetry-journal`
source/clear-linux/guides/telemetrics/telemetrics.rst
-19
View File
@@ -1,19 +0,0 @@
.. _telemetrics:
Telemetry
#########
The |CL-ATTR| telemetrics solution collects data from running |CL| systems
and helps to quickly identify and fix bugs in the OS. These guides will walk
you through setup, configuration, and customization of the telemetry client. The data collected from the client system is analyzed and presented by the
telemetry backend solution. For more details, learn how to
:ref:`telemetry-backend`.
.. toctree::
:maxdepth: 1
telemetry-enable
telemetry-config
telemctl
telemetry-journal
telemetry-api
source/clear-linux/guides/telemetrics/telemetry-api.rst
-170
View File
@@ -1,170 +0,0 @@
.. _telemetry-api:
Telemetry API
#############
Installing the ``telemetrics`` bundle includes the libtelemetry C library,
which exposes an API used by the telemprobd and telempostd daemons. You
can use these in your applications as well. The API documentation is found
in the :file:`telemetry.h` file in `Telemetrics client`_ repository.
Creating records with telem-record-gen
**************************************
The telemetrics bundle also provides a record generator tool called
``telem-record-gen``. This tool can be used to create records from shell
scripts, etc., when writing a probe in C is not desirable. Records are sent
to the backend server, and can also be echoed to stdout.
telem-record-gen usage
======================
.. code-block:: bash
telem-record-gen [OPTIONS] - create and send a custom telemetry record
.. code-block:: console
Help Options:
-h, --help Show help options
Application Options:
-V, --version Print the program version
-s, --severity Severity level (1-4) - (default 1)
-c, --class Classification level_1/level_2/level_3 (required)
-p, --payload Record body (max size = 8k) (required)
-P, --payload-file File to read payload from
-R, --record-version Version number for format of payload (default 1)
-e, --event-id Event id to use in the record
-o, --echo Echo record to stdout
-n, --no-post Do not post record just print
The :command:`-c` and :command:`-p` options are required; defaults are
supplied for most other options. The maximum payload size is 8k
(8192 bytes). Excess is ignored, regardless of source (file/commandline/
stdin). An empty payload is allowed, but even an empty payload must be
specified in one of the three ways shown below.
telem-record-gen examples
=========================
There are three ways to supply the payload to the record.
#. On the command line, use the :command:`-p <string>` option:
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -p 'payload goes here'
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023189
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2236710e4fc11e4a646ce956c7802788
payload goes here
#. Specify a file that contains the payload with the option
:command:'-P path/to/file'.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -P ./payload_file.txt
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: d73d6040afd7693cccdfece479df9795
payload read from file
#. If the :command:`-p` or :command:`-P` options are absent, the tool reads
from stdin so you can use it in a :file:`heredoc` in scripts.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o << HEOF
payload read from stdin
HEOF
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2f070e8e71679f2b1f28794e3a6c42ee
payload read from stdin
.. note::
Although only the classification and payload are specified, the tool supplies values for the remaining values.
Telemetry records and the REST API
==================================
If you have not configured the telemetry client to keep records locally, you
can view them using the Web UI of the server, or you can query them from the
server using the REST API provided by |CL| telemetrics. The API is
available at :file:`<server>/api/records`, and when queried, returns a JSON
response that contains a list of records. There are several parameters for
filtering queries, similar to the filters available through the telemetryui Records view.
* classification: The classification of the record
* severity: The severity of the record. Restricted to integer value
* machine_id: The id of the machine where this record was generated on
* build: The build on which the record was generated. Restricted to 256
characters.
* created_in_days: causes the query to return records created after the last
given days
* created_in_sec: returns the records created after the last given seconds
* limit: The maximum number of records to be returned.
Next Steps
==========
* :ref:`telemetry-backend`
* `Telemetrics client`_
Related topics
==============
* :ref:`telemetry-about`
.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/
source/clear-linux/guides/telemetrics/telemetry-config.rst
-101
View File
@@ -1,101 +0,0 @@
.. _telemetry-config:
Telemetry client configuration
##############################
The telemetry client will look for the configuration file located at
:file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the
file does not exist, the client will use the default configuration located
at :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. To modify or
customize the configuration, copy the file from
:file:`/usr/share/defaults/telemetrics` to
:file:`/etc/telemetrics` and edit it.
Configuration Options
*********************
The client uses the following configuration options from the config file:
* **server**: This specifies the web server to which telempostd sends the
telemetry records.
* **socket_path**: This specifies the path of the unix domain socket that
the telemprobd listens on for connections from the probes.
* **spool_dir**: This configuration option is related to spooling. If the
daemon is not able to send the telemetry records to the backend server due
to reasons such as the network availability, then it stores the records in
a spool directory. This option specifies that path of the spool directory.
This directory should be owned by the same user as the daemon.
- mkdir -p /var/spool/telemetry
- chown -R telemetry:telemetry /var/spool/telemetry
- systemctl restart telemprobd.service
* **record_expiry**: This is the time in minutes after which the records in
the spool directory are deleted by the daemon.
* **spool_process_time**: This specifies the time interval in seconds that
the daemon waits for before checking the spool directory for records. The
daemon picks up the records in the order of modification date and tries to
send the record to the server. It sends a maximum of 10 records at a time.
If it was able to send a record successfully, it deletes the record from
the spool. If the daemon finds a record older than the "record_expiry"
time, then it deletes that record. The daemon looks at a maximum of 20
records in a single spool run loop.
* **rate_limit_enabled**: This determines whether rate-limiting is enabled or
disabled. When enabled, there is a threshold on both records sent within a
window of time, and record bytes sent within a window a time.
* **record_burst_limit**: This is the maximum amount of records allowed to be
passed by the daemon within the record_window_length of time. If set to
-1, the rate-limiting for record bursts is disabled.
* **record_window_length**: The time in minutes (0-59) that
establishes the window length for the record_burst_limit. EX: if
record_burst_window=1000 and record_window_length=15, then no more than
1000 records can be passed within any given fifteen minute window.
* **byte_burst_limit**: This is the maximum amount of bytes that can be
passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled.
* **byte_window_length**: This is the time, in minutes (0-59), that
establishes the window length for the byte_burst_limit.
* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting
threshold has been reached. Currently the options are 'drop' or 'spool',
with spool being the default. If spool is chosen, records will be spooled
and sent at a later time.
* **record_retention_enabled**: When this key is enabled (true) the daemon
saves a copy of the payload on disk from all valid records. To avoid the
excessive use of disk space only the latest 100 records are kept. The
default value for this configuration key is false.
* **record_server_delivery_enabled**: This key controls the delivery of
records to server; when enabled (default value), the record will be posted
to the address in the configuration file. If this configuration key is
disabled (false), records will not be spooled or posted to backend. This
configuration key can be used in combination with record_retention_enabled
to keep copies of telemetry records locally only.
.. note::
Configuration options may change as the telemetry client evolves.
Please use the comments in the file itself as the most accurate
reference for configuration.
Setting a static machine id
===========================
The machine id reported by the telemetry client is rotated every 3 days for
privacy reasons. If you wish to have a static machine id for testing
purposes, you can opt in by creating a static machine id file named
"opt-in-static-machine-id" under the directory :file:`/etc/telemetrics/`.
Where "unique machine ID" is your desired static machine ID.
.. code-block:: bash
sudo mkdir -p /etc/telemetrics
.. code-block:: bash
sudo echo "unique machine ID" > /etc/telemetrics/opt-in-static-machine-id
.. note::
The machine id mentioned here is not the same as the system hostname. Learn how to :ref:`hostname`:
Next steps
==========
* :ref:`telemctl`
source/clear-linux/guides/telemetrics/telemetry-enable.rst
-114
View File
@@ -1,114 +0,0 @@
.. _telemetry-enable:
Enable telemetry
################
Telemetry enables developers to observe and proactively address issues on
|CL-ATTR| before end users are impacted. The telemetry functionality
is maintained in the ``telemetrics`` software bundle.
.. note::
The telemetry functionality adheres to `Intel privacy policies`_
regarding the collection and use of :abbr:`PII (Personally Identifiable
Information)` and is open source. Specifically, no intentionally
identifiable information about the user or system owner is collected.
End users may enable or disable the telemetry component of |CL| or even
redirect where the records go if they wish to collect records for themselves.
Install the telemetry software bundle
*************************************
During the initial installation of |CL|, you are requested to join the
stability enhancement program and allow |CL| to collect anonymous reports
to improve system stability. If you choose not to join this program, then the
telemetry software bundle is not added to your system.
To install the telemetry bundle, enter the following command as either the
root user or with :command:`sudo` privileges:
.. code-block:: bash
sudo swupd bundle-add telemetrics
This adds the telemetrics-client to your system, and you will automatically
opt-in for the service.
Enable telemetry
================
To start telemetry on your system, run the following command:
.. code-block:: bash
sudo telemctl start
This enables and starts the :command:`telemprobd` and :command:`telempostd`
daemons. Your system will begin to send telemetry data to the server defined
in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not
exist, the :command:`telemprobd` and :command:`telempostd` daemons will use
the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`.
Disable telemetry
=================
To disable both of the telemetry daemons, run the following command:
.. code-block:: bash
sudo telemctl stop
.. _incl-opt-in-out-telemetry:
Opt-in to telemetry
===================
To opt-in to the telemetry services, simply enter the opt-in
command and start the service:
.. code-block:: bash
sudo telemctl opt-in
This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists,
and starts the telemetry services.
.. note::
To opt-in but not immediately start telemetry services, you will need to
run the command :command:`sudo telemctl stop` after the :command:`opt-in`
command is entered. Once you are ready to start the service, enter the
command :command:`sudo telemctl start`.
Opt-out of telemetry
====================
To stop sending telemetrics data from your system, opt out of the
telemetry service:
.. code-block:: bash
sudo telemctl opt-out
This creates the file :file:`/etc/telemetrics/opt-out` and stops the
telemetry services.
.. _incl-opt-in-out-telemetry-end:
Remove the telemetry software bundle
====================================
To completely remove telemetrics from your system, use the :command:`swupd`
command to remove the telemetry software bundle:
.. code-block:: bash
sudo swupd bundle-remove telemetrics
Next steps
==========
* :ref:`telemetry-config`
.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
source/clear-linux/guides/telemetrics/telemetry-journal.rst
-108
View File
@@ -1,108 +0,0 @@
.. _telemetry-journal:
telemctl journal
################
The telemctl ``journal`` command gives you access to features and options of
the telemetry journal to assist with system analytics and debug. The
:command:`telemctl journal` has a number of options to help filter
records. Use :command:`-h` or :command:`--help` to view usage options.
.. code-block:: bash
sudo telemctl journal -h
::
-r, --record_id Print record with specific record_id
-e, --event_id Print records with specific event_id
-c, --classification Print records with specific classification
-b, --boot_id Print records with specific boot_id
-i, --include_record Include record content
-V, --verbose Verbose output
-h, --help Display this help message
Journal Output
**************
To see the listing of records in the journal, run the command:
.. code-block:: bash
sudo telemctl journal -V
This will produce output like the following:
.. list-table:: **Table 1. Journal Output**
:widths: 10 30 20 20 20
:header-rows: 1
* - Classification
- Time stamp
- Record ID
- Event ID
- Boot ID
* - org.clearlinux/heartbeat/ping
- Fri 2018-09-21 00:00:57 UTC
- 269e8e4026e6aa440c4d2ed71e38efcd
- b3a51b5e62a008ed0b56d1740be67d48
- 853a75aa-da3b-4356-a085-079abab3ffe1
* - org.clearlinux/hello/world
- Fri 2018-09-21 17:53:21 UTC
- b06c8d31adf5ccc7d5d3f8959d8d3e72
- 57c64c79a9b911d68f4dab10a00267d7
- 853a75aa-da3b-4356-a085-079abab3ffe1
* - org.clearlinux/crash/clr
- Fri 2018-09-21 17:57:59 UTC
- b62cd4278672ae3331cf121bc7a8e1c6
- b6adb5751382c48eebb7ee007fe1790a
- 853a75aa-da3b-4356-a085-079abab3ffe1
Each line gives information about a distinct record. The :command:`-V` or
:command:`--verbose` option adds the header to identify the Classification,
Time Stamp, Record ID, Event ID and Boot ID for each record. The journal
feature can filter records according to the Classification, Record ID, Event
ID and Boot ID by using the :command:`-c`,:command:`-r`, :command:`-e` and
:command:`-b` options accordingly.
Payload Information
********************
From the previous output, you may want to get more information about the
record with the "org.clearlinux/crash/clr" classification to help debug a
crash. You can use the :command:`-c` and :command:`-i` options to see the payload of the record, like this:
.. code-block:: bash
sudo telemctl journal -c org.clearlinux/crash/clr -i
.. code-block:: console
org.clearlinux/crash/clr Tue 2018-09-25 18:43:50 UTC 07ae583edbd13829965d67e9ba97d70c 69c600470769c841649266178375d67e d32c13d1-fda0-49c6-8431-e6c5b29cbefa
Process: /usr/bin/bash
PID: 685
Signal: 11
Backtrace (TID 685):
#0 kill() - [libc.so.6]
#1 bash_tilde_expand() - [/usr/bin/bash]
#2 maybe_execute_file() - [/usr/bin/bash]
#3 main() - [/usr/bin/bash]
#4 __libc_start_main() - [libc.so.6]
#5 _start() - [/usr/bin/bash]
If you have records of multiple crashes, you can use the :command:'-r'
option to specify the record more precisely, rather than going by
classification. You can also specify a classification of record and use the
:command:'-i' option to see the payload of each record with that
classification.
Next steps
==========
Adding telemetry to your applications:
* :ref:`telemetry-api`
source/clear-linux/reference/bundles/bundles.html.txt
+4105 -196
View File
File diff suppressed because it is too large Load Diff
source/clear-linux/reference/bundles/openssh-server.rst
+169
View File
@@ -0,0 +1,169 @@
.. _openssh-server:
openssh-server
##############
The **openssh-server** bundle provides the OpenSSH\* package needed to enable
an SSH service in |CL-ATTR|. Remote users require an SSH service to be able
to use an encrypted login shell.
|CL| enables the `sshd.socket` unit, which will listen on port 22 by default
and start the OpenSSH service as required. The first time OpenSSH starts, it
generates the server SSH keys needed for the service.
Prerequisites
*************
Assure the bundle :file:`openssh-server` is installed.
To check it it's on your host, enter:
.. code-block:: bash
sudo swupd bundle-list
To add it, enter:
.. code-block:: bash
sudo swupd bundle-add openssh-server
Change default port
*******************
Perform the following steps to change the default listening port for the
OpenSSH service:
#. Open the sshd.socket file:
.. code-block:: bash
sudo systemctl edit sshd.socket
#. Add the `[Socket]` section and `ListenStream` option to the sshd.socket
file as shown below. The first `ListenStream` entry removes the |CL|
default listen port value. The second `ListenStream` entry sets the new
default listen port value. In this example, we set the new default port
to 4200:
.. code-block:: console
[Socket]
ListenStream=
ListenStream=4200
Make sure to include a new line after the last line of text in the sshd.socket file.
#. Verify your changes:
.. code-block:: bash
cat /etc/systemd/system/sshd.socket.d/override.conf
You should see the following output:
.. code-block:: console
[Socket]
ListenStream=
ListenStream=4200
#. Reload the systemd daemon configurations:
.. code-block:: bash
sudo systemctl daemon-reload
#. Restart the sshd.socket unit:
.. code-block:: bash
sudo systemctl restart sshd.socket
#. Confirm the the sshd.socket unit is listening on your new port:
.. code-block:: bash
sudo systemctl status sshd.socket
.. note::
Output should show :guilabel:`Active:` as `active(listening)`.
Enable SFTP
***********
|CL| *disables* the :abbr:`SFTP (SSH File Transfer Protocol)` subsystem by
default due to security considerations. To enable the SFTP subsystem, perform
the following configuration of the :abbr:`SSHD (SSH Daemon)` service file:
#. Create a systemd drop-in directory for the SSHD service:
.. code-block:: bash
sudo mkdir -p /etc/systemd/system/sshd@.service.d
#. Create the following file:
:file:`/etc/systemd/system/sshd@.service.d/sftp.conf`
#. Add the OPTIONS environment variable to the sftp.conf file.
.. code-block:: console
[Service]
Environment="OPTIONS=-o Subsystem=\"sftp /usr/libexec/sftp-server\""
#. Reload systemd configuration:
.. code-block:: bash
sudo systemctl daemon-reload
Congratulations! The SFTP subsystem is enabled.
Enable root login
*****************
To enable root login via SSH, perform the following steps:
#. Create a *ssh* directory in :file:`/etc`, if it does not already exist.
.. code-block:: bash
mkdir /etc/ssh
#. Create the following file, if it does not already exist:
:file:`/etc/ssh/sshd_config`
#. Set the configuration variable in /etc/ssh/sshd_config
.. code-block:: console
PermitRootLogin yes
Enable X11-forwarding
*********************
X11 forwarding allows you to securely run graphical applications
(i.e., X clients) over the ssh conection. This will alow for remote gui apps
without the need for full VNC/remotedesktop. To enable X11-forwarding via
SSH, perform the following steps:
#. Create a *ssh* directory in :file:`/etc`, if it does not already exist.
.. code-block:: bash
mkdir /etc/ssh
#. Create the following file, if it does not already exist:
:file:`/etc/ssh/sshd_config`
#. Set the configuration variables.
.. code-block:: bash
AllowTcpForwarding yes
X11UseLocalhost yes
X11DisplayOffset 10
X11Forwarding yes
source/clear-linux/reference/image-types.rst
+1 -1
View File
@@ -76,6 +76,6 @@ Table 2 lists the currently available images that are platform specific.
* - vmware.vmdk
- Virtual Machine Disk for VMware\* platforms inclduing Player, Workstation, and ESXi.
.. _images: https://download.clearlinux.org/image
.. _images: https://cdn.download.clearlinux.org/image
.. _`optimized kernel`: https://clearlinux.org/documentation/clear-linux/reference/compatible-kernels
source/clear-linux/reference/reference.rst
+2
View File
@@ -12,8 +12,10 @@ features.
compatible-hardware
bundle-commands
bundles/bundles
bundles/openssh-server
how-to-clear-overview
collaboration/collaboration
compatible-kernels
system-requirements
image-types
source/clear-linux/tutorials/dlrs.rst
+102 -36
View File
@@ -4,38 +4,45 @@ Deep Learning Reference Stack
#############################
This tutorial shows you how to run benchmarking workloads in |CL-ATTR| using
TensorFlow\* and Kubeflow with the Deep Learning Reference Stack.
TensorFlow\* or PyTorch\* with the Deep Learning Reference Stack. We also
cover using Kubeflow for multi-node benchmarking.
The Deep Learning Reference Stack is available in two versions.
The first is `Eigen`_, which includes `TensorFlow`_ optimized for Intel®
architecture. The second is `Intel MKL-DNN`_, which includes the TensorFlow
framework optimized using Intel® Math Kernel Library for Deep Neural
Networks (Intel® MKL-DNN) primitives.
.. contents:: :local:
.. contents::
:local:
:depth: 1
The Deep Learning Reference Stack is available in four versions:
* `Eigen`_, which includes `TensorFlow`_ optimized for Intel® architecture.
* `Intel MKL-DNN`_, which includes the TensorFlow framework optimized using
Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) primitives.
* `PyTorch with OpenBLAS`_, which includes PyTorch with OpenBlas.
* `PyTorch with Intel MKL-DNN`_, which includes PyTorch optimized using
Intel® Math Kernel Library (Intel® MKL)and Intel MKL-DNN.
Release notes
=============
*************
View current `release notes`_ for the Deep Learning Reference Stack.
View current `benchmark results`_ for the Deep Learning Reference Stack.
* View current `release notes`_ for the Deep Learning Reference Stack.
* View current `TensorFlow benchmark results`_ for the Deep Learning
Reference Stack with TensorFlow.
* View current `PyTorch benchmark results`_ for the Deep Learning Reference
Stack with PyTorch.
.. note::
Performance test numbers in the Deep Learning Reference Stack were obtained using `runc` as the runtime.
Prerequisites
=============
*************
* |CL| installed on host system. If not installed, :ref:`bare-metal-install`
* |CL| installed on host system. :ref:`Install <bare-metal-install>`
* `containers-basic` bundle
* `cloud-native-basic` bundle
In |CL|, `containers-basic` provides Docker\*, which is required for
TensorFlow benchmarking. Use the :command:`swupd` utility to check if
`containers-basic` and `cloud-native-basic` are present:
TensorFlow and PyTorch benchmarking. Use the :command:`swupd` utility to
check if `containers-basic` and `cloud-native-basic` are present:
.. code-block:: bash
@@ -47,7 +54,7 @@ If you need to install the `containers-basic` or `cloud-native-basic`, enter:
sudo swupd bundle-add containers-basic cloud-native-basic
To ensure that kubernetes is correctly installed and configured,
To ensure that Kubernetes is correctly installed and configured, follow
:ref:`kubernetes`.
We have validated these steps against the following software package
@@ -59,7 +66,7 @@ versions:
* Go 1.11.12
TensorFlow single and multi-node benchmarks
============================================
*******************************************
This section describes running the `TensorFlow benchmarks`_ in single node.
For multi-node testing, replicate these steps for each node. These steps
@@ -73,13 +80,14 @@ TensorFlow.
.. code-block:: bash
docker run --name <image name> --rm -i -t <clearlinux/stacks-dlrs-TYPE> bash
docker run --name <image name> --rm -i -t <clearlinux/
stacks-dlrs-TYPE> bash
.. note::
You will enter the following commands in the running container.
Replace <docker_name> with the <image name> you specified above.
Launching the docker image with the :command:`-i` argument will put
you into interactive mode within the container. You will enter the
following commands in the running container.
#. Clone the benchmark repository:
@@ -98,28 +106,66 @@ TensorFlow.
You can replace the model with one of your choice supported by the
TensorFlow benchmarks.
PyTorch single and multi-node benchmarks
****************************************
This section describes running the `PyTorch benchmarks`_ for Caffe2 in
single node. We will be looking at validating the Caffe2 APIs with the
official benchmarks, but the same process applies for other cases.
#. Download either the `PyTorch with OpenBLAS`_ or the `PyTorch with Intel
MKL-DNN`_ docker image
from `Docker Hub`_.
#. Run the image with Docker:
.. code-block:: bash
docker run --name <image name> --rm -i -t <clearlinux/stacks-dlrs-TYPE> bash
.. note::
Launching the docker image with the :command:`-i` argument will put
you into interactive mode within the container. You will enter the
following commands in the running container.
#. Clone the benchmark repository:
.. code-block:: bash
git clone https://github.com/pytorch/pytorch.git
#. Next, execute the benchmark script to run the benchmark.
.. code-block:: bash
cd pytorch/caffe2/python
python convnet_benchmarks.py --batch_size 32 \
--cpu \
--model AlexNet
Kubeflow multi-node benchmarks
==============================
******************************
The benchmark workload will run in a Kubernetes cluster. We will use
`Kubeflow`_ for the Machine Learning workload deployment on three nodes.
Kubernetes setup
****************
================
Follow the instructions in the :ref:`kubernetes` tutorial to get set up on
|CL|. The kubernetes community also has
`instructions for creating a cluster`_.
Kubernetes networking
*********************
=====================
We used `flannel`_ as the network provider for these tests. If you are
comfortable with another network layer, refer to the Kubernetes
`networking documentation`_ for setup.
Images
******
======
We need to add `launcher.py` to our docker image to include the Deep
Learning Reference Stack and put the benchmarks repo in the correct
@@ -139,12 +185,14 @@ kubeflow. We are working to create these images as part of our release
cycle.
ksonnet\*
*********
=========
Kubeflow uses ksonnet* to manage deployments, so we need to install that before setting up Kubeflow.
Kubeflow uses ksonnet\* to manage deployments, so we need to install that
before setting up Kubeflow.
Since Clear Linux version 27550, the ksonnet was added to the bundle cloud-native-basic. But if using
old versions (not recommended), please manually install the ksonnet as below.
Since Clear Linux version 27550, the ksonnet was added to the bundle
cloud-native-basic. But if using old versions (not recommended), please
manually install the ksonnet as below.
On |CL|, follow these steps:
@@ -161,7 +209,7 @@ After the ksonnet installation is complete, ensure that binary `ks` is
accessible across the environment.
Kubeflow
********
========
Once you have Kubernetes running on your nodes, you can setup `Kubeflow`_ by
following these instructions from their `quick start guide`_.
@@ -194,7 +242,7 @@ Now you have all the required kubeflow packages, and you can deploy the primary
This creates the CustomResourceDefinition(CRD) endpoint to launch a TFJob.
Run a TFJob
===========
***********
#. Select this link for the `ksonnet registries for deploying TFJobs`_.
@@ -229,29 +277,47 @@ Run a TFJob
This will replicate and deploy three test setups in your Kubernetes cluster.
Results of Running this Tutorial
================================
********************************
You need to parse the logs of the Kubernetes pod to get the performance
numbers. The pods will still be around post completion and will be in
‘Completed’ state. You can get the logs from any of the pods to inspect the
benchmark results. More information about `Kubernetes logging`_ is available from the Kubernetes community.
benchmark results. More information about `Kubernetes logging`_ is available
from the Kubernetes community.
.. _TensorFlow: https://www.tensorflow.org/
.. _Kubeflow: https://www.kubeflow.org/
.. _Docker Hub: https://hub.docker.com/
.. _TensorFlow benchmarks: https://www.tensorflow.org/guide/performance/benchmarks
.. _PyTorch benchmarks: https://github.com/pytorch/pytorch/blob/master/caffe2/python/convnet_benchmarks.py
.. _instructions for creating a cluster: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
.. _flannel: https://github.com/coreos/flannel
.. _networking documentation: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
.. _quick start guide: https://www.kubeflow.org/docs/started/getting-started/
.. _Eigen: https://hub.docker.com/r/clearlinux/stacks-dlrs-oss/
.. _Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-dlrs-mkl/
.. _PyTorch with OpenBLAS: https://hub.docker.com/r/clearlinux/stacks-pytorch-oss
.. _PyTorch with Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-pytorch-mkl
.. _release notes: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs
.. _ksonnet registries for deploying TFJobs: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow/dlrs-tfjob
.. _Kubernetes logging: https://kubernetes.io/docs/concepts/cluster-administration/logging/
.. _benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack
.. _TensorFlow benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack
.. _PyTorch benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack-pytorch
source/clear-linux/tutorials/greengrass.rst
+147 -137
View File
@@ -1,40 +1,48 @@
.. _greengrass:
Enable AWS Greengrass* and OpenVINO™ on |CL-ATTR|
#################################################
Enable AWS Greengrass\* and OpenVINO™ toolkit
#############################################
Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers
to deploy inference functionalities [1] on Intel® IoT edge devices with
accelerators (CPU, Integrated GPU, Intel® FPGA, and Intel® Movidius™). These
functions provide a great developer experience and seamless migration of
visual analytics from cloud to edge in a secure manner using a containerized
environment. Hardware-accelerated FaaS provides the best-in-class
Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers to
deploy inference functionalities [1] on Intel® IoT edge devices with
accelerators (CPU, Integrated GPU, Intel® FPGA, and Intel® Movidius™
technology). These functions provide a great developer experience and seamless
migration of visual analytics from cloud to edge in a secure manner using a
containerized environment. Hardware-accelerated FaaS provides the best-in-class
performance by accessing optimized deep learning libraries on Intel® IoT
edge devices with accelerators.
This tutorial will demonstrate how to:
This tutorial demonstrates how to:
* Set up the Intel® edge device with |CL-ATTR|
* Install the OpenVINO™ and AWS Greengrass* software stacks
* Use AWS Greengrass and lambdas to deploy the FaaS samples from the cloud
* Install the OpenVINO™ toolkit and Amazon Web Services\* (AWS\*)
Greengrass\* software stacks
* Use AWS Greengrass\* and AWS Lambda\* to deploy the FaaS samples from the cloud
Supported Platforms
Refer to the following topics:
.. contents:: :local:
:depth: 1
Supported platforms
*******************
* Operating System: |CL| latest release
* Hardware: Intel® core platforms (Tutorial supports inference on CPU only)
* Hardware: Intel® core platforms (This tutorial supports inference on CPU only.)
Description of Samples
**********************
Sample description
==================
The AWS Greengrass samples are located at the `Edge-Analytics-FaaS`_. For this tutorial we will use the 1.0 version of the source code.
The AWS Greengrass samples are located at `Edge-Analytics-FaaS`_. This
tutorial uses the 1.0 version of the source code.
We provide the following AWS Greengrass samples:
|CL| provides the following AWS Greengrass samples:
* `greengrass_classification_sample.py`_
This AWS Greengrass sample classifies a video stream using classification
networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS*
networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS\*
IoT Cloud every second.
* `greengrass_object_detection_sample_ssd.py`_
@@ -46,8 +54,8 @@ We provide the following AWS Greengrass samples:
coordinates on AWS IoT Cloud every second.
Installing |CL| on the edge device
**********************************
Install the OS on the edge device
*********************************
Start with a clean installation of |CL| on a new system, using the
:ref:`bare-metal-install`, found in :ref:`get-started`.
@@ -56,8 +64,8 @@ Create user accounts
====================
After |CL| is installed, create two user accounts. Create an administrative
user in |CL|. You will also create a user account for the Greengrass
services to use (see Greengrass user below).
user in |CL| and create a user account for the Greengrass services to use (see
Greengrass user below).
#. Create a new user and set a password for that user. Enter the following
commands as ``root``:
@@ -68,7 +76,7 @@ services to use (see Greengrass user below).
passwd <userid>
#. Next, enable the :command:`sudo` command for your new ``<userid>``. Add
``<userid>`` to the ``wheel`` group:
``<userid>`` to the *wheel* group:
.. code-block:: bash
@@ -82,14 +90,13 @@ services to use (see Greengrass user below).
.. note::
By default |CL| does not create an :file:`/etc/fstab` file.
The Greengrass service needs to have the file created before
it will run.
By default, |CL| does not create an :file:`/etc/fstab` file.
You must create this file before the Greengrass service runs.
Add required bundles
====================
Use the ``swupd`` software updater utility to add the prerequisite bundles
Use the :command:`swupd` software updater utility to add the prerequisite bundles
for the OpenVINO software stack:
.. code-block:: bash
@@ -100,20 +107,20 @@ for the OpenVINO software stack:
Learn more about how to :ref:`swupd-guide`.
The ``computer-vision-basic`` bundle will install the OpenVINO software,
along with the edge device models needed.
The :command:`computer-vision-basic` bundle installs the OpenVINO™ toolkit,
and the sample models optimized for Intel® edge platforms.
Converting Deep Learning Models
===============================
Convert deep learning models
============================
Locate Sample Models
Locate sample models
--------------------
There are two types of provided models that can be used in conjunction with AWS Greengrass
for this tutorial: classification or object detection.
There are two types of provided models that can be used in conjunction with
AWS Greengrass for this tutorial: classification or object detection.
To complete this tutorial using an image classification model,
download the BVLC Alexnet model files `bvlc_alexnet.caffemodel`_ and `deploy.prototxt`_
download the BVLC AlexNet model files `bvlc_alexnet.caffemodel`_ and `deploy.prototxt`_
to the default model_location at :file:`/usr/share/openvino/models`.
Any custom pre-trained classification models can be used with the
classification sample.
@@ -123,12 +130,12 @@ are included with the computer-vision-basic bundle installation at :file:`/usr/s
These models are provided as an example; however, you may also use a custom SSD model
with the Greengrass object detection sample.
Running Model Optimizer
-----------------------
Run model optimizer
-------------------
Follow these instructions for `converting deep learning models to Intermediate Representation using Model Optimizer`_. To optimize either of the afformentioned sample models, run one of the following commands.
Follow these instructions for `converting deep learning models to Intermediate Representation using Model Optimizer`_. To optimize either of the sample models described above, run one of the following commands.
For classification using BVLC Alexnet model:
For classification using BVLC AlexNet model:
.. code-block:: bash
@@ -148,40 +155,40 @@ For object detection using SqueezeNetSSD-5Class model:
In these examples:
* ``<model_location>`` is :file:`/usr/share/openvino/models`
* ``<model_location>`` is :file:`/usr/share/openvino/models`.
* ``<data_type>`` is FP32 or FP16, depending on target device.
* ``<output_dir>`` is the directory where the user wants to store the
Intermediate Representation (IR). IR contains .xml format corresponding
to the network structure and .bin format corresponding to weights. This
.xml file should be passed to <PARAM_MODEL_XML>.
* ``<output_dir>`` is the directory where the Intermediate Representation
(IR) is stored. IR contains .xml format corresponding to the network
structure and .bin format corresponding to weights. This .xml file should be
passed to :command:`<PARAM_MODEL_XML>`.
* In the BVLC Alexnet model, the prototxt defines the input shape with
* In the BVLC AlexNet model, the prototxt defines the input shape with
batch size 10 by default. In order to use any other batch size, the
entire input shape needs to be provided as an argument to the model
optimizer. For example, to use batch size 1, you can provide
“--input_shape [1,3,227,227]”.
entire input shape must be provided as an argument to the model
optimizer. For example, to use batch size 1, you must provide:
``--input_shape [1,3,227,227]``
Configuring an AWS Greengrass group
===================================
Configure AWS Greengrass group
******************************
For each Intel® edge platform, we need to create a new AWS Greengrass group
For each Intel® edge platform, you must create a new AWS Greengrass group
and install AWS Greengrass core software to establish the connection between
cloud and edge.
#. To create an AWS Greengrass group, follow the
`AWS Greengrass developer guide`_
#. To create an AWS Greengrass group, follow the instructions in
`Configure AWS IoT Greengrass on AWS IoT`_.
#. To install and configure AWS Greengrass core on edge platform, follow
the instructions at `Start AWS Greengrass on the Core Device`_. In
step 8(b), download the x86_64 Ubuntu configuration of the AWS Greengrass
the instructions in `Start AWS Greengrass on the Core Device`_. In
step 8(b), download the x86_64 Ubuntu\* configuration of the AWS Greengrass
core software.
.. note::
You will not need to run the ``cgroupfs-mount.sh`` script in step #6
You do not need to run the :file:`cgroupfs-mount.sh` script in step #6
of Module 1 of the `AWS Greengrass developer guide`_ because this is
enabled already in |CL|.
@@ -190,13 +197,13 @@ cloud and edge.
.. note::
Security certificates are linked to your AWS* account.
Security certificates are linked to your AWS account.
Creating and Packaging Lambda Functions
=======================================
Create and package Lambda function
**********************************
#. Complete steps 1-4 of the tutorial at `Create and Package Lambda Function`_ .
#. Complete steps 1-4 of the AWS Greengrass tutorial at `Create and Package a Lambda Function`_.
.. note::
@@ -204,7 +211,7 @@ Creating and Packaging Lambda Functions
environment on the edge device.
#. In step 5, replace greengrassHelloWorld.py with the classification or object detection
#. In step 5, replace :file:`greengrassHelloWorld.py` with the classification or object detection
Greengrass sample from `Edge-Analytics-Faas`_:
* Classification: `greengrass_classification_sample.py`_
@@ -227,36 +234,35 @@ Creating and Packaging Lambda Functions
zip -r greengrass_lambda.zip greengrasssdk
greengrass_object_detection_sample_ssd.py
#. Return to the AWS Documentation and follow steps 6-11 to `complete creating lambdas`_.
#. Return to the AWS documentation section called `Create and Package a Lambda Function`_
and complete the procedure.
.. note::
In step 9(a) of the AWS documentation, while uploading the zip file,
make sure to name the handler as below depending on the AWS Greengrass
sample you are using:
make sure to name the handler to one of the following, depending on the
AWS Greengrass sample you are using:
* greengrass_object_detection_sample_ssd.function_handler (or)
* greengrass_object_detection_sample_ssd.function_handler
* greengrass_classification_sample.function_handler
Deploying Lambdas
=================
Configuring the Lambda function
-------------------------------
Configure Lambda function
*************************
After creating the Greengrass group and the lambda function, start
configuring the lambda function for AWS Greengrass.
After creating the Greengrass group and the Lambda function, start
configuring the Lambda function for AWS Greengrass.
#. Follow steps 1-8 in `Configure the Lambda Function`_ of the AWS
#. Follow steps 1-8 in `Configure the Lambda Function for AWS IoT Greengrass`_ in the AWS
documentation.
#. In addition to the details mentioned in step 8, change the Memory limit
to 2048MB to accommodate large input video streams.
to 2048 MB to accommodate large input video streams.
#. Add the following environment variables as key-value pairs when editing
the lambda configuration and click on update:
the Lambda configuration and click on update:
.. list-table:: **Table 1. Environment Variables: Lambda Configuration**
.. list-table:: **Table 1. Environment variables: Lambda configuration**
:widths: 20 80
:header-rows: 1
@@ -282,94 +288,100 @@ configuring the lambda function for AWS Greengrass.
(e.g. 1 for top-1 result, 5 for top-5 results)
#. Add subscription to subscribe, or publish messages from AWS Greengrass
lambda function by following the steps 10-14 in `Configure the Lambda Function`_
Lambda function by completing the procedure in `Configure the Lambda Function for AWS IoT Greengrass`_.
.. note::
The “Optional topic filter” field should be the topic
mentioned inside the lambda function.
The optional topic filter field is the topic mentioned inside the Lambda
function. In this tutorial, sample topics include the following:
:command:`openvino/ssd` or :command:`openvino/classification`
For example, openvino/ssd or openvino/classification
Add local resources
===================
Local Resources
---------------
#. Select `this link to add local resources and access privileges`_.
Refer to the AWS documentation for details about `local resources and access privileges`_.
Following are the local resources needed for the CPU:
The following table describes the local resources needed for the CPU:
.. list-table:: **Local Resources**
:widths: 20, 20, 20, 20
:header-rows: 1
.. list-table:: **Local resources**
:widths: 20, 20, 20, 20
:header-rows: 1
* - Name
- Resource type
- Local path
- Access
* - Name
- Resource type
- Local path
- Access
* - ModelDir
- Volume
- <MODEL_DIR> to be specified by user
- Read-Only
* - ModelDir
- Volume
- <MODEL_DIR> to be specified by user
- Read-Only
* - Webcam
- Device
- /dev/video0
- Read-Only
* - Webcam
- Device
- /dev/video0
- Read-Only
* - DataDir
- Volume
- <DATA_DIR> to be specified by user. Holds both input and output
data.
- Read and Write
* - DataDir
- Volume
- <DATA_DIR> to be specified by user. Holds both input and output
data.
- Read and Write
Deploy
------
Deploy Lambda function
**********************
To `deploy the lambda function to AWS Greengrass core device`_, select
“Deployments” on group page and follow the instructions.
Refer to the AWS documentation for instructions on how to
`deploy the lambda function to AWS Greengrass core device`_. Select
*Deployments* on the group page and follow the instructions.
Output Consumption
------------------
Output consumption
==================
There are four options available for output consumption. These options are
used to report, stream, upload, or store inference output at an interval
defined by the variable ``reporting_interval`` in the AWS Greengrass samples.
defined by the variable :command:`reporting_interval` in the AWS Greengrass samples.
a. IoT Cloud Output:
This option is enabled by default in the AWS Greengrass samples using a
variable ``enable_iot_cloud_output``. We can use it to verify the lambda
a. IoT cloud output:
This option is enabled by default in the AWS Greengrass samples using the
:command:`enable_iot_cloud_output` variable. You can use it to verify the lambda
running on the edge device. It enables publishing messages to IoT cloud
using the subscription topic specified in the lambda (For example,
‘openvino/classification’ for classification and ‘openvino/ssd’ for
object detection samples). For classification, top-1 result with class
using the subscription topic specified in the lambda. (For example, topics
may include :command:`openvino/classification` for classification and :command:`openvino/ssd`
for object detection samples.) For classification, top-1 result with class
label are published to IoT cloud. For SSD object detection, detection
results such as bounding box co-ordinates of objects, class label, and
results such as bounding box coordinates of objects, class label, and
class confidence are published.
Follow the instructions here to `view the output on IoT cloud`_
Follow the instructions here to `view the output on IoT cloud`_.
b. Kinesis Streaming:
b. Kinesis streaming:
This option enables inference output to be streamed from the edge device
to cloud using Kinesis [3] streams when ‘enable_kinesis_output’ is set
to cloud using Kinesis [3] streams when :command:`enable_kinesis_output` is set
to True. The edge devices act as data producers and continually push
processed data to the cloud. The users need to set up and specify
processed data to the cloud. You must set up and specify
Kinesis stream name, Kinesis shard, and AWS region in the AWS Greengrass
samples.
c. Cloud Storage using AWS S3 Bucket:
c. Cloud storage using AWS S3 bucket:
When the ‘enable_s3_jpeg_output’ variable is set to True, it enables uploading and storing processed frames (in JPEG format) in an AWS S3 bucket. The users need to set up and specify the S3 bucket name in the
AWS Greengrass samples to store the JPEG images. The images are named using the timestamp and uploaded to S3.
When the :command:`enable_s3_jpeg_output` variable is set to True, it enables
uploading and storing processed frames (in jpeg format) in an AWS S3
bucket. You must set up and specify the S3 bucket name in the AWS
Greengrass samples to store the JPEG images. The images are named using the
timestamp and uploaded to S3.
d. Local Storage:
d. Local storage:
When the ‘enable_s3_jpeg_output’ variable is set to True, it enables storing processed frames (in JPEG format) on the edge device. The
images are named using the timestamp and stored in a directory specified
by ‘PARAM_OUTPUT_DIRECTORY’.
When the :command:`enable_s3_jpeg_output` variable is set to True, it enables
storing processed frames (in jpeg format) on the edge device. The images
are named using the timestamp and stored in a directory specified by
:command:`PARAM_OUTPUT_DIRECTORY`.
References
-----------
**********
1. AWS Greengrass: https://aws.amazon.com/greengrass/
2. AWS Lambda: https://aws.amazon.com/lambda/
@@ -387,17 +399,15 @@ References
.. _converting deep learning models to Intermediate Representation using Model Optimizer: https://software.intel.com/en-us/articles/OpenVINO-ModelOptimizer
.. _AWS Greengrass developer guide: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html
.. _AWS Greengrass Developer Guide: https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html
.. _Configure AWS IoT Greengrass on AWS IoT: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html
.. _Start AWS Greengrass on the Core Device: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-device-start.html
.. _AWS Greengrass Core SDK: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html
.. _Configure the Lambda Function for AWS IoT Greengrass: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html
.. _complete creating lambdas: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html
.. _Configure the Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html
.. _Add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _deploy the lambda function to AWS Greengrass core device: https://docs.aws.amazon.com/greengrass/latest/developerguide/configs-core.html
@@ -407,4 +417,4 @@ References
.. _this link to add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _Create and Package Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html
.. _Create and Package a Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html
source/clear-linux/tutorials/kubernetes/kubernetes-bp.rst
+92
View File
@@ -0,0 +1,92 @@
.. _kubernetes-bp:
Kubernetes Best Practices on |CL|
#################################
Use swupd to update clusters
****************************
This tutorial shows you how to manage your Kubernetes cluster while using
:command:`swupd` to update |CL-ATTR|.
In our tutorial :ref:`kubernetes`, we explain how to set up a Kubernetes
cluster on |CL| using `kubeadm`. `Kubeadm documentation`_ often builds on the
assumption that the distribution uses a traditional package manager (e.g.,
RPM/DEB).
In contrast, |CL| uses `swupd` to update the OS, which in this case updates
all of the kubernetes node and client binaries simultaneously, as part of
the `cloud-native-basic` bundle (e.g., kubectl, kubeadm, kubelet). Running
:command:`sudo swupd update` requires special care to ensure the OS
incorporates the latest Kubernetes upgrades.
This document describes best practices to manage cluster upgrades with
`kubeadm` on a |CL|-based cluster.
Prerequisites
*************
Assure that you:
* Completed :ref:`kubernetes`
* Installed the bundle `cloud-native-basic`
.. note::
Other Linux\* distros shown in Kubernetes upgrade documentation reflect
`apt-get update`, `apt-mark hold kubeadm`, and similar commands; however, such commands **are not valid** on |CL|.
Update the control plane
************************
#. Read kubernetes documentation `before you begin`_.
#. On your master node, run the command:
.. code-block:: bash
sudo swupd update
.. note::
If the minor version of Kubernetes changes, |CL| shows a message-of-the-day, or `motd`. When the `motd` appears, you **must postpone** a kubelet restart on master and nodes until the control plane is properly updated. :command:`swupd update` does not restart services automatically unless explicitly configured to do so.
#. Now follow these instructions in kubernetes documentation.
* `Upgrade control plane`_
* `Drain control plane node`_
* `Restart Kubelet and undrain node`_
Update worker nodes
*******************
#. On each worker node, run the command:
.. code-block:: bash
sudo swupd update
#. Now follow these instructions in kubernetes documentation:
* `Drain node`_
* `Update kubelet configuration`_
* `Restart Kubelet and undrain node`_
.. _Kubeadm documentation: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/
.. _Restart Kubelet and undrain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#restart-the-kubelet-for-all-nodes
.. _Update kubelet configuration: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#upgrade-the-kubelet-config-on-worker-nodes
.. _Drain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes
.. _Restart kubelet and undrain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#restart-the-kubelet-for-all-nodes
.. _Upgrade control plane: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#upgrade-the-control-plane-node
.. _Drain control plane node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes
.. _Kubeadmn documentation: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
.. _before you begin: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#before-you-begin
source/clear-linux/tutorials/kubernetes.rst → source/clear-linux/tutorials/kubernetes/kubernetes.rst
+44 -39
View File
@@ -1,7 +1,7 @@
.. _kubernetes:
Run Kubernetes\* on |CL-ATTR|
#############################
Run Kubernetes\*
################
This tutorial describes how to install, configure, and run the
`Kubernetes container orchestration system`_ on |CL-ATTR| using CRI+O and
@@ -43,17 +43,16 @@ Kubernetes, a set of supported :abbr:`CRI (Container Runtime Interface)`
runtimes, and networking plugins, are included in the `cloud-native-basic`_
bundle.
.. note::
CRI-O’s default plugin_dir is :file:`/opt/bin/cni`.
CNI plugins are installed as part of ``cloud-native-basic``.
To install this framework, enter the following command:
.. code-block:: bash
sudo swupd bundle-add cloud-native-basic
.. note::
For more on networking plugins, see `Install pod network add-on`_.
Configure Kubernetes
********************
@@ -150,7 +149,6 @@ Configure and run CRI-O + kata-runtime
sudo systemctl daemon-reload
sudo systemctl restart crio
sudo systemctl restart kubelet
#. Initialize the master control plane with the command:
@@ -158,7 +156,6 @@ Configure and run CRI-O + kata-runtime
sudo kubeadm init --cri-socket=/run/crio/crio.sock
Install pod network add-on
**************************
@@ -166,47 +163,49 @@ You must choose and install a `pod network add-on`_ to allow your pods to
communicate. Check whether or not your add-on requires special flags when you
initialize the master control plane.
The CRI-O default plugin_dir is :file:`/opt/cni/bin`. This must be a
writable directory because third-party networking add-ons will install
themselves there.
.. note::
CNI plugins provided by |CL| are installed as part of *cloud-native-basic*
in :file:`/usr/libexec/cni/` and are currently *not* found by CRI-O by
default. These separate directories are required because `swupd` controls
the content of :file:`/usr` and leaves :file:`/opt` unchanged.
When using third-party network add-ons that rely on those plugins, such as
Weave or Flannel do, make them available by creating symlinks:
.. code-block:: bash
sudo mkdir -p /opt/cni/bin
.. code-block:: bash
for i in /usr/libexec/cni/*; do sudo ln -sf $i /opt/cni/bin; done
**Notes about Weave Net add-on**
The Weave Net add-on works by default when the above configuration is done.
**Notes about flannel add-on**
If you choose the `flannel` add-on, then you must add the following to the
`kubeadm init` command:
.. code-block:: bash
.. code-block:: bash
--pod-network-cidr 10.244.0.0/16
--pod-network-cidr 10.244.0.0/16
If you are using CRI-O and `flannel` and you want to use Kata Containers, edit the :file:`/etc/crio/crio.conf` file to add:
If you are using CRI-O and `flannel` and you want to use Kata Containers,
edit the :file:`/etc/crio/crio.conf` file to add:
.. code-block:: bash
[crio.runtime]
manage_network_ns_lifecycle = true
Create a symlink for the network overlays:
.. code-block:: bash
sudo ln -s /usr/libexec/cni /opt/cni/bin
.. note::
|CL| installs CNI plugins that are part of the `cloud-native-basic`
bundle to :file:`/usr/libexec/cni`. The directory is required because `
swupd verify` may use it to repair a system to a known good state.
**Notes about Weave Net add-on**
If you choose the `Weave Net` add-on, you must make the following
changes because it installs itself in the :file:`/opt/cni/bin` directory.
For using CRI-O and ``Weave Net``, complete the following step.
Add the `loopback` CNI plugin to the plugin path with the command:
.. code-block:: bash
sudo ln -s /usr/libexec/cni/loopback /opt/bin/cni/loopback
Use your cluster
****************
@@ -240,8 +239,8 @@ Read the Kubernetes documentation to learn more about:
* `Joining your nodes`_
Package configuration customization in |CL| (optional)
******************************************************
Package configuration customization (optional)
**********************************************
|CL| is a stateless system that looks for user-defined package configuration
files in the :file:`/etc/<package-name>` directory to be used as default. If
@@ -289,6 +288,12 @@ commands as a shell script to configure all of these services in one step:
EOF
done
Next steps
**********
:ref:`kubernetes-bp`
Troubleshooting
***************
source/clear-linux/tutorials/multi-boot/multi-boot.rst
+1 -1
View File
@@ -55,7 +55,7 @@ installation of the tested operating systems.
.. csv-table:: Table 1: OS specific installation information
:header: # , OS, Version, Partition Size [#]_, Swap Size [#]_, EFI Partition Size [#]_, Download Link
1,Clear Linux,16140,50 GB,8 GB,1 GB,https://download.clearlinux.org/releases/16140/clear/
1,Clear Linux,16140,50 GB,8 GB,1 GB,https://cdn.download.clearlinux.org/releases/16140/clear/
2,Windows,Server 2016,50 GB,N/A,Shared with #1,https://www.microsoft.com/en-us/cloud-platform/windows-server
3,Red Hat\*,Server 7.4 Beta,45 GB,Shared with #1,Shared with #1,https://access.redhat.com/downloads/
4,SUSE\*,Server 12 SP2,45 GB,Shared with #1,Shared with #1,https://www.suse.com/download-linux/
source/clear-linux/tutorials/nvidia.rst
+180
View File
@@ -0,0 +1,180 @@
.. _nvidia:
Install NVIDIA Drivers
######################
NVIDIA is a manufacture of graphics processing units (GPU), also known as
graphics cards.
NVIDIA devices on Linux have two popular device driver options: the opensource
drivers from the `nouveau project`_ or the proprietary drivers published by
NVIDIA. The nouveau drivers are built into the |CL-ATTR| kernel and are loaded
automatically at system boot if a compatible card is detected.
These instructions show how to use the proprietary NVIDIA drivers which
require a manual installation.
.. contents:: :local:
:depth: 2
Prerequisites
*************
* A |CL| system with a desktop installed
* A NVIDIA device installed
Install the LTS kernel and DKMS
*******************************
The Long Term Support (LTS) kernel variant is most likely to remain
compatible with NVIDIA drivers. The `Dynamic Kernel Module System (DKMS)
<kernel-modules-dkms>`_ allows the NVIDIA kernel modules to be automatically
integrated when kernel updates occur in |CL|. Install both using the
instructions below:
.. include:: ../guides/maintenance/kernel-modules-dkms.rst
:start-after: kernel-modules-dkms-install-begin:
:end-before: kernel-modules-dkms-install-end:
Download and install the NVIDIA Linux Driver
********************************************
Download the NVIDIA Linux Driver
================================
#. Identify the model of NVIDIA GPU that is installed.
.. code-block:: bash
lshw -C display
#. Go to the `NVIDIA Driver Downloads website`_ . Search for and download the
appropriate driver based on the model of NVIDIA GPU you have with *Linux
64-bit* selected as the Operating System .
#. Open a terminal and navigate to where the
:file:`NVIDIA-Linux-x86_64-<VERSION>.run` file was saved. In this
example, it was saved in the Downloads folder.
.. code-block:: bash
cd ~/Downloads/
#. Extract the contents of the .run file.
.. code-block:: bash
sh NVIDIA-Linux-x86_64-<VERSION>.run --extract-only
Disable the nouveau driver
==========================
#. The proprietary NVIDIA driver is incompatible with the nouveau driver and
needs to be disabled before installation can continue.
Disable the nouveau driver by creating a file under :file:`/etc/modprobe.d`
and reboot.
.. code-block:: bash
sudo mkdir /etc/modprobe.d
echo "blacklist nouveau" | sudo tee --append /etc/modprobe.d/nvidia-disable-nouveau.conf
echo "options nouveau modeset=0" | sudo tee --append /etc/modprobe.d/nvidia-disable-nouveau.conf
#. Reboot the system and log back in. It is normal for the graphical
environment to not start with no NVIDIA driver loaded.
Install the NVIDIA Linux Driver
===============================
#. Navigate into the directory where the NVIDIA installer was extracted.
.. code-block:: bash
cd ~/Downloads/NVIDIA-Linux-x86_64-<VERSION>/
#. Run the installer with the advanced options below.
.. code-block:: bash
sudo ./nvidia-installer --no-nvidia-modprobe --no-distro-scripts --no-opengl-files --no-libglx-indirect --no-install-libglvnd --no-install-compat32-libs --dkms --ui=none
#. The installer will prompt to register the kernel module sources with
DKMS. Enter Y for yes.
.. code-block:: bash
Welcome to the NVIDIA Software Installer for Unix/Linux
<snipped>
Would you like to register the kernel module sources with DKMS? This will allow DKMS to automatically build a new
module, if you install a different kernel later.
[default: (Y)es]: Y
#. The graphical interface may automatically start after the NVIDIA driver
is loaded. If it does restart, log back in.
#. Validate the nvidia kernel modules are loaded.
.. code-block:: bash
lsmod | grep ^nvidia
.. note::
The NVIDIA installer places files under the :file:`/usr` subdirectory which
are not managed by |CL| updates. The :command:`swupd verify --fix` command
should be avoided with the proprietary NVIDIA drivers in use.
Uninstalling the NVIDIA driver
******************************
The NVIDIA drivers and associated software can be uninstalled and nouveau
driver restored by:
#. Remove the previously created file :file:`etc/modprobe.d` that is
preventing nouveau from loading.
.. code-block:: bash
sudo rm /etc/modprobe.d/nvidia-disable-nouveau.conf
#. Run the :command:`sudo nvidia-uninstall`
#. Follow the prompts on the screen and reboot the system.
Additional resources
********************
* `Why aren't the NVIDIA Linux drivers open source? <https://nvidia.custhelp.com/app/answers/detail/a_id/1849/kw/Linux>`_
* `Where can I get support for NVIDIA Linux drivers? <https://nvidia.custhelp.com/app/answers/detail/a_id/44/kw/linux>`_
.. _`nouveau project`: https://nouveau.freedesktop.org/wiki/
.. _`NVIDIA Driver Downloads website`: https://www.nvidia.com/download/index.aspx
source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst
-477
View File
@@ -1,477 +0,0 @@
.. _telemetry-backend:
Create a telemetry backend server in |CL-ATTR|
##############################################
|CL-ATTR| includes a telemetry and analytics solution, also known as
telemetrics, as part of the OS that records events of interest and reports
them back to the development team using the telemetrics daemons that are
running on the |CL| client system.
End users can enable or disable the telemetry client component of |CL| and
also redirect where records go if they wish to collect records for themselves
by setting up and using their own telemetry backend server.
A telemetry backend server consists of two Flask applications:
* The :guilabel:`collector` is an ingestion app for records received from the
:guilabel:`telemetrics-client` probes.
* The :guilabel:`telemetryui` web app exposes several visualizations of the
telemetry data. The :guilabel:`telemetryui` app also provides a
REST API to perform queries on the data.
The applications run within a web stack, using the :guilabel:`nginx` web
server, the :guilabel:`uWSGI` application server, and
:guilabel:`PostgreSQL` as the underlying database server. For a detailed
description, visit the |CL| `telemetrics backend`_ server overview.
This tutorial walks you through creating a telemetrics backend server on
your local |CL| machine. The tutorial uses the :command:`deploy.sh` bash
shell script that is maintained in a the `telemetrics backend`_ GitHub repository.
Once the backend server isup and running, we show you how to redirect telemetry
records from the system you set up to this new server location.
.. note::
The telemetrics functionality adheres to `Intel privacy policies`_
regarding the collection and use of :abbr:`PII (Personally Identifiable
Information)` and is open Source. Specifically, no intentionally
identifiable information about the user or system owner is collected.
Prerequisites
*************
For this tutorial, start with a clean installation of |CL| on a new system
using the :ref:`bare-metal-install` getting started guide:
#. Choose to install |CL|.
#. Join the :guilabel:`Stability Enhancement Program` to install and
enable the telemetrics components.
#. Select the manual installation method with the following settings:
* Set the hostname to :guilabel:`clr-telem-server`,
* Create an administrative user named :guilabel:`clear` and add this user
to sudoers,
* Add all additional software bundles.
Download the clearlinux/telemetrics-backend Git repository
**********************************************************
With all prerequisite software bundles installed and logged in with your
administrative user, from your :file:`$HOME` directory, run :command:`git`
to clone the :guilabel:`telemetrics-backend` repository into the
:file:`$HOME/telemetrics-backend` directory:
.. code-block:: console
git clone https://github.com/clearlinux/telemetrics-backend
.. note::
You may need to set up the :envvar:`https_proxy` environment variable
if you have issues reaching github.com.
Run the deploy.sh script to install the backend server
******************************************************
#. Change your current working directory to :file:`telemetrics-backend/scripts`.
#. Run the :command:`./deploy.sh -h` to see the list of options for the
:command:`deploy.sh` script:
.. code-block:: console
cd telemetrics-backend/scripts
./deploy.sh -h
Deploy snapshot of the telemetrics-backend
-a Perform specified action (deploy, install, migrate, resetdb,
restart, uninstall; default: deploy)
-d Distro to deploy to (ubuntu, centos or clr; default: ubuntu)
-h Print these options
-H Set domain for deployment (only accepted value is "localhost" for
now)
-r Set repo location to deploy from
(default: https://github.com/clearlinux/telemetrics-backend)
-s Set source location (default: "master" branch from git repo)
-t Set source type (tarball, or git; default: git)
-u Perform complete uninstallation
The :command:`deploy.sh` is a bash shell script that allows you to perform the
following actions:
* *deploy* - install a complete instance of the telemetrics backend
server and all required components. This is the default action if no
*-a* argument is given on the command line.
* *install* - installs and enables all required components for the
telemetrics backend server.
* *migrate* - migrate database to new schema.
* *resetdb* - reset the database.
* *restart* - restart the nginx and uWSGI services.
* *uninstall* - uninstall all packages.
.. note::
The *uninstall* option does not perform any actions if the
distro is set to |CL| and will only uninstall packages if the distro is
Ubuntu
Next, we install the telemetrics backend server with the following options:
* *-a install* to perform an install
* *-d clr* to install to a |CL| distro
* *-H localhost* to set the domain to localhost
We do not need to set the following options since the values are set to the
correct values we want by default:
* *-r https://github.com/clearlinux/telemetrics-backend* sets the
repo location for :command:`git` to clone from.
* *-s master* to set the location, or branch.
* *-t git* to set the source type to git.
.. caution::
The :file:`deploy.sh` shell script has minimal error checking and makes
several changes to your system. Be sure that the options you define on the
cmdline are correct before proceeding.
To begin the installation with the options defined:
#. Run the shell script from the :file:`$HOME/telemetrics-backend/scripts`
directory:
.. code-block:: console
./deploy.sh -H localhost -a install -d clr
The script will start and list all the defined options and prompt you for the
:guilabel:`PostgreSQL` database password as shown below:
.. code-block:: console
Options:
host: localhost
distro: clr
action: install
repo: https://github.com/clearlinux/telemetrics-backend
source: master
type: git
DB password: (default: postgres):
#. For the :guilabel:`DB password:`, press the :kbd:`Enter` key to accept the
default password `postgres`.
The :command:`swupd` begins installing the required software bundles to set
up the telemetrics backend server. The output will look similar to what is
shown below:
.. code-block:: console
swupd-client bundle adder 3.12.7
Copyright (C) 2012-2017 Intel Corporation
Downloading packs...
Extracting application-server pack for version 18740
...5%
Extracting database-basic-dev pack for version 18670
...10%
Extracting database-basic pack for version 18670
...15%
Extracting os-clr-on-clr pack for version 18740
...21%
Extracting sysadmin-basic-dev pack for version 18740
...26%
Extracting storage-utils-dev pack for version 18770
...31%
Extracting os-core-update-dev pack for version 18760
...36%
Extracting network-basic-dev pack for version 18760
...42%
Extracting mixer pack for version 18790
...47%
Extracting os-installer pack for version 18800
...52%
Extracting mail-utils-dev pack for version 18760
...57%
Extracting koji pack for version 18800
...63%
Extracting go-basic pack for version 18800
...68%
Extracting dev-utils-dev pack for version 18820
...73%
Extracting python-basic-dev pack for version 18750
...78%
Extracting perl-basic-dev pack for version 18610
...84%
Extracting c-basic pack for version 18800
...89%
Extracting os-core-dev pack for version 18800
...94%
Extracting web-server-basic pack for version 18680
...100%
Installing bundle(s) files...
...100%
Calling post-update helper scripts.
Possible filedescriptor leak : 8 (socket:[30833])
Bundle(s) installation done.
.. note::
This script uses :command:`sudo` to run commands and you may be prompted to
enter your user password at any time while the script is executing. If this
occurs, enter your user password to execute the :command:`sudo` command.
.. code-block:: console
Password:
You may also see an informational message about setting the
:envvar:`https_proxy` environment variable if this variable isn't set.
Once the :command:`swupd` command is complete, the script begins processing
the requirements to install and implement the telemetrics server. Finally,
the script enables the server and provides output similar to:
.. code-block:: console
Collecting uwsgitop
Downloading uwsgitop-0.10.tar.gz
Requirement already satisfied: simplejson in /usr/lib/python3.6/site-packages (from uwsgitop)
Collecting argparse (from uwsgitop)
Downloading argparse-1.4.0-py2.py3-none-any.whl
Building wheels for collected packages: uwsgitop
Running setup.py bdist_wheel for uwsgitop ... done
Stored in directory: /root/.cache/pip/wheels/8a/99/e9/accc80bcaa989218da65daaae4205dc4f6288d3551655aa638
Successfully built uwsgitop
Installing collected packages: argparse, uwsgitop
Successfully installed argparse-1.4.0 uwsgitop-0.10
mkdir: created directory '/var/www'
mkdir: created directory '/var/www/telemetry'
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/www/telemetry/venv/bin/python3
Also creating executable in /var/www/telemetry/venv/bin/python
Installing setuptools, pip, wheel...done.
Collecting alembic==0.9.5 (from -r /tmp/requirements.txt.KDI3uU (line 1))
Downloading alembic-0.9.5.tar.gz (990kB)
100% |████████████████████████████████| 993kB 2.1MB/s
Collecting click==6.7 (from -r /tmp/requirements.txt.KDI3uU (line 2))
Downloading click-6.7-py2.py3-none-any.whl (71kB)
100% |████████████████████████████████| 71kB 8.3MB/s
Collecting Flask==0.12.2 (from -r /tmp/requirements.txt.KDI3uU (line 3))
Downloading Flask-0.12.2-py2.py3-none-any.whl (83kB)
100% |████████████████████████████████| 92kB 10.2MB/s
Collecting Flask-Migrate==2.1.0 (from -r /tmp/requirements.txt.KDI3uU (line 4))
Downloading Flask-Migrate-2.1.0.tar.gz
Collecting Flask-SQLAlchemy==2.2 (from -r /tmp/requirements.txt.KDI3uU (line 5))
Downloading Flask_SQLAlchemy-2.2-py2.py3-none-any.whl
Collecting Flask-WTF==0.14.2 (from -r /tmp/requirements.txt.KDI3uU (line 6))
Downloading Flask_WTF-0.14.2-py2.py3-none-any.whl
Collecting itsdangerous==0.24 (from -r /tmp/requirements.txt.KDI3uU (line 7))
Downloading itsdangerous-0.24.tar.gz (46kB)
100% |████████████████████████████████| 51kB 12.4MB/s
Collecting Jinja2==2.9.6 (from -r /tmp/requirements.txt.KDI3uU (line 8))
Downloading Jinja2-2.9.6-py2.py3-none-any.whl (340kB)
100% |████████████████████████████████| 348kB 3.5MB/s
Collecting Mako==1.0.7 (from -r /tmp/requirements.txt.KDI3uU (line 9))
Downloading Mako-1.0.7.tar.gz (564kB)
100% |████████████████████████████████| 573kB 1.9MB/s
Collecting MarkupSafe==1.0 (from -r /tmp/requirements.txt.KDI3uU (line 10))
Downloading MarkupSafe-1.0.tar.gz
Collecting psycopg2==2.7.3 (from -r /tmp/requirements.txt.KDI3uU (line 11))
Downloading psycopg2-2.7.3.tar.gz (425kB)
100% |████████████████████████████████| 430kB 4.0MB/s
Collecting python-dateutil==2.6.1 (from -r /tmp/requirements.txt.KDI3uU (line 12))
Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
100% |████████████████████████████████| 194kB 6.8MB/s
Collecting python-editor==1.0.3 (from -r /tmp/requirements.txt.KDI3uU (line 13))
Downloading python-editor-1.0.3.tar.gz
Collecting six==1.10.0 (from -r /tmp/requirements.txt.KDI3uU (line 14))
Downloading six-1.10.0-py2.py3-none-any.whl
Collecting SQLAlchemy==1.1.13 (from -r /tmp/requirements.txt.KDI3uU (line 15))
Downloading SQLAlchemy-1.1.13.tar.gz (5.2MB)
100% |████████████████████████████████| 5.2MB 394kB/s
Collecting uWSGI==2.0.15 (from -r /tmp/requirements.txt.KDI3uU (line 16))
Downloading uwsgi-2.0.15.tar.gz (795kB)
100% |████████████████████████████████| 798kB 1.5MB/s
Collecting Werkzeug==0.12.2 (from -r /tmp/requirements.txt.KDI3uU (line 17))
Downloading Werkzeug-0.12.2-py2.py3-none-any.whl (312kB)
100% |████████████████████████████████| 317kB 2.2MB/s
Collecting WTForms==2.1 (from -r /tmp/requirements.txt.KDI3uU (line 18))
Downloading WTForms-2.1.zip (553kB)
100% |████████████████████████████████| 563kB 1.7MB/s
Skipping bdist_wheel for psycopg2, due to binaries being disabled for it.
Building wheels for collected packages: alembic, Flask-Migrate, itsdangerous, Mako, MarkupSafe, python-editor, SQLAlchemy, uWSGI, WTForms
Running setup.py bdist_wheel for alembic ... done
Stored in directory: /root/.cache/pip/wheels/d1/0e/b9/fb570150b350298e1d8f1ff38a400ae709580b36e43bc3ac91
Running setup.py bdist_wheel for Flask-Migrate ... done
Stored in directory: /root/.cache/pip/wheels/3d/29/d4/66747eca8b8a28973aa639f39e96a402b3dcab335e608048dd
Running setup.py bdist_wheel for itsdangerous ... done
Stored in directory: /root/.cache/pip/wheels/fc/a8/66/24d655233c757e178d45dea2de22a04c6d92766abfb741129a
Running setup.py bdist_wheel for Mako ... done
Stored in directory: /root/.cache/pip/wheels/33/bf/8f/036f36c35e0e3c63a4685e306bce6b00b6349fec5b0947586e
Running setup.py bdist_wheel for MarkupSafe ... done
Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57
Running setup.py bdist_wheel for python-editor ... done
Stored in directory: /root/.cache/pip/wheels/84/d6/b8/082dc3b5cd7763f17f5500a193b6b248102217cbaa3f0a24ca
Running setup.py bdist_wheel for SQLAlchemy ... done
Stored in directory: /root/.cache/pip/wheels/f0/50/ca/3cb6e78527eb05e180d19632343ee14d2e5c164da2e61fbd2d
Running setup.py bdist_wheel for uWSGI ... done
Stored in directory: /root/.cache/pip/wheels/26/d0/48/e7b0eed63b5d191e89d94e72196aafae93b2b6505a9feafdd9
Running setup.py bdist_wheel for WTForms ... done
Stored in directory: /root/.cache/pip/wheels/36/35/f3/7452cd24daeeaa5ec5b2ea13755316abc94e4e7702de29ba94
Successfully built alembic Flask-Migrate itsdangerous Mako MarkupSafe python-editor SQLAlchemy uWSGI WTForms
Installing collected packages: SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, click, Werkzeug, Jinja2, itsdangerous, Flask, Flask-SQLAlchemy, Flask-Migrate, WTForms, Flask-WTF, psycopg2, uWSGI
Running setup.py install for psycopg2 ... done
Successfully installed Flask-0.12.2 Flask-Migrate-2.1.0 Flask-SQLAlchemy-2.2 Flask-WTF-0.14.2 Jinja2-2.9.6 Mako-1.0.7 MarkupSafe-1.0 SQLAlchemy-1.1.13 WTForms-2.1 Werkzeug-0.12.2 alembic-0.9.5 click-6.7 itsdangerous-0.24 psycopg2-2.7.3 python-dateutil-2.6.1 python-editor-1.0.3 six-1.10.0 uWSGI-2.0.15
mkdir: created directory '/var/log/uwsgi'
Once all the server components have been installed you are prompted to enter
the :guilabel:`PostgreSQL` database password to change it as illustrated below:
.. code-block:: console
Enter password for 'postgres' user:
New password:
Retype new password:
passwd: password updated successfully
Enter `postgres` for the current value of the password and then enter a new
password, retype it to verify the new password and the :guilabel:`PostgreSQL`
database password will be updated.
The script finalizes installation and finishes.
.. code-block:: console
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /usr/lib/systemd/system/postgresql.service.
Cloning into 'telemetrics-backend'...
remote: Counting objects: 344, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 344 (delta 30), reused 50 (delta 20), pack-reused 268
Receiving objects: 100% (344/344), 130.20 KiB | 1.40 MiB/s, done.
Resolving deltas: 100% (177/177), done.
'/tmp/telemetrics-backend/scripts/collector_uwsgi.ini' -> '/tmp/telemetrics-backend/collector/collector_uwsgi.ini'
'/tmp/telemetrics-backend/scripts/telemetryui_uwsgi.ini' -> '/tmp/telemetrics-backend/telemetryui/telemetryui_uwsgi.ini'
mkdir: created directory '/var/www/telemetry/collector/uwsgi-spool'
mkdir: created directory '/var/www/telemetry/telemetryui/uwsgi-spool'
'/tmp/telemetrics-backend/scripts/uwsgi.service' -> '/etc/systemd/system/uwsgi.service'
mkdir: created directory '/etc/nginx'
mkdir: created directory '/etc/nginx/conf.d'
'/usr/share/nginx/conf/nginx.conf.example' -> '/etc/nginx/nginx.conf'
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service.
mkdir: created directory '/etc/uwsgi'
mkdir: created directory '/etc/uwsgi/vassals'
Created symlink /etc/systemd/system/multi-user.target.wants/uwsgi.service → /etc/systemd/system/uwsgi.service.
ALTER ROLE
sed: can't read /tmp/telemetrics-backend/collector/config.py: No such file or directory
cp: cannot stat '/tmp/telemetrics-backend/collector/config.py': No such file or directory
sed: can't read /tmp/telemetrics-backend/telemetryui/config.py: No such file or directory
cp: cannot stat '/tmp/telemetrics-backend/telemetryui/config.py': No such file or directory
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/www/telemetry/venv/bin/python3
Not overwriting existing python script /var/www/telemetry/venv/bin/python (you must use /var/www/telemetry/venv/bin/python3)
Installing setuptools, pip, wheel...done.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 3230c615d6e0, empty message
INFO [alembic.runtime.migration] Running upgrade 3230c615d6e0 -> 466cf2f35d67, empty message
Install complete (installation folder: /var/www/telemetry)
Once the installation is complete you can use your web browser and view the
new server by opening the web browser on the system you installed the backend
server onto and type in ``localhost`` in the address bar. You should see a
web page similar to the one shown in figure 1:
.. figure:: figures/telemetry-backend-1.png
:alt: Telemetry UI
Figure 1: :guilabel:`Telemetry UI`
Redirect telemetry records
**************************
Telemetry records from your system are sent to the server location defined in
the :file:`/usr/share/defaults/telemetrics/telemetrics.conf` configuration
file. You can customize this by copying this file to
:file:`/etc/telemetrics/telemetrics.conf` and changing the ``server=``
setting to your new server location.
#. Create the :file:`/etc/telemetrics` directory and make it your current
working directory.
.. code-block:: console
sudo mkdir -p /etc/telemetrics
cd /etc/telemetrics
#. Copy the default :file:`telemetrics.conf` file to the new
:file:`/etc/telemetrics` directory.
.. code-block:: console
sudo cp /usr/share/defaults/telemetrics/telemetrics.conf .
#. Edit the new :file:`/etc/telemetrics/telemetrics.conf` file with your
editor using the :command:`sudo` directive and change the
:guilabel:`server=` setting to ``http://localhost/v2/collector`` and save
this change in the new file.
.. code-block:: console
server=http://localhost/v2/collector
You can also use the fully qualified domain name for your server instead of
:guilabel:`localhost`.
#. Restart the telemetry daemons to reload the configuration file.
.. code-block:: console
telemctl restart
Test the new telemetry backend server
*************************************
|CL| includes a telemetry test probe called :command:`hprobe` that will send a
``hello`` record to the telemetry backend server. To test that the telemetry
records are now going to your new destination, run the :command:`hprobe`
command to send a ``hello`` record to the server as follows:
.. code-block:: console
hprobe
The record should show up on your new server console as shown in figure 2:
.. figure:: figures/telemetry-backend-2.png
:alt: Telemetry UI
Figure 2: :guilabel:`Telemetry UI`
Congratulations! You've just set up and enabled a new telemetrics backend
server, redirected the records from your local machine to this new server and
tested it using the :command:`hprobe` command to send a ``hello`` record to
it.
Related topics
**************
* `Telemetry feature description`_
* :ref:`Telemetry architecture<telemetry-about>`
* :ref:`telemetry-enable`
* https://github.com/clearlinux/telemetrics-client
* https://github.com/clearlinux/telemetrics-backend
.. _`telemetrics backend`:
https://github.com/clearlinux/telemetrics-backend
.. _`Intel privacy policies`:
https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
.. _`Telemetry feature description`:
https://clearlinux.org/features/telemetry
source/clear-linux/tutorials/telemetry-e2e.rst
-617
View File
@@ -1,617 +0,0 @@
.. _telemetry-e2e:
Develop with telemetry
######################
This tutorial shows you how to set up a telemetry backend server to
manage your records and how to instrument your application with the telemetry
API.
|CL-ATTR| includes a telemetry and analytics solution (also known as
telemetrics) as part of the OS, which records events of interest and reports
them back to the development team using the telemetrics client daemons.
The |CL| telemetry client can be enabled or disabled and records can be
redirected to a desired location. More detailed information about using and
configuring the telemetrics client is found in
the :ref:`telemetrics` guide.
.. contents:: :local:
:depth: 1
Prerequisites
=============
For this tutorial, you can use an existing |CL| system, or you can start with a clean installation of |CL| on a new system.
New Installation
****************
To setup a new system for your telemetry backend server, follow the :ref:`bare-metal-install` getting started guide and:
#. Choose to install |CL|.
#. Join the :guilabel:`Stability Enhancement Program` during the installation process to enable the telemetrics client components.
#. Select the manual installation method with the following settings:
* Set the hostname to :guilabel:`clr-telem-server`
* Create an administrative user named :guilabel:`clear` and add this user to sudoers :ref:`enable-user-space`
* Choose the :file:`dev-utils`, :file:`network-basic`, and :file:`openssh-server` bundles from the bundle list
.. note::
Bundles can also be added to your system after this install process completed. The bundles listed here are a minimal set needed to complete the setup of the telemetry backend server and applications.
Existing System
***************
If you are using an existing |CL| system, make sure you have installed the telemetry and dev-utils bundles. Use the :command:`swupd` utility with the `bundle-list` option and check for "telemetrics" in the list:
.. code-block:: bash
sudo swupd bundle-list
If you need to install the bundles, use :command:`swupd` to do so.
.. code-block:: bash
sudo swupd bundle-add telemetrics dev-utils
More information about enabling and configuring the telemetry client can be found at :ref:`telemetry-enable`.
You will need to run some of the commands in this tutorial with root privileges. You can create a new user or add your user to the sudoers list :ref:`enable-user-space`.
Setting up the telemetry backend server
=======================================
We'll be using the :file:`deploy.sh` file from the `clearlinux/telemetrics-backend`_ Git repository to install required dependencies for the web server applications. The script also configures nginx and uwsgi, deploys snapshots of the applications, and starts all required services.
Clone the clearlinux/telemetrics-backend Git repository
*******************************************************
With all prerequisite software bundles installed, log in with your administrative user, and from your :file:`$HOME` directory, run :command:`git` to clone the :guilabel:`telemetrics-backend` repository into the :file:`$HOME/telemetrics-backend` directory:
.. code-block:: bash
git clone https://github.com/clearlinux/telemetrics-backend
.. note::
You may need to set up the :envvar:`https_proxy` environment variable if you have issues reaching github.com.
Run the deploy.sh script to install the backend server
******************************************************
Change your current working directory to :file:`telemetrics-backend/scripts`.
.. code-block:: bash
cd telemetrics-backend/scripts
Run the :command:`./deploy.sh -h` to see the list of options for the :command:`deploy.sh` script:
.. code-block:: console
./deploy.sh -h
Deploy snapshot of the telemetrics-backend
-a Perform specified action (deploy, install, migrate, resetdb,
restart, uninstall; default: deploy)
-d Distro to deploy to (ubuntu, centos or clr; default: ubuntu)
-h Print these options
-H Set domain for deployment (only accepted value is "localhost" for
now)
-r Set repo location to deploy from
(default: https://github.com/clearlinux/telemetrics-backend)
-s Set source location (default: "master" branch from git repo)
-t Set source type (tarball, or git; default: git)
-u Perform complete uninstallation
The :command:`deploy.sh` is a bash shell script that allows you to perform the following actions:
* *deploy* - install a complete instance of the telemetrics backend server and all required components. This is the default action if no *-a* argument is given on the command line.
* *install* - installs and enables all required components for the telemetrics backend server.
* *migrate* - migrate database to new schema.
* *resetdb* - reset the database.
* *restart* - restart the nginx and uWSGI services.
* *uninstall* - uninstall all packages.
.. note::
The *uninstall* option does not perform any actions if the distro is set to |CL| and will only uninstall packages if the distro is Ubuntu
Next, we will install the telemetrics backend server with the following options:
* *-a install* to perform an install
* *-d clr* to install to a |CL| distro
* *-H localhost* to set the domain to localhost
We do not need to set the following options since the values are set to the correct values we want by default:
* *-r https://github.com/clearlinux/telemetrics-backend* sets the repo location for :command:`git` to clone from.
* *-s master* to set the location, or branch.
* *-t git* to set the source type to git.
.. caution::
The :file:`deploy.sh` shell script has minimal error checking and makes several changes to your system. Be sure that the options you define on the cmdline are correct before proceeding.
To begin the installation with the options defined:
Run the shell script from the :file:`$HOME/telemetrics-backend/scripts` directory:
.. code-block:: bash
./deploy.sh -H localhost -a install -d clr
The script will start and list all the defined options and prompt you for the :guilabel:`PostgreSQL` database password as shown below:
.. code-block:: console
Options:
host: localhost
distro: clr
action: install
repo: https://github.com/clearlinux/telemetrics-backend
source: master
type: git
DB password: (default: postgres):
For the :guilabel:`DB password:`, press the :kbd:`Enter` key to accept the default password `postgres`.
The :command:`swupd` begins installing the required software bundles to set up the telemetrics backend server. The output will look similar to the following:
.. code-block:: console
swupd-client bundle adder 3.12.7
Copyright (C) 2012-2017 Intel Corporation
Downloading packs...
Extracting application-server pack for version 18740
...5%
Extracting database-basic-dev pack for version 18670
...10%
Extracting database-basic pack for version 18670
...15%
...
Extracting c-basic pack for version 18800
...89%
Extracting os-core-dev pack for version 18800
...94%
Extracting web-server-basic pack for version 18680
...100%
Installing bundle(s) files...
...100%
Calling post-update helper scripts.
Possible filedescriptor leak : 8 (socket:[30833])
Bundle(s) installation done.
.. note::
This script uses :command:`sudo` to run commands and you may be prompted to enter your user password at any time while the script is executing. If this occurs, enter your user password to execute the :command:`sudo` command.
.. code-block:: console
Password:
You may also see an informational message about setting the :envvar:`
https_proxy` environment variable if this variable isn't set.
Once the :command:`swupd` command is complete, the script begins processing
the requirements to install and implement the telemetrics server. Finally,
the script enables the server and provides output that finishes with
something similar to:
.. code-block:: console
.
.
Successfully built alembic Flask-Migrate itsdangerous Mako MarkupSafe python-editor SQLAlchemy uWSGI WTForms
Installing collected packages: SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, click, Werkzeug, Jinja2, itsdangerous, Flask, Flask-SQLAlchemy, Flask-Migrate, WTForms, Flask-WTF, psycopg2, uWSGI
Running setup.py install for psycopg2 ... done
Successfully installed Flask-0.12.2 Flask-Migrate-2.1.0 Flask-SQLAlchemy-2.2 Flask-WTF-0.14.2 Jinja2-2.9.6 Mako-1.0.7 MarkupSafe-1.0 SQLAlchemy-1.1.13 WTForms-2.1 Werkzeug-0.12.2 alembic-0.9.5 click-6.7 itsdangerous-0.24 psycopg2-2.7.3 python-dateutil-2.6.1 python-editor-1.0.3 six-1.10.0 uWSGI-2.0.15
Once all the server components have been installed you are prompted to enter the :guilabel:`PostgreSQL` database password to change it as illustrated below:
.. code-block:: console
Enter password for 'postgres' user:
New password:
Retype new password:
passwd: password updated successfully
Enter `postgres` for the current value of the password and then enter a new
password, retype it to verify the new password and the :guilabel:`PostgreSQL`
database password will be updated.
The script finalizes installation and finishes.
.. code-block:: console
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /usr/lib/systemd/system/postgresql.service.
Cloning into 'telemetrics-backend'...
remote: Counting objects: 344, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 344 (delta 30), reused 50 (delta 20), pack-reused 268
Receiving objects: 100% (344/344), 130.20 KiB | 1.40 MiB/s, done.
Resolving deltas: 100% (177/177), done.
...
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/www/telemetry/venv/bin/python3
Not overwriting existing python script /var/www/telemetry/venv/bin/python (you must use /var/www/telemetry/venv/bin/python3)
Installing setuptools, pip, wheel...done.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 3230c615d6e0, empty message
INFO [alembic.runtime.migration] Running upgrade 3230c615d6e0 -> 466cf2f35d67, empty message
Install complete (installation folder: /var/www/telemetry)
Once the installation is complete you can use your web browser and view the new server by opening the web browser on your system and type in
``localhost`` in the address bar.
You should see a web page similar to the one shown in figure 1:
.. figure:: telemetry-backend/figures/telemetry-backend-1.png
:alt: Telemetry UI
:scale: 50%
Figure 1: :guilabel:`Telemetry UI`
Redirect telemetry records
**************************
Telemetry records generated by the telemetrics clients are sent to the
server location defined in the :file:`/usr/share/defaults/telemetrics/
telemetrics.conf` configuration file. You can customize this setting by
copying this file to :file:`/etc/telemetrics/telemetrics.conf` and changing
the ``server=`` setting to your new server location.
#. Create the :file:`/etc/telemetrics` directory and make it your current
working directory.
.. code-block:: bash
sudo mkdir -p /etc/telemetrics
cd /etc/telemetrics
#. Copy the default :file:`telemetrics.conf` file to the new
:file:`/etc/telemetrics` directory.
.. code-block:: bash
sudo cp /usr/share/defaults/telemetrics/telemetrics.conf
#. Edit the new :file:`/etc/telemetrics/telemetrics.conf` file with your
editor using the :command:`sudo` directive and change the
:guilabel:`server=` setting to ``http://localhost/v2/collector`` and save
this change in the new file.
.. code-block:: console
server=http://localhost/v2/collector
You can also use the fully qualified domain name for your server instead of :guilabel:`localhost`.
#. Restart the telemetry daemons to reload the configuration file.
.. code-block:: bash
telemctl restart
Test the new telemetry backend server
*************************************
|CL| includes a telemetry test probe called :command:`hprobe` that will send
a ``hello`` record to the telemetry backend server. To test that the
telemetry records are now going to your new destination, run the :command:`
hprobe` command to send a ``hello`` record to the server as follows:
.. code-block:: bash
hprobe
The record should show up on your new server console as shown in figure 2:
.. figure:: telemetry-backend/figures/telemetry-backend-2.png
:alt: Telemetry UI
:scale: 50%
Figure 2: :guilabel:`Telemetry UI`
You have now set up the |CL| telemetry backend server, and redirected records from your client to your server.
Creating custom telemetry events
================================
For the following steps, we'll be sending records to the backend server we've just set up. If you prefer to keep records locally and not send them to a server, follow the :ref:`telemetrics` guide and enable :record_retention_enabled: in your :file:`etc/telemetrics/telemetrics.conf` to keep the records locally.
There are two ways to create custom telemetry events: using :command:`telem-record-gen` and using the telemetry API in your applications.
Using telem-record-gen
**********************
Enabling telemetry during installation gives us everything we need to create custom telemetry events, even from C programs, because the telemetry bundle provides a simple pipe-based :abbr:`CLI (Commandline Interface)` program named :file:`telem-record-gen` that can be called trivially:
.. code-block:: bash
~ $ telem-record-gen --help
.. code-block:: console
Usage:
telem-record-gen [OPTIONS] - create and send a custom telemetry record
Help Options:
-h, --help Show help options
Application Options:
-f, --config-file Path to configuration file (not implemented yet)
-V, --version Print the program version
-s, --severity Severity level (1-4) - (default 1)
-c, --class Classification level_1/level_2/level_3
-p, --payload Record body (max size = 8k)
-P, --payload-file File to read payload from
-R, --record-version Version number for format of payload (default 1)
-e, --event-id Event id to use in the record
.. note::
The C library (:file:`libtelemetry.so - man 3 telemetry`) uses the same API parameters and will yield the same effect as :command:`telem-record-gen`.
Let's try generating a simple heartbeat event with
:command:`telem-record-gen`, similar to the hprobe heartbeat probe that |CL|
includes by default.
.. code-block:: bash
~ $ telem-record-gen -c org.clearlinux/hello/world -p "hello there"
We won't see anything happen on the console, but we can track existing and
previous telemetry events with :command:`telemctl`:
.. code-block:: bash
~$ sudo telemctl journal -V -c org.clearlinux/hello/world -i
.. code-block:: console
org.clearlinux/hello/world Tue 2018-11-06 23:00:48 UTC 72e55923fd21c75142c24dcfe0ae0a79 143f2580dcf80267f8f1dfe448f3c975 75f547ff-e55b-44b1-9333-1106098bd448
hello there
Using the telemetry API in your C application
*********************************************
.. note::
More details about the :ref:`the telemetry API <telemetry-api>` are
available in the telemetry guide.
Confirm that the telemetrics header file is located on the system at
:file:`usr/include/telemetry.h` The `latest version`_ of the file can also
be found on github for reference, but installing the `telemetry` bundle will
install the header file that matches your |CL| version.
You will need to include the following headers in your code to use the API:
::
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <telemetry.h>
Use the following code to create the variables we need to hold the data for the record we will be creating:
::
uint32_t severity = 1;
uint32_t payload_version = 1;
char classification[30] = "org.clearlinux/hello/world";
struct telem_ref *tm_handle = NULL;
char *payload;
int ret = 0;
Severity:
| Type: uint32_t
| Value: Severity field value. Accepted values are in the range 1-4, with 1 being the lowest severity, and 4 being the highest severity. Values provided outside of this range are clamped to 1 or 4. [low, med, high, crit]
Payload_version:
| Type: uint32_t
| Value: Payload format version. The only supported value right now is 1, which indicates that the payload is a freely-formatted (unstructured) string. Values greater than 1 are reserved for future use.
Classification:
| Type: char array
| Value: It should have the form, DOMAIN/PROBENAME/REST: DOMAIN is the reverse domain to use as a namespace for the probe (e.g. org.clearlinux); PROBENAME is the name of the probe; and REST is an arbitrary value that the probe should use to classify the record. The maximum length for the classification string is 122 bytes. Each sub-category may be no longer than 40 bytes long. Two / delimiters are required.
Tm_handle:
| Type: Telem_ref struct pointer
| Value: Struct pointer declared by the caller, The struct is initialized if the function returns success.
Payload:
| Type: char pointer
| Value: The payload to set
For this example, we'll set the payload to “hello” by using ``asprintf()``
::
if (asprintf(&payload, "hello\n") < 0) {
exit(EXIT_FAILURE);
}
The functions ``asprintf()`` and ``vasprintf()`` are analogs of ``sprintf(3)`` and ``vsprintf(3)``, except that they allocate a string large enough to hold the output including the terminating null byte ('\0'), and return a pointer to it via the first argument. This pointer should be passed to ``free(3)`` to release the allocated storage when it is no longer needed.
Create the new telemetry record
*******************************
The function ``tm_create_record()`` initializes a telemetry record and sets the severity and classification of that record, as well as the payload version number. The memory needed to store the telemetry record is allocated and should be freed with ``tm_free_record()`` when no longer needed.
::
if ((ret = tm_create_record(&tm_handle, severity, classification, payload_version)) < 0) {
printf("Failed to create record: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
Set the payload field of a telemetrics record
*********************************************
The function ``tm_set_payload()`` attaches the provided telemetry record data to the telemetry record. The current maximum payload size is 8192b.
::
if ((ret = tm_set_payload(tm_handle, payload)) < 0) {
printf("Failed to set record payload: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
free(payload);
The ``free()`` function frees the memory space pointed to by ptr, which must have been returned by a previous call to ``malloc()``, ``calloc()``, or ``realloc()``. Otherwise, or if ``free(ptr)`` has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed.
Send a record to the telemetrics daemon
***************************************
The function ``tm_send_record()`` delivers the record to the local
``telemprobd(1)`` service. Since the telemetry record was allocated by the
program it should be freed with ``tm_free_record()`` when it is no longer
needed.
::
if ((ret = tm_send_record(tm_handle)) < 0) {
printf("Failed to send record to daemon: %s\n", strerror(-ret));
ret = 1;
goto fail;
} else {
printf("Successfully sent record to daemon.\n");
ret = 0;
}
fail:
tm_free_record(tm_handle);
tm_handle = NULL;
return ret;
Full sample application with compiling flags
============================================
Create a new file test.c add the following code.
::
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <telemetry.h>
int main(int argc, char **argv)
{
uint32_t severity = 1;
uint32_t payload_version = 1;
char classification[30] = "org.clearlinux/hello/world";
struct telem_ref *tm_handle = NULL;
char *payload;
int ret = 0;
if (asprintf(&payload, "hello\n") < 0) {
exit(EXIT_FAILURE);
}
if ((ret = tm_create_record(&tm_handle, severity, classification,
payload_version)) < 0) {
printf("Failed to create record: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
if ((ret = tm_set_payload(tm_handle, payload)) < 0) {
printf("Failed to set record payload: %s\n", strerror(-ret));
ret = 1;
goto fail;
}
free(payload);
if ((ret = tm_send_record(tm_handle)) < 0) {
printf("Failed to send record to daemon: %s\n", strerror(-ret));
ret = 1;
goto fail;
} else {
printf("Successfully sent record to daemon.\n");
ret = 0;
}
fail:
tm_free_record(tm_handle);
tm_handle = NULL;
return ret;
}
Compile with the gcc compiler, using this command:
.. code-block:: bash
gcc test.c -ltelemetry -o test_telem
Test to ensure the program is working:
.. code-block:: bash
./test_telem
Successfully sent record to daemon.
Verify record was received
*****************************
To verify that the heartbeat message was received by the telemetry backend
server you can check the telemetry client journal, and specify the
classification as org.clearlinux/hello/world
.. code-block:: bash
sudo telemctl journal -V -c org.clearlinux/hello/world -i
.. code-block:: console
Classification Time stamp Record ID Event ID Boot ID
org.clearlinux/hello/world Tue 2018-11-06 22:58:25 UTC b11db07c58c90d8f496ff963df6c43de 24699c2d60c12d154692875b599ca957 75f547ff-e55b-44b1-9333-1106098bd448
hello
Total records: 1
A full example of the `heartbeat probe`_ in C is documented in the source code. For more information about telemetrics in |CL| refer to the
:ref:`telemetrics` guide.
You can also look for the record on the telemetry backend server.
.. _latest version: https://github.com/clearlinux/telemetrics-client/tree/master/src
.. _heartbeat probe: https://github.com/clearlinux/telemetrics-client/tree/master/src/probes/hello.c
.. _clearlinux/telemetrics-backend: https://github.com/clearlinux/telemetrics-backend
source/clear-linux/tutorials/tutorials.rst
+4 -3
View File
@@ -14,6 +14,7 @@ Explore our tutorials to discover what you can do with |CL|!
.. toctree::
:maxdepth: 1
:glob:
wordpress/wordpress
flatpak/flatpak
@@ -24,12 +25,12 @@ Explore our tutorials to discover what you can do with |CL|!
hadoop
fmv
aws-web/aws-web
telemetry-e2e
telemetry-backend/telemetry-backend
smb/smb
spark
kata
kata_migration
kubernetes
kubernetes/kubernetes*
greengrass
dlrs
yubikey-u2f
nvidia
source/clear-linux/tutorials/yubikey-u2f.rst
+126
View File
@@ -0,0 +1,126 @@
.. _yubikey-u2f:
Enable YubiKey U2F Support
##########################
YubiKey\* is a USB security token manufactured by `Yubico`_. Depending on the
model, a YubiKey can support different authentication protocols including:
One-Time Password (OTP), Smart card, FIDO2, and Universal 2nd Factor (U2F).
These instructions will go over configuring a YubiKey for U2F authentication
through a web browser on a |CL-ATTR| system.
A list of `websites
accepting U2F authentication with the YubiKey`_ is available on the Yubico
website. See the Yubico website to learn more about the Yubikey:
https://www.yubico.com/getstarted/meet-the-yubikey/
.. contents:: :local:
:depth: 1
Prerequisites
*************
This tutorial assumes you have:
#. |CL| installed and running.
#. Mozilla Firefox installed on |CL|.
#. A YubiKey.
Enable Linux udev rules for YubiKey
***********************************
:command:`udev` is the Linux device manager which handles events when USB
devices are added and removed. Custom rules needs to be created to properly
identify the YubiKey and provide applications access.
These instructions are derived from: `Yubico support article Using Your U2F
YubiKey with Linux
<https://support.yubico.com/support/solutions/articles/15000006449>`_
#. Create the udev rules folder under :file:`/etc`
.. code:: bash
sudo mkdir -p /etc/udev/rules.d/
#. Download the u2f rules from the Yubico GitHub:
.. code:: bash
curl -O https://raw.githubusercontent.com/Yubico/libu2f-host/master/70-u2f.rules
#. Move the downloaded :file:`70-u2f.rules` file into the :file:`/etc/udev`
folder
.. code:: bash
sudo mv 70-u2f.rules /etc/udev/rules.d/
#. The udev rules should automatically be reloaded. However, they can be
manually reloaded or reboot the system:
.. code:: bash
sudo udevadm control --reload-rules && udevadm trigger
#. Plugin and validate the YubiKey appears as a USB device:
.. code:: bash
lsusb
Enable U2F in Mozilla Firefox
*****************************
Firefox comes with U2F web authentication support disabled by default. U2F
needs to be enabled in the advanced settings.
These instructions are derived from: `Yubico support article Enabling U2F
support in Mozilla Firefox
<https://support.yubico.com/support/solutions/articles/15000017511-enabling-u2f-support-in-mozilla-firefox>`_
#. Launch Mozilla Firefox
#. In the URL bar, type :command:`about:config` to access the advanced
settings.
.. code:: bash
about:config
#. Click the *I accept the risk!* button to continue to the advanced settings
#. Search for the :command:`security.webauth.u2f` and double-click it
so *Value* becomes **true**.
Your YubiKey is now usable on |CL| with Mozilla Firefox with websites that
support U2F authentication.
Related topics
**************
- |CL| :ref:`security`
.. _`Yubico`: https://www.yubico.com/
.. _`websites accepting U2F authentication with the YubiKey`: https://www.yubico.com/works-with-yubikey/catalog/#protocol=universal-2nd-factor-(u2f)&usecase=all&key=all