diff --git a/.gitignore b/.gitignore index 43790bbd..3316fc85 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,6 @@ source/_build *.swp *~ .*~ + +# ignore VS code settings +.vscode/ diff --git a/source/_themes/clearlinux/header.html b/source/_themes/clearlinux/header.html index a20c36c5..bc58546f 100755 --- a/source/_themes/clearlinux/header.html +++ b/source/_themes/clearlinux/header.html @@ -16,7 +16,7 @@
diff --git a/source/clear-linux/concepts/swupd-about.rst b/source/clear-linux/concepts/swupd-about.rst index ef1d9ee0..99edfa1b 100644 --- a/source/clear-linux/concepts/swupd-about.rst +++ b/source/clear-linux/concepts/swupd-about.rst @@ -1,132 +1,101 @@ .. _swupd-about: -Software update -############### +swupd: software updater +####################### -|CL-ATTR| does software updates differently than traditional Linux-based -operating systems. Where traditional distributions rely on packages for -software deployment, |CL| uses the concept of a "bundle" for -deployment. Traditional Linux packages provide a particular utility or -library; |CL| bundles provide all necessary packages to enable a -specific function. - -With |CL|, updating equates to an entirely new OS version with a -specific set of bundles, as compared to a package-based distribution in -which packages may be updated individually. |CL| updates are -efficient, updating only changed files instead of entire packages. - -System administrators can customize or add bundles to the OS, while still -taking advantage of a controlled update stream. This enables system -administrators to focus on the pieces that make their deployment unique. - - -Bundles -======= - -While we use packages to manage compiling source code into installable -binaries, we do not deploy software through packages. Instead, we use bundles -to deploy software, where each bundle encapsulates a particular functionality --- functionality that is enabled by composing all the required upstream -open-source projects and packages into one logical unit: a bundle. This -simplifies installing features on |CL|. - -For additional resources regarding available bundles, useful bundle commands, -and compatible |CL| kernels, visit our :ref:`bundles-about` -page. +:command:`swupd` is an operating system software manager and update program +that operates at a file-level to enable verifiable integrity and update +efficiency. +Visit the `swupd man page`_ for more details. Versioning ========== -In a traditional distribution, the process of describing current software -versioning usually involves: +Using package managers to keep track of software version compatibility or compare multiple systems on many Linux distributions can be cumbersome. -- Listing and keeping track of the current OS release (generally - uninformative about any singular packages or functionality). +With |CL| :command:`swupd`, versioning happens at the individual +file-level. This means |CL| generates an entirely new OS version with any set +of software changes to the system (including software downgrades or removals). This rolling release versioning model is similar to +:command:`git` internal version tracking, where any of the individual file +commits are tracked and move the pointer forward when changed. -- Keeping track of packages and repositories being used, and updating them - individually. - -- Listing and tracking every package available and installed on the - system, none of which are directly tied to the current OS release. - -This can be done effectively, but given the nearly endless combinations of -packages and versions of packages a server may have, it quickly becomes -non-trivial to define what "version" the system is and what software it -is running without explicitly going through each system and inspecting -every package. - -With |CL|, we need only track: - -- One single number - -A number representing the **current** release of the OS is sufficient to -describe the versions of all the software on the OS. Each build is -composed of a specific set of bundles made from a particular version of -packages. This matters on a daily basis to system administrators, who -need to determine which of their systems do not have the latest security -fixes, or which combinations of software have been tested. Every release -of the same number is guaranteed to contain the same versions of software, -so there's no ambiguity between two systems running the same version of |CL|. +While administrators can pick and choose which `bundles`_ a system has +installed, a single |CL| version number strictly represents one combination +of all software versions that can be installed onto a system of that |CL| +version. This method of whole OS versioning offers unique advantages. +Namely, system administrators can quickly compare multiple |CL| systems that share the same version for important software and security fixes. Updating ======== -Another notable difference between package-based distributions and |CL| -is how updates are managed. On a package-based OS, system administrators update -each individual package or piece of software to a newer (or older!) version. With -|CL|, an update translates to an entirely new OS version, containing one -or many updates. It is not possible to update a piece of the system while -remaining on the same version of |CL|. +|CL| promotes regular and automated updating of software to ensure +integration of new enhancements and security fixes. Refer to :ref:`security` +documentation for more information. -How is this useful? Although it seems, at first, like a huge restriction -or limitation, this method has many non-obvious benefits. Imagine a -cloud environment composed of numerous machines. Here, a homogeneous set of -software makes sense -- from the system administrator's level down to the -user level. Homogeneous systems allow users to focus on their contributions -and/or code, rather than configuring environments or worrying about -synchronizing versions and updates. At the system admin level, it ensures -security is tighter and makes it far easier to monitor and update patches. +Learn how to update your system using :ref:`swupd `. -|CL| promotes regular updating of the OS and will automatically check -for updates and apply them by default. +Update efficiency +----------------- -To learn how to run an update of your system, visit our :ref:`swupd-guide` page. +Because :command:`swupd` operates at the individual file-level instead of a +package-level, |CL| updates are small and fast. +On many Linux\* distributions, updates to a particular software package +require the whole software package to be downloaded and replaced +--even for one line of code. -Update speed -============ +In |CL|, updates are generated using the :ref:`mixer ` tool. Mixer calculates the difference between two |CL| versions and makes available +*binary deltas*, which contain only the changed portion of files. This +*binary delta technology* [1]_ means :command:`swupd` on |CL| systems only +needs to download and apply a small fraction of a package in order to +receive an update. -Software updates with |CL| are also efficient. Bundles simply describe -a set of files, and the update technology updates *only* files that actually -changed by using binary-delta technology for efficiency [1]_. Operating systems -that use packages as the unit of deployment require full package updates (thus -hogging resources), even when one small file in that package has changed. +The :ref:`mixer ` tool additionally computes updates files in +multiple compression formats, allowing :command:`swupd` to utilize the most +efficiently compressed format for a |CL| system to minimize the cost +to update. -It is quite common for a full OS update fixing a security hole to be -only 15 kilobytes in total update size. If only several kilobytes need -to be changed, it does not make sense to re-download and reinstall an -entire package or suite of programs just to incorporate a minuscule (yet -important) update. Through binary deltas, the OS is able to update only -those bits that changed, yielding very small update content (deltas) -that can be applied exceedingly fast. As a result, major security patches -and core update take merely seconds. +Update integrity +---------------- +:command:`swupd` operates against a published manifest of files for a +particular |CL| version that contains the unique hash of each file. This is +the basis of the :command:`swupd verify` subcommand, which allows a |CL| +system to check for and remediate any discrepancies to system files. As +necessary, :command:`swupd verify` provides a useful way for software +developers to return to a known filesystem state. -Customize the OS -================ +Bundles +======= -While we realize our definition of bundles makes sense to us, data center -operators may have special needs and ideas. Therefore, we provide a -:ref:`mixer tool `. This tool allows users to customize and add bundles -or even add their own software, while keeping the operating -system and its updates as the basis. Using this tool, system administrators -can focus on the customization their deployments require while staying on -a controlled update stream. +|CL-ATTR| approaches software management differently than many other +Linux-based operating systems. -To learn more about mixing, visit our :ref:`mixer-about` page. +Instead of deploying granular software packages, |CL| uses the concept of +bundles with pre-associated software. Each bundle encapsulates a particular +use-case, which is enabled by composing all the required upstream open-source +projects and packages into one logical unit. +This bundle-based approach offers some unique advantages: + +* Bundles provide a particular functionality, or stack, which + include all associated runtime dependencies. + +* Software package dependencies are resolved on the server, so file-level + conflicts do not occur on the target system after an update. + +* All combinations of bundles are able to co-exist on a |CL| system. + +For more information on bundles, visit: + +* :ref:`bundles` +* :ref:`bundles-about` +* :ref:`bundle-commands` +* :ref:`compatible-kernels` + +.. [1] The software update technology for |CL-ATTR| was first presented at the Linux Plumbers conference in 2012. + +.. _swupd man page: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst -.. [1] The software update technology for |CL-ATTR| was first presented at the - Linux Plumbers conference in 2012. diff --git a/source/clear-linux/concepts/telemetry-about.rst b/source/clear-linux/concepts/telemetry-about.rst index 11943877..1c371078 100644 --- a/source/clear-linux/concepts/telemetry-about.rst +++ b/source/clear-linux/concepts/telemetry-about.rst @@ -38,7 +38,7 @@ Architecture If you want to capture your own records for analysis, you must set up your own backend server. -.. figure:: figures/telemetry-about-1.png +.. figure:: ../guides/telemetrics/figures/telemetry-e2e.png :scale: 75% :alt: Clear Linux Telemetry Architecture. @@ -48,7 +48,7 @@ The telemetry client provides the front end of a complete telemetrics solution and includes the following components: * **telemprobd**, a daemon that prepares the telemetry records and spools them on disk prior to delivery -* **telempostd**, a daemon that sends the records to the telemetry backend server or leaves them on disk until deleting after the record expires. +* **telempostd**, a daemon that sends the records to the telemetry backend server or leaves them on disk until deleting after the record expires. * **probes**, that collect specific types of data from the operating system. * **libtelemetry**, that telemetrics probes use to create telemetrics records and @@ -76,8 +76,7 @@ Next steps To put this concept into practice, see the following resources: -* :ref:`telemetry-enable` -* :ref:`telemetry-backend` +* :ref:`telem-guide` * `Telemetry feature description`_ .. _`Telemetry feature description`: diff --git a/source/clear-linux/get-started/bare-metal-install-beta/bare-metal-install-beta.rst b/source/clear-linux/get-started/bare-metal-install-beta/bare-metal-install-beta.rst index 0545c003..3c62466f 100644 --- a/source/clear-linux/get-started/bare-metal-install-beta/bare-metal-install-beta.rst +++ b/source/clear-linux/get-started/bare-metal-install-beta/bare-metal-install-beta.rst @@ -5,8 +5,11 @@ Install |CL-ATTR| from the live desktop beta image The live desktop beta image allows you to boot |CL-ATTR| into a GNOME desktop without modifying the host system. Using the live image, you can -explore the possibilities of developing with |CL|. You can also launch the -new installer and install |CL| on your target system. +explore the possibilities of developing with |CL|. + +You can also launch the installer to install |CL| on your target system. +If you proceed with installation, this document assumes you follow the +`Recommended options`_. .. contents:: :local: :depth: 1 @@ -32,9 +35,7 @@ Preliminary steps #. Follow your OS instructions to create a bootable USB drive. - * :ref:`bootable-usb-linux-all` - * :ref:`bootable-usb-mac-all` - * :ref:`bootable-usb-windows-all` + * :ref:`bootable-usb` Install from live image *********************** @@ -86,18 +87,18 @@ Minimum installation requirements ********************************* To fulfill minimum installation requirements, complete the -`Required options`_. `Advanced options`_ are optional. +`Required options`_. We also encourage you to install `Recommended options`_ for a full desktop experience. `Advanced options`_ are optional. .. note:: - * The :kbd:`Install` button is only highlighted **after** you complete the - `Required options`_, and after you enter required values in submenus. + * The :kbd:`Install` button is **only highlighted after** you complete + `Required options`_. - * You must choose whether or not to participate in :ref:`telemetrics` + * You must choose whether or not to participate in :ref:`telem-guide` before you can finish installation. - * You may wish to `Test Network Settings`_ before you - `Configure Network Interfaces`_. Assure that a ``Success`` message is received before installation. + * You may `Test Network Settings`_ before installation + `Configure Network Interfaces`_. Assure a *Success* message appears before installation. Main Menu ********* @@ -242,7 +243,7 @@ Configure Media #. Select :kbd:`Enter` to :kbd:`Confirm`. -#. Choose one partitioning method and continue below: +#. Select one partitioning method and continue: * `Auto Partition`_ * `Add Partition`_ @@ -438,7 +439,7 @@ Telemetry ========= To fulfill the :guilabel:`Required options`, choose whether or not to -participate in `telemetry`_. :ref:`telemetrics` is a |CL| feature that +participate in `telemetry`_. :ref:`telem-guide` is a |CL| feature that reports failures and crashes to the |CL| development team for improvements. For more detailed information, visit our :ref:`telemetry-about` page. @@ -455,6 +456,22 @@ For more detailed information, visit our :ref:`telemetry-about` page. Figure 14: Enable Telemetry +Recommended options +******************* + +After you complete the `Required options`_, we highly recommend completing +a few `Advanced options`_ at minimum: + +* `Bundle Selection`_ Add basic utlities and tools: + + * :file:`desktop-autostart` + * :file:`user-basic` + +* `User Manager`_ Assign a new user with administrative rights +* `Assign Hostname`_ Simplify your development environment + +This document assumes you follow these additional steps. + Skip to finish installation =========================== @@ -462,10 +479,9 @@ After selecting values for all :guilabel:`Required options`, you may skip to `Finish installation`_. Otherwise, continue below. In the Main Menu, select -:guilabel:`Advanced options` to configure network interfaces or proxy -settings, add bundles, add/manage users, add kernel arguments, and more. +:guilabel:`Advanced options` for additional configuration. -Advanced Options +Advanced options **************** Configure Network Interfaces @@ -588,12 +604,20 @@ Bundle Selection #. Select :kbd:`Spacebar` to select the checkbox for each desired bundle. +#. We recommend adding :file:`desktop-autostart` and :file:`user-basic`. + .. figure:: figures/bare-metal-install-beta-19.png :scale: 100% :alt: Bundle Selection Figure 19: Bundle Selection + .. note:: + + The default bundle choices and selections differ between the + :file:`clear-.live-desktop-beta.img` and the + :file:`clear-.installer.img`. + #. Select :kbd:`Confirm` or :kbd:`Cancel`. You are returned to the :guilabel:`Advanced options` menu. @@ -815,23 +839,26 @@ Finish installation #. When you are satisfied with your installation configuration, navigate to :guilabel:`Install` and select :kbd:`Enter`. +#. Select :guilabel:`reboot`. + .. note:: - When installation is finished, a ``reboot`` button appears. - -#. Select ``reboot``. + If you do not perform `Recommended options`_, upon rebooting, a + :file:`login:` prompt will appear. At the prompt, enter `root` and change your password immediately. #. When the system reboots, remove any installation media present. -**Congratulations!** + .. note:: -You have successfully installed |CL| on bare metal using the new installer. + Allow time for the graphical login to appear. This shows the administrative user that you created in `Recommended options`_. + +#. Log in as the adminstrative user. Next steps ********** :ref:`enable-user-space` -.. _Navigate to the image directory: https://download.clearlinux.org/image/ +.. _Navigate to the image directory: https://cdn.download.clearlinux.org/image/ .. _Autoproxy: https://clearlinux.org/features/autoproxy .. _telemetry: https://clearlinux.org/features/telemetry diff --git a/source/clear-linux/get-started/bare-metal-install-beta/figures/bare-metal-install-beta-19.png b/source/clear-linux/get-started/bare-metal-install-beta/figures/bare-metal-install-beta-19.png index 720d06dd..409d3714 100644 Binary files a/source/clear-linux/get-started/bare-metal-install-beta/figures/bare-metal-install-beta-19.png and b/source/clear-linux/get-started/bare-metal-install-beta/figures/bare-metal-install-beta-19.png differ diff --git a/source/clear-linux/get-started/bare-metal-install/bare-metal-install.rst b/source/clear-linux/get-started/bare-metal-install/bare-metal-install.rst index 47cd92c1..26477c80 100644 --- a/source/clear-linux/get-started/bare-metal-install/bare-metal-install.rst +++ b/source/clear-linux/get-started/bare-metal-install/bare-metal-install.rst @@ -34,11 +34,11 @@ Optionally, you can use this command: .. code-block:: bash - curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep "installer") + curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep "installer") #. Follow your OS instructions to create a bootable USB drive. - * :ref:`bootable-usb-beta-all` + * :ref:`bootable-usb` #. After downloading the image, verify and decompress the file per your OS. diff --git a/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-19.png b/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-19.png index 720d06dd..409d3714 100644 Binary files a/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-19.png and b/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-19.png differ diff --git a/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-28.png b/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-28.png index 4e3acf29..f41d8167 100644 Binary files a/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-28.png and b/source/clear-linux/get-started/bare-metal-install/figures/bare-metal-install-beta-28.png differ diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb-beta-all.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb-beta-all.rst deleted file mode 100644 index 5c5d3014..00000000 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb-beta-all.rst +++ /dev/null @@ -1,203 +0,0 @@ -.. _bootable-usb-beta-all: - -Create a bootable USB on your OS -################################ - -Follow these instructions to create a bootable |CL-ATTR| USB drive based on -your OS. - -* :ref:`bootable-usb-linux-all` -* :ref:`bootable-usb-mac-all` -* :ref:`bootable-usb-windows-all` - -Return to :ref:`get-started` - -Requirements: -************* - -* Use a **16GB** or larger USB drive. - -.. _bootable-usb-linux-all: - -Create a bootable USB drive on Linux -************************************ - -.. include:: ../../guides/maintenance/download-verify-decompress-linux.rst - :Start-after: verify-linux: - - -Burn the |CL| image onto a USB drive -==================================== - -.. caution:: - - |CAUTION-BACKUP-USB| - -#. Open a terminal emulator and get root privilege. - - .. code-block:: bash - - sudo -s - -#. Go to the directory with the decompressed image. -#. Plug in the USB drive. -#. Identify the USB drive using the :command:`lsblk` command. This shows all - drives attached to the system, including the primary hard disk. In the - example output below, there are 4 drives - (`/dev/sda`, `/dev/sdb`, `/dev/sdc`, and `/dev/sdd`) attached, where - `/dev/sda` is primary drive in this case. The remaining are 3 USB drives. - The output also shows the mounted partitions (under the `MOUNTPOINT` - column) for each drive. - - .. code-block:: bash - - lsblk - - Example output: - - .. code-block:: console - - NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT - sdd 8:48 1 15G 0 disk - ├─sdd2 8:50 1 5G 0 part /run/media/user1/960c184f-3bb7-42b7-bcaf-0c1282 - ├─sdd3 8:51 1 8G 0 part /run/media/user1/704f3382-b26d-4f34-af1b-cb9aab - └─sdd1 8:49 1 2G 0 part - sdb 8:16 1 14.8G 0 disk - └─sdb1 8:17 1 14.8G 0 part /run/media/user1/PATRIOT_USB - sdc 8:32 1 7.3G 0 disk - └─sdc1 8:33 1 7.3G 0 part /run/media/user1/LINUX MINT - sda 8:0 0 335.4G 0 disk - ├─sda4 8:4 0 28G 0 part - ├─sda2 8:2 0 3.7G 0 part [SWAP] - ├─sda7 8:7 0 6G 0 part /home - ├─sda5 8:5 0 1G 0 part /boot - ├─sda3 8:3 0 954M 0 part /boot/efi - ├─sda1 8:1 0 28G 0 part - ├─sda8 8:8 0 30G 0 part / - └─sda6 8:6 0 7.9G 0 part [SWAP] - -#. Before an image can be burned onto a USB drive, it should be un-mounted. - Some Linux* distros may automatically mount a USB drive when it is plugged - in. To unmount, use the :command:`umount` command followed by the device - identifier/partition. For example: From the above :command:`lsblk` output, - `/dev/sdd` has 2 mounted partitions. To unmount them, enter: - - .. code-block:: bash - - umount /dev/sdd2 - umount /dev/sdd3 - -#. Burn the image onto the USB drive. The command-line example below burns an - uncompressed image onto `/dev/sdd`: - - .. code-block:: bash - - dd if=./clear-[version number]-[image type] of=/dev/sdd bs=4M status=progress - -.. _bootable-usb-mac-all: - -Create a bootable USB drive on macOS* -************************************* - -.. include:: ../../guides/maintenance/download-verify-decompress-mac.rst - :start-after: verify-mac: - - -Burn the |CL| image onto a USB drive -==================================== - -.. caution:: - - |CAUTION-BACKUP-USB| - -#. Launch the Terminal app. -#. Go to the directory with the decompressed image. -#. Plug in a USB drive and get its identifier by entering the command - :command:`diskutil list`. See Figure 1. - - .. code-block:: console - - diskutil list - - .. figure:: figures/bootable-usb-mac-1.png - :scale: 100 % - :alt: Get USB drive identifier - - Figure 1: macOS* - Get USB drive identifier - -#. Unmount the USB drive identified in the previous step. The command-line - example below umounts `/dev/disk2`: - - .. code-block:: console - - diskutil umountDisk /dev/disk2 - -#. Burn the image onto the drive using the :command:`dd` command. The - command-line example below burns an uncompressed image onto `/dev/disk2`: - - .. code-block:: console - - sudo dd if=./clear-[version number]-[image type] of=/dev/rdisk2 bs=4m - - - Adding an ‘r’ in front of the disk identifier should help speed up the - imaging process. - - You can press :kbd:`-T` to check imaging progress. - -#. Eject the USB drive. - - .. code-block:: console - - diskutil eject /dev/disk2 - -.. _bootable-usb-windows-all: - -Create a bootable USB drive on Windows\* -**************************************** - -.. include:: ../../guides/maintenance/download-verify-decompress-windows.rst - :Start-after: verify-windows: - -Burn the |CL| image onto a USB drive -==================================== - -.. caution:: - - |CAUTION-BACKUP-USB| - -#. Download the `Rufus`_ utility to burn the image onto a USB drive. - -#. Plug in the USB drive and open Rufus. - -#. Click the :guilabel:`SELECT` button. See Figure 1. - - .. figure:: figures/bootable-usb-windows-1.png - :scale: 80 % - :alt: Rufus utility - Click the SELECT button - - Figure 1: Rufus utility - Click the SELECT button - -#. Find and select the previously extracted |CL| image file. - Then, click the :guilabel:`Open` button. See Figure 2. - - .. figure:: figures/bootable-usb-windows-2.png - :scale: 80 % - :alt: Rufus utility - Show and select |CL| image file - - Figure 2: Rufus utility - Show and select |CL| image file - -#. Click the :guilabel:`START` button. See Figure 3. - - .. figure:: figures/bootable-usb-windows-3.png - :scale: 80 % - :alt: Rufus utility - Click the START button - - Figure 3: Rufus utility - Click START button - -Return to install from live image -********************************* - -Return to :ref:`get-started` - -.. _Rufus: https://rufus.ie/ diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb-linux.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb-linux.rst deleted file mode 100644 index 5df94cee..00000000 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb-linux.rst +++ /dev/null @@ -1,105 +0,0 @@ -.. _bootable-usb-linux: - -Create a bootable USB drive on Linux\* -###################################### - -Follow these instructions to create a bootable |CL-ATTR| USB drive. -Use an **8GB** or larger USB drive. Download either a live image, -``clear--live.img.xz`` or an installer image, -``clear--installer.img.xz``, from our `image`_ download page. - -Instructions are also available for other operating systems: - -* :ref:`bootable-usb-mac` -* :ref:`bootable-usb-windows` - -.. include:: ../../reference/image-types.rst - :start-after: incl-image-filename: - :end-before: incl-image-filename-end: - -.. include:: ../../guides/maintenance/download-verify-decompress-linux.rst - :Start-after: verify-linux: - -.. _copy-usb-linux: - -Burn the |CL| image onto a USB drive -************************************ - -.. caution:: - - |CAUTION-BACKUP-USB| - -#. Open a terminal emulator and get root privilege. - - .. code-block:: bash - - sudo -s - -#. Go to the directory with the decompressed image. -#. Plug in the USB drive. -#. Identify the USB drive using the :command:`lsblk` command. This shows all - drives attached to the system, including the primary hard disk. In the - example output below, there are 4 drives - (`/dev/sda`, `/dev/sdb`, `/dev/sdc`, and `/dev/sdd`) attached, where - `/dev/sda` is primary drive in this case. The remaining are 3 USB drives. - The output also shows the mounted partitions (under the `MOUNTPOINT` - column) for each drive. - - .. code-block:: bash - - lsblk - - Example output: - - .. code-block:: console - - NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT - sdd 8:48 1 15G 0 disk - ├─sdd2 8:50 1 5G 0 part /run/media/user1/960c184f-3bb7-42b7-bcaf-0c1282 - ├─sdd3 8:51 1 8G 0 part /run/media/user1/704f3382-b26d-4f34-af1b-cb9aab - └─sdd1 8:49 1 2G 0 part - sdb 8:16 1 14.8G 0 disk - └─sdb1 8:17 1 14.8G 0 part /run/media/user1/PATRIOT_USB - sdc 8:32 1 7.3G 0 disk - └─sdc1 8:33 1 7.3G 0 part /run/media/user1/LINUX MINT - sda 8:0 0 335.4G 0 disk - ├─sda4 8:4 0 28G 0 part - ├─sda2 8:2 0 3.7G 0 part [SWAP] - ├─sda7 8:7 0 6G 0 part /home - ├─sda5 8:5 0 1G 0 part /boot - ├─sda3 8:3 0 954M 0 part /boot/efi - ├─sda1 8:1 0 28G 0 part - ├─sda8 8:8 0 30G 0 part / - └─sda6 8:6 0 7.9G 0 part [SWAP] - -#. Before an image can be burned onto a USB drive, it should be un-mounted. - Some Linux distros may automatically mount a USB drive when it is plugged - in. To unmount, use the :command:`umount` command followed by the device - identifier/partition. For example: From the above :command:`lsblk` output, - `/dev/sdd` has 2 mounted partitions. To unmount them, enter: - - .. code-block:: bash - - umount /dev/sdd2 - umount /dev/sdd3 - -#. Burn the image onto the USB drive. The command-line example below burns an - uncompressed image onto `/dev/sdd`: - - .. code-block:: bash - - dd if=./clear-[version number]-[image type] of=/dev/sdd bs=4M status=progress - -.. _usb-next: - -Next steps -********** - -With a bootable |CL| USB drive, you can: - -* :ref:`bare-metal-install` -* :ref:`boot-live-image` -* :ref:`multi-boot` - -.. _image: https://download.clearlinux.org/image -.. _releases: https://download.clearlinux.org/releases diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb-mac.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb-mac.rst deleted file mode 100644 index 3d55c209..00000000 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb-mac.rst +++ /dev/null @@ -1,81 +0,0 @@ -.. _bootable-usb-mac: - -Create a bootable USB drive on macOS -#################################### - -Follow these instructions to create a bootable |CL-ATTR| USB drive. -Use an **8GB** or larger USB drive. Download either a live image, -``clear--live.img.xz`` or an installer image, -``clear--installer.img.xz``, from our `image`_ download page. - -Instructions are also available for other operating systems: - -* :ref:`bootable-usb-linux` -* :ref:`bootable-usb-windows` - -.. include:: ../../reference/image-types.rst - :start-after: incl-image-filename: - :end-before: incl-image-filename-end: - -.. include:: ../../guides/maintenance/download-verify-decompress-mac.rst - :start-after: verify-mac: - - -Burn the |CL| image onto a USB drive -************************************ - -.. caution:: - - |CAUTION-BACKUP-USB| - -#. Launch the Terminal app. -#. Go to the directory with the decompressed image. -#. Plug in a USB drive and get its identifier by entering the command - :command:`diskutil list`. See Figure 1. - - .. code-block:: console - - diskutil list - - .. figure:: figures/bootable-usb-mac-1.png - :scale: 100 % - :alt: Get USB drive identifier - - Figure 1: macOS - Get USB drive identifier - -#. Unmount the USB drive identified in the previous step. The command-line - example below umounts `/dev/disk2`: - - .. code-block:: console - - diskutil umountDisk /dev/disk2 - -#. Burn the image onto the drive using the :command:`dd` command. The - command-line example below burns an uncompressed image onto `/dev/disk2`: - - .. code-block:: console - - sudo dd if=./clear-[version number]-[image type] of=/dev/rdisk2 bs=4m - - - Adding an ‘r’ in front of the disk identifier should help speed up the - imaging process. - - You can press :kbd:`-T` to check imaging progress. - -#. Eject the USB drive. - - .. code-block:: console - - diskutil eject /dev/disk2 - -Next steps -********** - -With a bootable |CL| USB drive, you can: - -* :ref:`bare-metal-install` -* :ref:`boot-live-image` -* :ref:`multi-boot` - -.. _image: https://download.clearlinux.org/image diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb-windows.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb-windows.rst deleted file mode 100644 index 8a6d3dca..00000000 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb-windows.rst +++ /dev/null @@ -1,69 +0,0 @@ -.. _bootable-usb-windows: - -Create a bootable USB drive on Windows\* -######################################## - -Follow these instructions to create a bootable |CL-ATTR| USB drive. -Use an **8GB** or larger USB drive. Download either a live image, -``clear--live.img.xz`` or an installer image, -``clear--installer.img.xz``, from our `image`_ download page. - -Instructions are also available for other operating systems: - -* :ref:`bootable-usb-mac` -* :ref:`bootable-usb-linux` - -.. include:: ../../reference/image-types.rst - :start-after: incl-image-filename: - :end-before: incl-image-filename-end: - -.. include:: ../../guides/maintenance/download-verify-decompress-windows.rst - :Start-after: verify-windows: - -Burn the |CL| image onto a USB drive -************************************ - -.. caution:: - - |CAUTION-BACKUP-USB| - -#. Download the `Rufus`_ utility to burn the image onto a USB drive. - -#. Plug in the USB drive and open Rufus. - -#. Click the :guilabel:`SELECT` button. See Figure 1. - - .. figure:: figures/bootable-usb-windows-1.png - :scale: 80 % - :alt: Rufus utility - Click the SELECT button - - Figure 1: Rufus utility - Click the SELECT button - -#. Find and select the previously extracted |CL| image file. - Then, click the :guilabel:`Open` button. See Figure 2. - - .. figure:: figures/bootable-usb-windows-2.png - :scale: 80 % - :alt: Rufus utility - Show and select |CL| image file - - Figure 2: Rufus utility - Show and select |CL| image file - -#. Click the :guilabel:`START` button. See Figure 3. - - .. figure:: figures/bootable-usb-windows-3.png - :scale: 80 % - :alt: Rufus utility - Click the START button - - Figure 3: Rufus utility - Click START button - -Next steps -********** - -With a bootable |CL| USB drive, you can: - -* :ref:`bare-metal-install` -* :ref:`boot-live-image` -* :ref:`multi-boot` - -.. _Rufus: http://rufus.akeo.ie/ -.. _image: https://download.clearlinux.org/image diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb.rst index f774b69c..3994b20f 100644 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb.rst +++ b/source/clear-linux/get-started/bootable-usb/bootable-usb.rst @@ -1,32 +1,192 @@ .. _bootable-usb: -Create a bootable |CL-ATTR| USB drive -##################################### +Create a bootable USB drive +########################### Instructions to create a |CL-ATTR| USB drive vary depending on your operating -system. +system. Follow the instructions applicable to your system: -.. _download-usb-image: +* :ref:`bootable-usb-linux` +* :ref:`bootable-usb-mac` +* :ref:`bootable-usb-windows` -Download the latest |CL| image -****************************** +Prerequisites +************* -There are 2 types of |CL| images suitable for burning onto and running -off a USB drive: +* Use an **8GB** or larger USB drive. +* `Download`_ the |CL| live boot image or interactive installer image. -* Live image: :file:`clear-[version number]-live.img.xz` -* Installer image: :file:`clear-[version number]-installer.img.xz` +.. _bootable-usb-linux: -Go to the |CL| `image`_ repository and download the desired type. +Create a bootable USB drive on Linux\* +************************************** -With the appropriate image downloaded, choose the step-by-step instructions -applicable to your system: +Make sure you have have completed all `Prerequisites`_. -.. toctree:: - :maxdepth: 1 +Before burning the image onto your USB drive, +:ref:`verify and decompress your image `. - bootable-usb-linux - bootable-usb-windows - bootable-usb-mac +Burn the |CL| image onto a USB drive +==================================== -.. _image: https://download.clearlinux.org/image +.. caution:: + + |CAUTION-BACKUP-USB| + +#. Open a terminal emulator and get root privilege. + + .. code-block:: bash + + sudo -s + +#. Go to the directory with the decompressed image. + +#. Plug in the USB drive. + +#. Identify the USB drive using the :command:`lsblk` command. This shows all + drives attached to the system, including the primary hard disk. In the + example output below, there are 4 drives + (`/dev/sda`, `/dev/sdb`, `/dev/sdc`, and `/dev/sdd`) attached, where + `/dev/sda` is primary drive. The remaining are three USB drives. The output + also shows the mounted partitions (under the `MOUNTPOINT` column) for each + drive. + + .. code-block:: bash + + lsblk + + Example output: + + .. code-block:: console + + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT + sdd 8:48 1 15G 0 disk + ├─sdd2 8:50 1 5G 0 part /run/media/user1/960c184f-3bb7-42b7-bcaf-0c1282 + ├─sdd3 8:51 1 8G 0 part /run/media/user1/704f3382-b26d-4f34-af1b-cb9aab + └─sdd1 8:49 1 2G 0 part + sdb 8:16 1 14.8G 0 disk + └─sdb1 8:17 1 14.8G 0 part /run/media/user1/PATRIOT_USB + sdc 8:32 1 7.3G 0 disk + └─sdc1 8:33 1 7.3G 0 part /run/media/user1/LINUX MINT + sda 8:0 0 335.4G 0 disk + ├─sda4 8:4 0 28G 0 part + ├─sda2 8:2 0 3.7G 0 part [SWAP] + ├─sda7 8:7 0 6G 0 part /home + ├─sda5 8:5 0 1G 0 part /boot + ├─sda3 8:3 0 954M 0 part /boot/efi + ├─sda1 8:1 0 28G 0 part + ├─sda8 8:8 0 30G 0 part / + └─sda6 8:6 0 7.9G 0 part [SWAP] + +#. You must unmount a USB drive before you can burn an image onto it. Note that + some Linux distros automatically mount a USB drive when it is plugged in. + Unmount a USB drive with the :command:`umount` command followed by the device + identifier/partition. For example: + + .. code-block:: bash + + umount /dev/sdd2 + umount /dev/sdd3 + +#. Burn the image onto the USB drive. The example below burns an uncompressed + image onto ``: + + .. code-block:: bash + + dd if=./clear-[version number]-[image type] of= bs=4M status=progress && sync + +.. _bootable-usb-mac: + +Create a bootable USB drive on macOS\* +************************************** + +Make sure you have have completed all `Prerequisites`_. + +Before burning the image onto your USB drive, +:ref:`verify and decompress your image `. + +Burn the |CL| image onto a USB drive +==================================== + +.. caution:: + + |CAUTION-BACKUP-USB| + +#. Launch the Terminal app. + +#. Go to the directory with the decompressed image. + +#. Plug in a USB drive and get its identifier: + + .. code-block:: bash + + diskutil list + + This will list available disks and their partitions, as shown in Figure 1. + + .. figure:: figures/bootable-usb-mac-1.png + :scale: 100 % + :alt: Get USB drive identifier + + Figure 1: macOS - Get USB drive identifier + +#. Unmount the USB drive identified in the previous step. For example: + + .. code-block:: bash + + diskutil umountDisk /dev/disk2 + +#. Burn the image onto the drive using the :command:`dd` command. The example + below burns an uncompressed image onto ``: + + .. code-block:: bash + + sudo dd if=./clear-[version number]-[image type] of= bs=4m + + To speed up the imaging process, add an ‘r’ in front of the disk identifier. + For example `/dev/rdisk2`. + + Press :kbd:`-T` to check imaging progress. + +#. Eject the USB drive. + + .. code-block:: bash + + diskutil eject /dev/disk2 + +.. _bootable-usb-windows: + +Create a bootable USB drive on Windows\* +**************************************** + +Make sure you have have completed all `Prerequisites`_. + +Before burning the image onto your USB drive, +:ref:`verify and decompress your image `. + +Burn the |CL| image onto a USB drive +==================================== + +.. caution:: + + |CAUTION-BACKUP-USB| + +#. Download the `Rufus`_ utility to burn the image onto a USB drive. + +#. Plug in the USB drive and open Rufus. + +#. Under `Boot selection`, click the :guilabel:`SELECT` button. + +#. Find and select the previously extracted |CL| image file. + +#. Click the :guilabel:`START` button. See Figure 2. + + .. figure:: figures/bootable-usb-windows-3.png + :scale: 80 % + :alt: Rufus utility + + Figure 2: Rufus utility + + +.. _Rufus: https://rufus.ie/ +.. _Download: https://clearlinux.org/downloads diff --git a/source/clear-linux/get-started/compatibility-check.rst b/source/clear-linux/get-started/compatibility-check.rst index 76b795b1..b5fa8a27 100644 --- a/source/clear-linux/get-started/compatibility-check.rst +++ b/source/clear-linux/get-started/compatibility-check.rst @@ -19,7 +19,7 @@ below. .. code-block:: console - curl -O https://download.clearlinux.org/current/clear-linux-check-config.sh + curl -O https://cdn.download.clearlinux.org/current/clear-linux-check-config.sh #. Make the script executable. @@ -58,4 +58,4 @@ below. SUCCESS: Carry-less Multiplication extensions (pclmulqdq) SUCCESS: EFI Firmware -.. _clear-linux-check-config.sh: https://download.clearlinux.org/current/clear-linux-check-config.sh +.. _clear-linux-check-config.sh: https://cdn.download.clearlinux.org/current/clear-linux-check-config.sh diff --git a/source/clear-linux/get-started/get-started.rst b/source/clear-linux/get-started/get-started.rst index 15ebfb9b..6b0b2148 100644 --- a/source/clear-linux/get-started/get-started.rst +++ b/source/clear-linux/get-started/get-started.rst @@ -17,7 +17,6 @@ Pre-install compatibility-check bootable-usb/bootable-usb - bootable-usb/bootable-usb-beta-all Install |CL| ************ diff --git a/source/clear-linux/get-started/live-image.rst b/source/clear-linux/get-started/live-image.rst index 05285773..685b7e1e 100644 --- a/source/clear-linux/get-started/live-image.rst +++ b/source/clear-linux/get-started/live-image.rst @@ -29,7 +29,7 @@ Boot the |CL| live image .. _create and enable user space: https://clearlinux.org/documentation/clear-linux/guides/maintenance/enable-user-space -.. _`image`: https://download.clearlinux.org/image +.. _`image`: https://cdn.download.clearlinux.org/image .. _`Intel® Virtualization Technology (Intel® VT)`: http://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/7zipwin.png b/source/clear-linux/get-started/virtual-machine-install/figures/7zipwin.png deleted file mode 100644 index 574a82b1..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/7zipwin.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-cd.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-cd.png deleted file mode 100644 index a1978856..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-cd.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-convert-image.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-convert-image.png deleted file mode 100644 index f1e64ea1..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-convert-image.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-create-vm.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-create-vm.png deleted file mode 100644 index 307b45f4..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-create-vm.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-efi.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-efi.png deleted file mode 100644 index eb33bb79..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-efi.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-hdisk.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-hdisk.png deleted file mode 100644 index 839fc9d0..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-hdisk.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-memory-size.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-memory-size.png deleted file mode 100644 index d8c5b71c..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-memory-size.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-x.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox-x.png deleted file mode 100644 index 8357efa1..00000000 Binary files a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-x.png and /dev/null differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-first-login.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-first-login.png new file mode 100644 index 00000000..5e8ed691 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-first-login.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-remove-non-lts-kernels.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-remove-non-lts-kernels.png new file mode 100644 index 00000000..2a253748 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-cl-remove-non-lts-kernels.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-convert-raw-to-VDI.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-convert-raw-to-VDI.png new file mode 100644 index 00000000..3c20c833 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-convert-raw-to-VDI.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-disk.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-disk.png new file mode 100644 index 00000000..7995d8d7 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-disk.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-choose-disk.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-choose-disk.png new file mode 100644 index 00000000..c672ac9b Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-choose-disk.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-existing-disk.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-existing-disk.png new file mode 100644 index 00000000..d8b4846a Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-existing-disk.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-new-disk.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-new-disk.png new file mode 100644 index 00000000..7c2d1cc6 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-new-disk.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-no-disk.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-no-disk.png new file mode 100644 index 00000000..68f9dddf Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-create-vm-no-disk.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-IMG.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-IMG.png new file mode 100644 index 00000000..6aeb25e1 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-IMG.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-ISO.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-ISO.png new file mode 100644 index 00000000..c9b8932f Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-extract-cl-ISO.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-new-vm.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-new-vm.png new file mode 100644 index 00000000..ae8929bb Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-new-vm.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox-no-vtx.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-no-vtx.png similarity index 100% rename from source/clear-linux/get-started/virtual-machine-install/figures/vbox-no-vtx.png rename to source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-no-vtx.png diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-shutdown-vm.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-shutdown-vm.png new file mode 100644 index 00000000..11ef90fe Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-shutdown-vm.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-start-vm.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-start-vm.png new file mode 100644 index 00000000..76c012ec Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-start-vm.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-created.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-created.png new file mode 100644 index 00000000..1e23ccbd Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-created.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-insert-ga-cd.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-insert-ga-cd.png new file mode 100644 index 00000000..4eae68ce Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-insert-ga-cd.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-EFI.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-EFI.png new file mode 100644 index 00000000..2dc55dff Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-EFI.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-browse-ISO.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-browse-ISO.png new file mode 100644 index 00000000..ed229d3f Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-browse-ISO.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-mount-ISO.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-mount-ISO.png new file mode 100644 index 00000000..765df5ca Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-mount-ISO.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-unmount-ISO.png b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-unmount-ISO.png new file mode 100644 index 00000000..4e08843f Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vbox/vbox-vm-settings-unmount-ISO.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/hyper-v.rst b/source/clear-linux/get-started/virtual-machine-install/hyper-v.rst index 201df498..948d5746 100644 --- a/source/clear-linux/get-started/virtual-machine-install/hyper-v.rst +++ b/source/clear-linux/get-started/virtual-machine-install/hyper-v.rst @@ -57,4 +57,4 @@ Your virtual machine running |CL| is ready! .. _Windows Server Virtualization: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/ .. _Microsoft documentation: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v .. _Create A Virtual Network documentation: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/connect-to-network -.. _downloads: https://download.clearlinux.org/image/ +.. _downloads: https://cdn.download.clearlinux.org/image/ diff --git a/source/clear-linux/get-started/virtual-machine-install/kvm.rst b/source/clear-linux/get-started/virtual-machine-install/kvm.rst index 5f8cd200..62dbd248 100644 --- a/source/clear-linux/get-started/virtual-machine-install/kvm.rst +++ b/source/clear-linux/get-started/virtual-machine-install/kvm.rst @@ -50,12 +50,12 @@ Download and launch the virtual machine *************************************** #. Download the latest pre-built |CL| KVM image file from - the `image `_ directory. Look for + the `image `_ directory. Look for ``clear--kvm.img.xz``. You can also use this command: .. code-block:: bash - curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm) + curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm) #. Uncompress the downloaded image: @@ -64,7 +64,7 @@ Download and launch the virtual machine unxz clear--kvm.img.xz #. Download the `OVMF file`_ file that provides UEFI support for - virtual machines from the `image `_ directory. + virtual machines from the `image `_ directory. #. Copy :file:`OVMF.fd` to the working directory, as shown below. @@ -79,9 +79,14 @@ Download and launch the virtual machine For non-Clear Linux hosts, the preferred approach is to download it from https://cdn.download.clearlinux.org/image/OVMF.fd #. Download the sample `QEMU-KVM launcher`_ script from the - `image `_ directory. This script + `image `_ directory. This script will launch the |CL| VM and provide console interaction within the same terminal emulator window. + + .. code-block:: bash + + curl -O https://cdn.download.clearlinux.org/image/start_qemu.sh + #. Make the script executable: @@ -103,7 +108,7 @@ SSH access into the virtual machine To interact with the |CL| VM through SSH instead of the console it was launched from, follow these steps. -#. Enable SSH in the |CL| VM: +#. Configure SSH in the |CL| VM to allow root login: .. code-block:: bash @@ -111,6 +116,12 @@ launched from, follow these steps. PermitRootLogin yes EOF +#. Start SSH server in the |CL| VM: + + .. code-block:: bash + + systemctl start sshd + #. From the host, SSH into the |CL| VM. The port number ``10022`` is defined in the ``start_qemu.sh`` script. @@ -135,7 +146,7 @@ To add :abbr:`GDM (GNOME Display Manager)` to the |CL| VM, follow these steps: .. code-block:: bash - swupd bundle-add desktop-apps-extras + swupd bundle-add tigervnc * On Ubuntu\* 16.04 LTS Desktop: @@ -232,5 +243,5 @@ To add :abbr:`GDM (GNOME Display Manager)` to the |CL| VM, follow these steps: .. _Intel® Virtualization Technology: https://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html .. _Intel®Virtualization Technology for Directed I/O: https://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices -.. _QEMU-KVM launcher: https://download.clearlinux.org/image/start_qemu.sh -.. _OVMF file: https://download.clearlinux.org/image/OVMF.fd +.. _QEMU-KVM launcher: https://cdn.download.clearlinux.org/image/start_qemu.sh +.. _OVMF file: https://cdn.download.clearlinux.org/image/OVMF.fd diff --git a/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst b/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst index 34f93ac9..c257550e 100644 --- a/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst +++ b/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst @@ -35,5 +35,6 @@ appropriate set of step-by-step instructions to proceed. vmw-player vmw-player-preconf hyper-v + virtualbox-cl-installer ../../guides/maintenance/increase-virtual-disk-size.rst diff --git a/source/clear-linux/get-started/virtual-machine-install/virtualbox-cl-installer.rst b/source/clear-linux/get-started/virtual-machine-install/virtualbox-cl-installer.rst new file mode 100644 index 00000000..4a574a36 --- /dev/null +++ b/source/clear-linux/get-started/virtual-machine-install/virtualbox-cl-installer.rst @@ -0,0 +1,277 @@ +.. _virtualbox-cl-installer: + +Install |CL-ATTR| as a VirtualBox\* Virtual Machine +################################################### + +VirtualBox\* is a type 2 hypervisor from Oracle. This document explains how +to create a virtual machine on the `VirtualBox hypervisor`_ with |CL-ATTR| +as the guest operating system. + +These instructions make use of the |CL| installer to create a brand new |CL| +installation. A preinstalled disk image is also available to get started +with |CL| faster. See: :ref:`virtualbox` + + +.. contents:: :local: + :depth: 2 + + +.. include:: virtualbox.rst + :start-after: vbox-prereqs-begin: + :end-before: vbox-prereqs-end: + + + +Download and extract the |CL| installer ISO image +************************************************* + +The appropriate |CL| installer image needs to be downloaded and extracted. + +.. note:: + The :file:`installer.iso` is for limited use on special cases where ISO + image format is required, such as |VB|. The preferred installer for |CL| + for UEFI systems is the :file:`-installer.img`. + +#. Download the **installer ISO** image (:file:`clear--installer.iso.xz`) of + |CL|. On the `downloads page`_, this is listed as + **Clear Linux OS for Virtual Provisioning**. + + + You can also use this command to download from a terminal: + + .. code-block:: bash + + curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep installer.iso) + +#. Validate the integrity of the downloaded image by checking the file hash + and signatures. Refer to the document on :ref:`validate-signatures` for + detailed steps. + +#. Decompress the downloaded image. Uncompressed image size is ~ **5GB**. + + - On Windows you can use `7zip`_ to extract the file by right-clicking the + file to *Extract Here* (in the same directory) + + .. image:: ./figures/vbox/vbox-extract-cl-ISO.png + :alt: 7zip extract here command + + - On Linux : + + .. code-block:: bash + + xz -d clear--installer.iso.xz + + + + +#. The originally downloaded compressed archive file + (:file:`clear--installer.iso.xz`) can now be deleted. + + +Create a new |VB| virtual machine +********************************* + +A new :abbr:`VM (Virtual Machine)` needs to be created in |VBM| for |CL| to be installed onto. + +General instructions for creating a virtual machine and details about using +different settings are available on the +`VirtualBox manual section on Creating a VM`_. + + +#. Launch the |VBM| from your host system. + + +#. Click the *New* button to create a new VM. + + .. image:: ./figures/vbox/vbox-new-vm.png + :alt: Create a new VM in VirtualBox + + +#. A *Create Virtual Machine* window will appear. + Select the following settings: + + - Type: **Linux** + - Version: **Linux 2.6 / 3.x / 4.x (64-bit)** + - Memory size: **1024 MB** (this can be adjusted appropriately) + - Hard disk: **Create a virtual hard disk now** + + .. image:: ./figures/vbox/vbox-create-vm-new-disk.png + :alt: Create a new VM in VirtualBox with a new disk + + +#. Click the *Create* button. + +#. A *Create Virtual Hard Disk* window will appear. + Select the following settings: + + - File size: **8.00 GB** (this can be adjusted appropriately) + + - Hard disk file type: **VDI (Virtual Box Disk Image)** + + - Storage on physical hard disk: **Dynamically allocated** + + .. image:: ./figures/vbox/vbox-create-disk.png + :alt: Create a new virtual hard disk in VirtualBox + +#. Click the *Create* button. + +#. A new virtual machine will be created and appear in the |VBM|. Click + *Settings* to configure the |CL| VM. + + .. image:: ./figures/vbox/vbox-vm-created.png + :alt: A VM selected in VirtualBox Manager + +#. A *VM - Settings* window will appear. Navigate to the *System* pane from + the left-hand and select the following setting: + + - **Enable I/O APIC** + - **Enable EFI (special OSes only)** + + + .. image:: ./figures/vbox/vbox-vm-settings-EFI.png + :alt: Enable EFI on a VirtualBox VM settings + +.. note:: + By default, only 1 virtual CPU is allocated to the new VM. Consider + increasing the number of virtual processors allocated to the virtual + machine under Settings --> System --> Processor for increased + performance. + + +Install |CL| on the |VB| VM +*************************** + +|CL| is ready to be installed. + + +Mount the installation ISO +========================== +At this point, the newly created VM has a blank virtual hard disk with no +operating system.The |CL| installer ISO needs to be mounted as a virtual +CD-ROM on the VM before powering the VM on. + +#. From the *VM - Settings* window, navigate to the *Storage* pane from the + left-hand side. + +#. From the middle *Storage Devices* column, click the blue CD disk labeled + *Empty* under the *Controller: IDE* from. + +#. From the right-hand *Attributes* column, click the blue CD disk next to + the *Optical Drive* drop down menu and click *Choose Virtual Optical + Disk File...* + + .. image:: ./figures/vbox/vbox-vm-settings-mount-ISO.png + :alt: Mounting an ISO in VirtualBox VM Settings + +#. A *choose a virtual optical disk file* browser window will appear. + Navigate to the extracted ISO file, select it, and click *Open*. + + .. image:: ./figures/vbox/vbox-vm-settings-browse-ISO.png + :alt: Mounting an ISO in VirtualBox VM Settings + +#. Click *OK* to exit the *VM Settings* menu and return to the main + |VBM|. + + + +Install |CL| using the installer +================================ + +#. Start the VM from the |VBM| by selecting the |CL| VM and + clicking *Start* + + .. image:: ./figures/vbox/vbox-start-vm.png + :alt: Starting a VirtualBox VM + +#. A new window of the VM console will appear and boot into the |CL| + installer. Follow the steps in the `Install Clear Linux OS`_ to + install |CL| onto the VM virtual disk. + +.. note:: + Do not choose a different kernel from the installer. **kernel-lts**, the + Long Term Support (LTS) kernel is required for |VB| driver compatibility. + + +#. After |CL| installation is complete, the VM will reboot and return to the + |CL| installer. + +.. note:: + To release the mouse cursor from the VM console window, press the right Ctrl key on the keyboard. + + +Unmount the installation ISO +============================ + +The |CL| installer ISO needs to be unmounted to allow the VM to boot from the +virtual hard disk, which |CL| has been installed to. + + +#. Power off the |CL| VM. + + .. image:: ./figures/vbox/vbox-shutdown-vm.png + :alt: Powering off a VirtualBox VM + +#. Click *Settings* to configure the |CL| VM. + + .. image:: ./figures/vbox/vbox-vm-created.png + :alt: A VM selected in VirtualBox Manager + +#. From the *VM - Settings* window, navigate to the *Storage* pane from the + left-hand side. + +#. From the middle *Storage Devices* column, click the blue CD disk labeled + *clear--installer.iso* under the *Controller: IDE* from. + +#. From the right-hand *Attributes* column, click the blue CD disk next to + the *Optical Drive* drop down menu and click *Remove Disk from Virtual + Drive* + + .. image:: ./figures/vbox/vbox-vm-settings-unmount-ISO.png + :alt: Unmounting an ISO in VirtualBox VM Settings + +#. Click *OK* to exit the *VM Settings* menu and return to the main + |VBM|. + + + + +.. include:: virtualbox.rst + :start-after: vbox-start-vm-and-lga-begin: + :end-before: vbox-start-vm-and-lga-end: + + + + + +.. include:: virtualbox.rst + :start-after: vbox-troubleshooting-begin: + :end-before: vbox-troubleshooting-end + + + + +.. |VB| replace:: VirtualBox +.. |VBM| replace:: VirtualBox Manager + + +.. _appropriate instructions: https://www.virtualbox.org/manual/ch02.html + +.. _official VirtualBox website: https://www.virtualbox.org/wiki/Downloads + +.. _VirtualBox hypervisor: https://www.virtualbox.org/ + +.. _downloads page: https://clearlinux.org/downloads + +.. _`VirtualBox manual section on Creating a VM`: https://www.virtualbox.org/manual/UserManual.html#gui-createvm + +.. _`VirtualBox manual section on Running a VM`: https://www.virtualbox.org/manual/ch01.html#intro-starting-vm-first-time + +.. _`Virtual Media Manager`: https://www.virtualbox.org/manual/ch05.html#vdis + +.. _`Install Clear Linux OS`: https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install#install-cl-on-your-target-system\ + +.. _7zip: http://www.7-zip.org/ + +.. _Virtualization Technology: https://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html + +.. _`VirtualBox manual section on VBoxManage`: https://www.virtualbox.org/manual/ch08.html#vboxmanage-convertfromraw \ No newline at end of file diff --git a/source/clear-linux/get-started/virtual-machine-install/virtualbox.rst b/source/clear-linux/get-started/virtual-machine-install/virtualbox.rst index a526f7a8..bebb3791 100644 --- a/source/clear-linux/get-started/virtual-machine-install/virtualbox.rst +++ b/source/clear-linux/get-started/virtual-machine-install/virtualbox.rst @@ -1,196 +1,370 @@ .. _virtualbox: -Run pre-configured |CL-ATTR| as a VirtualBox\* guest OS -########################################################### +Run pre-configured |CL-ATTR| as a VirtualBox\* Virtual Machine +############################################################## -This instruction explains how to deploy a pre-configured |CL-ATTR| image as a guest on the `VirtualBox hypervisor`_ . +VirtualBox\* is a type 2 hypervisor from Oracle. This document explains how +to create a virtual machine on the `VirtualBox hypervisor`_ with |CL-ATTR| +as the guest operating system. -Download VirtualBox -******************* +These instructions make use of a preinstalled |CL| disk image to setup a |CL| +virtual machine without manual installation. |CL| can also be installed from +scratch on a |VB| using the |CL| installed. +See: :ref:`virtualbox-cl-installer` -VirtualBox\* is a type 2 hypervisor from Oracle. Download and use **version 5.0 or greater** from the `official VirtualBox website`_. +.. contents:: :local: + :depth: 2 -.. _create_vm_vbox: + + +.. _vbox-prereqs-begin: Prerequisites ************* -The instruction assumes that you have: +Before continuing make sure that you have: -#. Enabled virtualization technology in the host machine's BIOS. +#. Enabled virtualization, such as Intel® + `Virtualization Technology`_ (Intel® VT), on the host system from + EFI/BIOS. + +#. Downloaded and installed |VB| **version 6.0 or greater** from + the `official VirtualBox website`_ per the `appropriate instructions`_ + for your platform. + +.. _vbox-prereqs-end: + + + +Download and extract |CL| +************************* + +The |CL| live image needs to be downloaded and extracted. The live image will +be used to created a |VB| virtual disk image that can be used with a +:abbr:`VM (Virtual Machine)`. + +#. Download the **live image** (:file:`clear--live.img.xz - vbox-img convert --srcfilename clear-XXXX-live.img --dstfilename clear-XXXX-live.vdi --srcformat raw --dstformat vdi +#. There originally downloaded compressed archive file + (:file:`clear--live.img.xz`) can now be deleted. - .. note:: Be sure you have VirtualBox directory in your PATH (i.e., on - Windows :file:`C:\\Program Files\\Oracle\\VirtualBox`). - + On windows: launch a **Command Prompt** program and type +Convert |CL| live image to a |VB| Disk Image +******************************************** - .. code-block:: console +The |CL| live image is in a RAW disk image. The live image needs to be +converted to a :abbr:`VDI (VirtualBox Disk Image)` format which |VB| +can utilize. - set PATH=%PATH%;"C:\Program Files\Oracle\VirtualBox" +#. Launch a terminal and navigate to the directory containing the + extracted live image. - .. image:: ./figures/vbox-convert-image.png - :alt: Convert image in Windows command propt -#. Create a virtual machine using the VirtualBox assistant: +#. Convert RAW live image to a :abbr:`VDI (VirtualBox Disk Image)` + format using the command-line VirtualBox Disk Utility. - a. Type: **Linux** + .. code-block:: bash + + VBoxManage convertfromraw clear--live.img clear-VM.vdi --format VDI + + .. note:: + The :command:`PATH` environment variable may need to be updated to make the + :command:`VBoxManage` command easily accessible from the terminal. + For example, using Windows PowerShell: + + .. code-block:: bash + + $env:PATH += ";C:\Program Files\Oracle\VirtualBox" + + + .. image:: ./figures/vbox/vbox-convert-raw-to-VDI.png + :alt: Convert image in Windows command prompt + + For more information on the :command:`VBoxManage` command, + see the `VirtualBox manual section on VBoxManage`_. + + +#. The originally extracted live image file + (:file:`clear--live.img`) can now be deleted. + + +#. Move the converted :file:`clear-VM.vdi` disk image file to a permanent + location. The VDI will be attached to the |VB| VM and should not be + deleted. + + + +Create a new |VB| virtual machine +********************************* + +A new VM needs to be created in |VBM| to attach the VDI with |CL| installed. + +General instructions for creating a virtual machine and details about using +different settings are available on the +`VirtualBox manual section on Creating a VM`_. + + +#. Launch the |VBM| from your host system. + + +#. Click the *New* button to create a new VM. + + .. image:: ./figures/vbox/vbox-new-vm.png + :alt: Create a new VM in VirtualBox + + +#. A *Create Virtual Machine* window will appear. + Select the following settings: - b. Version: **Linux 2.6 / 3.x / 4.x (64-bit)** + - Type: **Linux** + - Version: **Linux 2.6 / 3.x / 4.x (64-bit)** + - Memory size: **1024 MB** (this can be adjusted appropriately) + - Hard disk: **Use an existing virtual hard disk file** - .. image:: ./figures/vbox-create-vm.png - :alt: Create a new image in VirtualBox + Click the folder icon next to the drop down menu: - c. Select default memory size. - - .. image:: ./figures/vbox-memory-size.png - - d. Attach the virtual disk created in step number 3 as a virtual hard - disk file. Click the folder icon (lower right) to browse to find the - VDI file. - - .. image:: ./figures/vbox-hdisk.png - -#. After it is created, go to settings to enable **EFI support** - - * System -> Enable EFI (special OSes only) - - .. image:: ./figures/vbox-efi.png - :alt: Enable EFI on VirtualBox + .. image:: ./figures/vbox/vbox-create-vm-existing-disk.png + :alt: Create a new VM in VirtualBox with an existing disk -Run your new VM -*************** +#. A new window will appear for choosing an existing disk. Click the *Add* + button, browse to the saved VDI file, and click *Choose*. -|CL| supports VirtualBox kernel modules used -by the Linux kernel 4.14 :abbr:`LTS (Long Term Support)` -(*kernel-lts bundle*).This kernel was selected because |CL| OS's main kernel -(``kernel-native``) bundle keeps up-to-date with the upstream Linux kernel, -and sometimes VirtualBox kernel modules aren't compatible with pre-kernel -releases. + .. image:: ./figures/vbox/vbox-create-vm-choose-disk.png + :alt: Create a new VM in VirtualBox with an existing disk -On the first boot, |CL| requests a user login. - -#. Type **root**. - -#. Enter a new password when prompted. - -To install the VirtualBox kernel modules, here are the steps: - -#. Install the bundle that supports VirtualBox modules:: - - swupd bundle-add kernel-lts - -#. Set a timeout in the bootmanager to shows a menu at boot time:: - - clr-boot-manager set-timeout 10 - -#. Update the bootloader entries with:: - - clr-boot-manager update - -#. Reboot your system with:: - - reboot - - and choose **clear-linux-lts-4.14.XX-YYY** kernel version. - -#. (*Optional*) Unset timeout to boot directly to LTS version:: - - clr-boot-manager set-timeout 0 - -#. (*Mandatory*) Update bootmanger to always use LTS version:: - - clr-boot-manager update - -Install Guest Additions -======================= - -The kernel modules are shipped with the ``kernel-lts`` bundle. Insert Guest -Additions CD image using *Devices* menu you'll need to install the *user* -Linux Guest Additions. To install the VirtualBox Guest Additions, -follow these steps: +#. Click the *Create* button. -#. Insert Guest Additions CD image using *Devices* menu +#. A new virtual machine will be created and appear in the |VBM|. Click + *Settings* to configure the |CL| VM. - .. image:: ./figures/vbox-cd.png + .. image:: ./figures/vbox/vbox-vm-created.png + :alt: A VM selected in VirtualBox Manager + +#. A *VM - Settings* window will appear. Navigate to the *System* pane from + the left-hand and select the following setting: + + - **Enable I/O APIC** + - **Enable EFI (special OSes only)** + + + .. image:: ./figures/vbox/vbox-vm-settings-EFI.png + :alt: Enable EFI on a VirtualBox VM settings + + + +.. note:: + By default, only 1 virtual CPU is allocated to the new VM. Consider + increasing the number of virtual processors allocated to the virtual + machine under Settings --> System --> Processor for increased + performance. + +.. _vbox-start-vm-and-lga-begin: + +Start the |CL| VM +***************** + +The |CL| VM can now be powered on and setup. + +General instructions for using a |VB| virtual machine are available on the +`VirtualBox manual section on Running a VM`_. + +#. Start the VM from the |VBM| by selecting the |CL| VM and clicking *Start* + + .. image:: ./figures/vbox/vbox-start-vm.png + :alt: Starting a VirtualBox VM + +#. |CL| will boot and prompt for login. + + - Enter **root** for the username. + +#. You will be immediately prompted to set a new password for the **root** + user. Reference :ref:`security` for more information about |CL| security + concepts. + + .. image:: ./figures/vbox/vbox-cl-first-login.png + :alt: Initial login to Clear Linux OS on a VirtualBox VM + + + +Install |VB| Linux Guest Additions +================================== + +The |VB| Linux Guest Additions provide drivers for full compatibility and +functionality. + +|CL| provides |VB| guest drivers and an install script in the **kernel-lts** +(Long Term Support) bundle by |CL|. + + +#. Validate the installed kernel is **kernel-lts** by checking the output + of the :command:`uname -r` command. It should end in **.lts**. + + .. code-block:: bash + + uname -r + 4..lts + + If the running kernel is not **lts**: install the LTS kernel manually, + update the bootloader, and check again: + + .. code-block:: bash + + swupd bundle-add kernel-lts + clr-boot-manager set-kernel $(basename $(realpath /usr/lib/kernel/default-lts)) + clr-boot-manager update + reboot + +#. Remove any kernel bundles that are not *kernel-lts* or *kernel-install* + to simplify and avoid conflicts: + + .. code-block:: bash + + swupd bundle-list | grep kernel + swupd bundle-remove + + .. image:: ./figures/vbox/vbox-cl-remove-non-lts-kernels.png + :alt: Initial login to Clear Linux OS on a VirtualBox VM + +#. From the VM Console window, click *Devices* on the top menu bar, and + select *Insert Guest Additions CD image...* to mount the |VB| driver + installation to the |CL| VM. + + .. image:: ./figures/vbox/vbox-vm-insert-ga-cd.png :alt: VirtualBox CD -#. Install Linux users Guest Additions:: +.. note:: + To release the mouse cursor from the VM console window, press the right Ctrl key on the keyboard. - install-vbox-lga -#. Reboot your system:: - - reboot +#. |CL| provides a script called :command:`install-vbox-lga` to help patch + and install |VB| drivers for |CL|. Inside |CL| VM run this command: + .. code-block:: bash + + install-vbox-lga + +#. After the script completes successfully, reboot the |CL| VM. + + .. code-block:: bash + + reboot + +#. After the VM reboot, login and verify the |VB| drivers are loaded: + + .. code-block:: bash + + lsmod | grep ^vbox + + You should see drivers loaded with names beginning with **vbox**: (vboxguest, vboxsf, vboxvideo). + + +The |CL| VM running on |VB| is ready to be used. + +.. _vbox-start-vm-and-lga-end: + +.. _vbox-troubleshooting-begin: Troubleshooting -=============== +*************** -On Windows OS, *VirtualBox* cannot do a **Hardware Virtualization** when -*Hyper-V* is enabled. +#. **Problem:** Out of disk space inside of |CL| and not be able to install + additional bundles. -.. image:: ./figures/vbox-no-vtx.png - :alt: VirtualBox hardware acceleration error + **Solution:** The |CL| images are small to minimize download time and + initial disk space . -To disable *Hyper-V* you should execute:: + Power off the VM and resize the virtual disk for the |CL| VM using the |VB| + `Virtual Media Manager`_ found under the File menu. Afterwards, power the + |CL| VM on and follow the instructions here to have |CL| detect the resized + disk. :ref:`increase-virtual-disk-size` - bcdedit /set {current} hypervisorlaunchtype off +#. **Problem:** On a Microsoft Windows OS, |VB| encounters an error when + trying to start a VM indicating *VT-X/AMD-v hardware acceleration is not + available on your system.* -in an **Administrator: Command Prompt**, then reboot your system. -To enable Hyper-V again, you should execute:: + .. image:: ./figures/vbox/vbox-no-vtx.png + :alt: VirtualBox hardware acceleration error - bcdedit /set {current} hypervisorlaunchtype Auto + + **Solution:** First, double check the `Prerequisites`_ section to make sure + *Hardware accelerated virtualization* extensions have been enabled in the + host system's EFI/BIOS. + + *Hardware accelerated virtualization*, may get disabled for |VB| when another + hypervisor, such as *Hyper-V* is enabled. + + To disable *Hyper-V* execute this command in an + **Administrator: Command Prompt or Powershell**, and reboot the system: + + .. code-block:: bash + + bcdedit /set {current} hypervisorlaunchtype off + + + To enable Hyper-V again, execute this command in an + **Administrator: Command Prompt or Powershell**, and reboot the system: + + .. code-block:: bash + + bcdedit /set {current} hypervisorlaunchtype Auto + +.. _vbox-troubleshooting-end: + + + + +.. |VB| replace:: VirtualBox +.. |VBM| replace:: VirtualBox Manager .. _appropriate instructions: https://www.virtualbox.org/manual/ch02.html + .. _official VirtualBox website: https://www.virtualbox.org/wiki/Downloads + .. _VirtualBox hypervisor: https://www.virtualbox.org/ -.. _latest: https://download.clearlinux.org/image/ + +.. _downloads page: https://clearlinux.org/downloads + +.. _`VirtualBox manual section on Creating a VM`: https://www.virtualbox.org/manual/UserManual.html#gui-createvm + +.. _`VirtualBox manual section on Running a VM`: https://www.virtualbox.org/manual/ch01.html#intro-starting-vm-first-time + +.. _`Virtual Media Manager`: https://www.virtualbox.org/manual/ch05.html#vdis + +.. _`Install Clear Linux OS`: https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install#install-cl-on-your-target-system\ + .. _7zip: http://www.7-zip.org/ + .. _Virtualization Technology: https://www.intel.com/content/www/us/en/virtualization/virtualization-technology/intel-virtualization-technology.html + +.. _`VirtualBox manual section on VBoxManage`: https://www.virtualbox.org/manual/ch08.html#vboxmanage-convertfromraw diff --git a/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst b/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst index 0af68bd7..15d82c56 100644 --- a/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst +++ b/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst @@ -64,7 +64,7 @@ this command: .. code-block:: bash - curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep vmware) + curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep vmware) Visit :ref:`image-types` for additional information about all available |CL| images. @@ -306,8 +306,8 @@ For other guides on using the VMWare Player and ESXi, see: .. _VMware ESXi: https://www.vmware.com/products/esxi-and-esx.html .. _VMware Workstation 14 Player: https://www.vmware.com/products/workstation-player.html .. _VMware Workstation Player guide: https://docs.vmware.com/en/VMware-Workstation-Player/index.html -.. _latest: https://download.clearlinux.org/image/ -.. _image: https://download.clearlinux.org/image +.. _latest: https://cdn.download.clearlinux.org/image/ +.. _image: https://cdn.download.clearlinux.org/image diff --git a/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst b/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst index c7f0f6ab..eeeba384 100644 --- a/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst +++ b/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst @@ -297,6 +297,6 @@ For other guides on using the VMWare Player and ESXi, see: .. _VMware Workstation Player guide: https://docs.vmware.com/en/VMware-Workstation-Player/index.html -.. _latest: https://download.clearlinux.org/image/ +.. _latest: https://cdn.download.clearlinux.org/image/ -.. _image: https://download.clearlinux.org/image +.. _image: https://cdn.download.clearlinux.org/image diff --git a/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-install-cl.rst b/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-install-cl.rst index 6436f172..9d135fd8 100644 --- a/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-install-cl.rst +++ b/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-install-cl.rst @@ -291,4 +291,4 @@ Related topics .. _VMware ESXi: https://www.vmware.com/products/esxi-and-esx.html .. _VMware Workstation Player: https://www.vmware.com/products/workstation-player.html -.. _image: https://download.clearlinux.org/image +.. _image: https://cdn.download.clearlinux.org/image diff --git a/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-preconfigured-cl-image.rst b/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-preconfigured-cl-image.rst index eb1ee92d..5c3530fa 100644 --- a/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-preconfigured-cl-image.rst +++ b/source/clear-linux/get-started/virtual-machine-install/vmware-esxi-preconfigured-cl-image.rst @@ -36,7 +36,7 @@ this command: .. code-block:: bash - curl -O https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep vmware) + curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep vmware) Visit :ref:`image-types` for additional information about all available |CL| images. @@ -290,4 +290,4 @@ Related topics .. _VMware documentation on Enable the Secure Shell (SSH): https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.html.hostclient.doc/GUID-B649CB74-832F-467B-B6A4-8BA67AD5C1F0.html .. _VMware documentation on General ESXi Security Recommendations: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-B39474AF-6778-499A-B8AB-E973BE6D4899.html .. _VMware Workstation Player: https://www.vmware.com/products/workstation-player.html -.. _image: https://download.clearlinux.org/image +.. _image: https://cdn.download.clearlinux.org/image diff --git a/source/clear-linux/guides/clearlinux/autoproxy.rst b/source/clear-linux/guides/clearlinux/autoproxy.rst index ae2f4188..129cae51 100644 --- a/source/clear-linux/guides/clearlinux/autoproxy.rst +++ b/source/clear-linux/guides/clearlinux/autoproxy.rst @@ -3,12 +3,151 @@ Autoproxy ######### -About -===== +Autoproxy is provided to enable |CL-ATTR| to work smoothly behind a +corporate proxy. -.. include:: autoproxy_about.rst.txt +.. contents:: + :local: + :depth: 1 -Guide -===== +Description +*********** -.. include:: autoproxy_guide.rst.txt \ No newline at end of file +Autoproxy tries to detect a Proxy Auto-Config (PAC) script and use it to +automatically resolve the proxy needed for a given connection. With +Autoproxy, you can use |CL| inside any proxy environment without having to +manually configure the proxies. + +Corporate and private networks can be very complex, needing to restrict and +control network connections for security reasons. The typical side effects +are limited or blocked connectivity and requiring manual configuration of +proxies to perform the most mundane tasks such as cloning a repo or checking +for updates. With |CL|, all of the work is done behind the scenes to +effortlessly use your network and have connections “just work”. + +This feature removes massive complications in network connectivity due to +proxy issues. You can automate tasks like unit testing without worrying +about the proxy not being set and you can remove unset proxies from the +equation when dealing with network unavailability across systems. + +How it works +************ + +We designed Autoproxy around tools provided by most Linux +distributions with a few minor additions and modifications. We leveraged the +DHCP and network information provided from systemd and created a +PAC-discovery daemon. The daemon uses the information to resolve a URL for a +PAC file. The daemon then passes the URL into PACrunner\*. PACrunner +downloads the PAC file and uses the newly implemented Duktape\* engine to +parse it. + +.. figure:: figures/autoproxy_0.png + :width: 400px + + Figure 1: Autoproxy Flow + +From that point on, any cURL\* or network requests query PACrunner for the +correct proxy to use. We modified the cURL library to communicate with +PACrunner over DBus. However, cURL will ignore PACrunner and run normally if +no PAC file is loaded or if you set any proxies manually. Thus, your +environment settings are respected and no time is wasted trying to resolve a +proxy. All these steps happen in the background with no user interaction. + +Troubleshooting +=============== + +Autoproxy allows |CL| to operate seamlessly behind a proxy +because :ref:`swupd ` and other |CL| tools are implemented on +top of libcurl. Tools that do not use libcurl, like git, must +be configured independently. + +If you are familiar with PAC files and WPAD, you can use +:command:`pacdiscovery` and :command:`FindProxyForURL` to +troubleshoot problems with autproxy. + +.. note:: + + Learn more about WPAD, PAC files, and PAC functions at `findproxyforurl`_. + +.. _findproxyforurl: http://findproxyforurl.com/ + +Run :command:`pacdiscovery` with no arguments to indicate + +1. if there is a problem resolving the :command:`WPAD` host name resolution: + + .. code-block:: bash + + pacdiscovery + + .. code-block:: console + + failed getaddrinfo: No address associated with hostname + Unable to find wpad host + +2. or if the :command:`pacrunner` service is disabled (masked). + + .. code-block:: bash + + pacdiscovery + + .. code-block:: console + + PAC url: http://autoproxy.your.domain.com/wpad.dat + Failed to create proxy config: Unit pacrunner.service is masked. + +Unmask the :command:`pacrunner` service by running: + +.. code-block:: bash + + systemctl unmask pacrunner.service + +:command:`FindProxyForURL` with :command:`busctl` can also indicate if the +:command:`pacrunner.service` is masked. + +.. code-block:: bash + + busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client + +.. code-block:: console + + FindProxyForURL ss "http://www.google.com" "google.com" + Unit pacrunner.service is masked. + dig wpad, dig wpad. + +:command:`FindProxyForURL` returns the URL and port of the proxy server when +an external URL and host are provided as arguments. + +.. code-block:: bash + + busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client + +.. code-block:: console + + FindProxyForURL ss "http://www.google.com" "google.com" + s "PROXY proxy.your.domain.com:" + +If a proxy server is not avialable, or if :command:`pacrunner` is running +without a PAC file, :command:`FindProxyForURL` will return "DIRECT". + +.. code-block:: bash + + busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client + +.. code-block:: console + + FindProxyForURL ss "http://www.google.com" "google.com" + s "DIRECT" + +Once :command:`pacdiscovery` is able to look up :command:`WPAD`, restart the +:command:`pacrunner` service: + +.. code-block:: bash + + systemctl stop pacrunner + systemctl restart pacdiscovery + +.. note:: + + A "domain" or "search" entry in :file:`/etc/resolv.conf` is required + for short name lookups to resolve. The :file:`resolv.conf` man page has + additional details. \ No newline at end of file diff --git a/source/clear-linux/guides/clearlinux/autoproxy_about.rst.txt b/source/clear-linux/guides/clearlinux/autoproxy_about.rst.txt deleted file mode 100644 index c74a381a..00000000 --- a/source/clear-linux/guides/clearlinux/autoproxy_about.rst.txt +++ /dev/null @@ -1,46 +0,0 @@ -The |CL-ATTR| is the first Linux distribution to support autoproxy. The OS -can discover a Proxy Auto-Config (PAC) script and use it to automatically -resolve the proxy needed for a given connection. With Autoproxy, you can use -|CL| inside any proxy environment without having to manually -configure the proxies: - -* Automate unit testing without worrying about the proxy not being set -* Remove unset proxies from the equation when dealing with network - unavailability across systems. - -* Automate unit testing without worrying about the proxy not being set -* Remove unset proxies from the equation when dealing with network - unavailability across systems. - -Corporate and private networks can be very complex, needing to restrict and -control network connections for security reasons. The typical side effects -are limited or blocked connectivity and requiring manual configuration of -proxies to perform the most mundane tasks such as cloning a repo or checking -for updates. With |CL|, all of the work is done behind the scenes to -effortlessly use your network and have connections “just work”. - -How Autoproxy works -------------------- - -We designed autoproxy around general tools provided by nearly any Linux -distribution with a few minor additions and modifications. We leveraged the -DHCP and network information provided from systemd and created a -PAC-discovery daemon. The daemon uses the information to resolve a URL for a -PAC file. The daemon then passes the URL into PACrunner*. PACrunner -downloads the PAC file and uses the newly implemented Duktape* engine to -parse it. - -.. figure:: figures/autoproxy_0.png - :width: 400px - - Figure 1: Autoproxy Flow - -From that point on, any cURL* or network requests query PACrunner for the -correct proxy to use. We modified the cURL library to communicate with -PACrunner over DBus. However, cURL will ignore PACrunner and run normally if -no PAC file is loaded or if you set any proxies manually. Thus, your -environment settings are respected and no time is wasted trying to resolve a -proxy. - -More importantly: all these steps happen in the background, very quickly, and -with no user interaction. \ No newline at end of file diff --git a/source/clear-linux/guides/clearlinux/autoproxy_guide.rst.txt b/source/clear-linux/guides/clearlinux/autoproxy_guide.rst.txt deleted file mode 100644 index ca4f64b8..00000000 --- a/source/clear-linux/guides/clearlinux/autoproxy_guide.rst.txt +++ /dev/null @@ -1,79 +0,0 @@ -Autoproxy allows |CL| to operate seamlessly behind a proxy -because :ref:`software update ` and other |CL| tools are -implemented on top of libcurl. Tools that do not use libcurl, like git, must -be configured independently. - -If you encounter problems with autoproxy functioning, use -:command:`sudo pacdiscovery` and :command:`FindProxyForURL` to -help troubleshoot assuming a familiarity with PAC files and WPAD. - -.. note:: - - Learn more about WPAD, PAC files, and PAC functions at `findproxyforurl`_. - -.. _findproxyforurl: http://findproxyforurl.com/ - -Running :command:`pacdiscovery` with no arguments will immediately indicate -one of the following: - -1. If there is a problem resolving the :command:`WPAD` host name resolution: - - .. code:: console - - $ pacdiscovery - failed getaddrinfo: No address associated with hostname - Unable to find wpad host - -2. Or if the :command:`pacrunner` service is disabled (masked): - - .. code:: console - - $ pacdiscovery - PAC url: http://autoproxy.your.domain.com/wpad.dat - Failed to create proxy config: Unit pacrunner.service is masked. - -Unmask the :command:`pacrunner` service by running: - -.. code:: console - - $ systemctl unmask pacrunner.service - -:command:`FindProxyForURL` with :command:`busctl` can also indicate if the -:command:`pacrunner.service` is masked. - -.. code:: console - - $ busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client FindProxyForURL ss "http://www.google.com" "google.com" - Unit pacrunner.service is masked. - dig wpad, dig wpad. - -:command:`FindProxyForURL` returns the URL and port of the proxy server when -an external URL and host are provided as arguments. - -.. code:: console - - $ busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client FindProxyForURL ss "http://www.google.com" "google.com" - s "PROXY proxy.your.domain.com:" - -If a proxy server is not available, or if :command:`pacrunner` is running -without a PAC file, :command:`FindProxyForURL` will return "DIRECT". Otherwise, -it should return your local proxy settings as shown above. - -.. code:: console - - $ busctl call org.pacrunner /org/pacrunner/client org.pacrunner.Client FindProxyForURL ss "http://www.google.com" "google.com" - s "DIRECT" - -Once :command:`pacdiscovery` is able to look up :command:`WPAD`, restart the -:command:`pacrunner` service: - -.. code:: console - - $ systemctl stop pacrunner - $ systemctl restart pacdiscovery - -.. note:: - - A "domain" or "search" entry in :file:`/etc/resolv.conf` is required for short - name lookups to resolve. The :file:`resolv.conf` man page has additional - details. \ No newline at end of file diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index 5b3867df..f8c360e4 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -269,7 +269,7 @@ challenges your monitoring systems, and business continuity plans. .. _`mixin process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixin .. _`mixer process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixer -.. _`downloads page`: https://download.clearlinux.org/image/ +.. _`downloads page`: https://cdn.download.clearlinux.org/image/ .. _`containers page`: https://clearlinux.org/containers .. _`systemd journal-remote service`: https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html .. _`native telemetry solution`: https://clearlinux.org/features/telemetry diff --git a/source/clear-linux/guides/guides.rst b/source/clear-linux/guides/guides.rst index 09662ae0..919a8c24 100644 --- a/source/clear-linux/guides/guides.rst +++ b/source/clear-linux/guides/guides.rst @@ -20,9 +20,6 @@ Clear Linux Tooling :maxdepth: 1 :glob: - clearlinux/* - telemetrics/telemetrics - Maintenance =========== diff --git a/source/clear-linux/guides/maintenance/architect-lifecycle.rst b/source/clear-linux/guides/maintenance/architect-lifecycle.rst new file mode 100644 index 00000000..bf9e8787 --- /dev/null +++ b/source/clear-linux/guides/maintenance/architect-lifecycle.rst @@ -0,0 +1,76 @@ +.. _architect-lifecycle: + +Architect the life-cycle of |CL-ATTR| +##################################### + +This guide provides DevOps with a model to architect the life-cycle of a |CL| +derivative that integrates custom software and content using distinct +workflows. + +Maintaining a |CL| derivative requires: + +* Monitoring upstream |CL| for new releases +* Building software packages and staging +* Employing CI/CD automation for building releases +* Integrating Quality Assurance for testing and validation + +This guide provides the foundation of the recommended infrastructure. + +.. contents:: + :local: + :depth: 1 + +Prerequisites +************* + +* A repository with software RPM artifacts and a CI/CD system with a |CL| + machine for building `mixes` + +* Experience using :ref:`mixer ` to create a |CL|-based distro + +* Experience using :ref:`swupd ` for maintaining the |CL| + build environment + +* Familiarity with |CL| architecture and reuse of its content in releases + +Description +*********** + +Coordinated infrastructure is deployed to automate the life-cycle +of your |CL| derivative. We divide deployment of this infrastucture in two +parts: *Content Workflow*; and *Release Workflow*, shown in Figure 1. Distro Factory manages the *Release Workflow* while capturing the requirements for +maintaining a long-term release cadence. + +.. figure:: figures/architect-lifecycle-1.png + :scale: 100% + :alt: Architect the life-cycle + + Figure 1: Architect the life-cycle + +Content workflow +**************** + +The *Content Workflow* (Figure 1) orchestrates the processes used to manage +the creation of content for the distribution. This includes everything from detecting a new release in a custom software repository to generating RPM package files. The RPM files serve as intermediary artifacts that track software dependencies and provide file-level data consumed in a *Release Workflow*. The `Watcher Pipeline`_ checks |CL| and a content provider, such as Koji, to determine if a new release is necessary. + +Release workflow +**************** + +The *Release Workflow* (Figure 1) gathers the content of the RPMs and +ensures it can be consumed by :ref:`mixer `. A content web server +hosts the |CL| derivative, to which targets connect for updating their OSes. +As an integral part of this toolchain, the *Release Pipeline* enables these +derivatives to incorporate |CL| content into their own custom +content. The *Watcher Pipeline* triggers the `Release Pipeline`_ to create +new releases. + +Implementation +************** + +Distro factory implements the *Release workflow*. To get started on a full implementation, visit |CL| `Distro factory documentation`_. + +.. _Distro factory documentation: https://github.com/clearlinux/clr-distro-factory/wiki#clear-linux-distro-factory + +.. _Release Pipeline: https://github.com/clearlinux/clr-distro-factory/wiki/Release + +.. _Watcher Pipeline: https://github.com/clearlinux/clr-distro-factory/wiki/Watcher diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 257f8e2c..0b955d3d 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -1,178 +1,167 @@ .. _autospec: -Build RPMs with autospec -######################## +autospec +######## -This guide shows you how to create RPMs with :ref:`autospec `, -a tool that assists in automated creation and maintenance of RPM packaging -on |CL-ATTR|. +**autospec** is a tool to assist in the automated creation and maintenance of +RPM packaging in |CL-ATTR|. Where a standard RPM build process using +:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec +requires only a tarball and package name to start. -See our :ref:`autospec concept page ` for a detailed explaination -of how ``autospec`` works on |CL|. For a general understanding of how RPMs work, -we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ . +.. contents:: + :local: + :depth: 1 + +Description +*********** + +The autospec tool attempts to infer the requirements of the :file:`.spec` file +by analyzing the source code and :file:`Makefile` information. It will +continuously run updated builds based on new information discovered from build +failures until it has a complete and valid :file:`.spec` file. If needed, you +can influence the behavior of autospec and customize the build by providing +optional `control files`_ to the autospec tool. + +autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for +additional information on using mock. + +For a general understanding of how RPMs work, visit the `rpm website`_ or the +`RPM Packaging Guide`_ . + +How it works +************ + +Learn the autospec tool set up and process. + +.. contents:: + :local: + :depth: 1 Prerequisites -************* +============= -This guide requires that you: +The setup for building source in |CL| must be completed before using the +autospec tool. -* Have installed |CL| on a host machine or virtual environment. For detailed - instructions on installing |CL|, visit the :ref:`get-started` section. +Refer to `Setup environment to build source`_ for instructions on completing +setup. -* :ref:`install-tooling` +Create an RPM +============= -.. _install-tooling: +The basic autospec process is described in the following steps: -Install the |CL| tooling framework -================================== +#. The :command:`make autospec` command generates a :file:`.spec` file based on + analysis of code and existing control files. -#. Install the `os-clr-on-clr` developer bundle on your host system. + Any control files should be located in the same directory as the resulting + :file:`.spec` file. - .. code-block:: bash + View the `autospec README`_ for more information on `control files`_. - sudo swupd bundle-add os-clr-on-clr +#. autospec creates a build root with mock config. -#. Download the :file:`user-setup.sh` script. +#. autospec attempts to build an RPM from the generated :file:`.spec`. - .. code-block:: bash +#. autospec detects any missed declarations in the :file:`.spec`. - curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh +#. If build errors occur, autospec will scan the build log to try and detect + the root cause. -#. Make :file:`user-setup.sh` executable. +#. If autospec detects the root cause and knows how to continue, it will restart + the build automatically at step 1 with updated build instructions. - .. code-block:: bash +#. Otherwise, autospec will stop the build for user inspection to resolve the + errors. Respond to the build process output by fixing source code issues + and/or editing control files to resolve issues, which may include + dependencies or exclusions. See `autospec README`_ for more information on + control files. - chmod +x user-setup.sh + The user resumes the process at step 1 after errors are resolved. -#. Run the script as an unprivileged user. + If a binary dependency doesn't exist in |CL|, you will need to build it + before running autospec again. - .. code-block:: bash +Following these steps, autospec continues to rebuild the package, based on +new information discovered from build failures, until it has a valid +:file:`.spec`. If no build errors occur, RPM packages are successfully built. - ./user-setup.sh +Examples +******** -#. After the script completes, log out and log in again to complete - the setup process. +Complete `Setup environment to build source`_ before using these examples. - The `user-setup script`_ creates a folder called :file:`clearlinux`, which - contains the :file:`Makefile`, :file:`packages`, and :file:`projects` - subfolders. +.. contents:: + :local: + :depth: 1 - The :file:`projects` folder contains the main tools, `autospec` - and `common`, used for making packages in |CL|. +Example 1: Build RPM with existing spec file +============================================ -Create a RPM with autospec -************************** +This example shows how to build a RPM from a pre-packaged upstream package, with +an existing spec file. The example uses the ``dmidecode`` package. -Choose one of the following options to build RPMs and manage source -code: - -* :ref:`build-a-new-rpm` and spec file using ``make autospecnew``. - -* :ref:`build-source-code-with-existing-spec-file` using ``make build``, without changing the - spec file. - -* :ref:`generate-a-new-spec-file` using ``make autospec``, based on changes in the control files. - -.. _build-a-new-rpm: - -Option 1: Build a new RPM -========================= - -Use this method to build a new RPM with no spec file. In this example, -we build a new helloclear RPM. - -#. Navigate to the autospec workspace. +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: .. code-block:: bash cd ~/clearlinux - -#. Enter the command: - - .. code-block:: bash - - make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" - NAME="helloclear" + make clone_dmidecode .. note:: - For a local tarball, use this type of *URL*: \file:// - -#. If build failures or dependency issues occur, continue below. - Otherwise, skip directly to `Next steps`_. - - #. Navigate to the specific package. + You can clone all package repos at once using: .. code-block:: bash - cd ~/clearlinux/packages/[package-name] + make [-j NUM] clone-packages - #. Respond to the build process output by editing control files to resolve - issues, which may include dependencies or exclusions. - See `autospec readme`_ + The optional NUM is the number of threads to use. - #. Run this command: + For a list of available packages, view the + :file:`~/clearlinux/projects/common/packages` file. - .. code-block:: bash - - make autospec - - Repeat the last two steps above until all errors are resolved and you - complete a successful build. - -**Congratulations!** - -You've successfully created a RPM. - -Skip to `Next steps`_. - -.. _build-source-code-with-existing-spec-file: - -Option 2: Build source code with an existing spec file -====================================================== - -Use this method if you only want to build the RPM using the spec file. This -method assumes that a spec file already exists. In this example, we run a -``make build`` on the ``dmidecode`` package. - -#. Navigate to the ``dmidecode`` package in clearlinux: +#. Navigate to the local copy of the ``dmidecode`` package and build it: .. code-block:: bash cd ~/clearlinux/packages/dmidecode/ - -#. To download the tarball and build, run the command: - - .. code-block:: bash - make build -**Congratulations!** +#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are + in :file:`./results`. -You've successfully created a RPM. +Example 2: Build a new RPM +========================== -Skip to `Next steps`_. +This example shows how to build a new RPM with no spec file. The example will +create a simple helloclear RPM. -.. _generate-a-new-spec-file: - -Option 3: Generate a new spec file with a pre-defined package -============================================================= - -Use this method to modify an existing package. In this example, you will -modify an existing |CL| package called ``dmidecode`` to create a custom -RPM. You will make a simple change to this package, change the revision to -a new number that is higher than the |CL| OS version, and rebuild the package. - -#. Navigate to clearlinux: +#. Navigate to the autospec workspace and build the helloclear RPM. The + :file:`Makefile` provides a :command:`make autospecnew` that can + automatically generate an RPM package using the autospec tool. You must pass + the URL to the source tarball and the NAME of the RPM you wish to create: .. code-block:: bash cd ~/clearlinux + make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" -#. Copy the ``dmidecode`` package. + The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and + additional RPMs are in :file:`./packages/helloclear/results`. + +Example 3: Generate a new spec file with a pre-defined package +============================================================== + +This example shows how to modify an existing package to create a custom RPM. In +this example you will make a simple change to the ``dmidecode`` package and +rebuild the package. + +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: .. code-block:: bash + cd ~/clearlinux make clone_dmidecode #. Navigate into the *dmidecode* directory: @@ -181,9 +170,9 @@ a new number that is higher than the |CL| OS version, and rebuild the package. cd packages/dmidecode -#. With an editor, open the :file:`excludes` file and add these lines: +#. Open the :file:`excludes` file with an editor and add these lines: - .. code-block:: bash + .. code-block:: console /usr/bin/biosdecode /usr/bin/ownership @@ -197,49 +186,211 @@ a new number that is higher than the |CL| OS version, and rebuild the package. These files aren't needed by dmidecode, so we can remove them without any issues. -#. Save the file and exit. - -#. At :file:`~/clearlinux/packages/dmidecode`, build the modified - ``dmidecode`` package: +#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package: .. code-block:: bash make autospec - When the process completes, you will see new RPM packages in the - :file:`results/` folder. +#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. -#. To view the new RPM packages, enter: +Example 4: Provide control files to autospec +============================================ + +This example shows how to modify control files to correct build failures that +autospec is unable to resolve. In this example you will add a missing license +and dependencies in order for autospec to complete a successful build. + +#. Navigate to the autospec workspace: .. code-block:: bash - ls /clearlinux/packages/dmidecode/results/ + cd ~/clearlinux -**Congratulations!** +#. If you have not already, clone all upstream package repos: -You've successfully created a RPM. + .. code-block:: bash -Next steps + make [-j NUM] clone-packages + + The optional NUM is the number of threads to use. + + .. note:: + + In a later step of this example, we will search the cloned package repos + for a missing dependency. + +#. Build the opae-sdk RPM: + + .. code-block:: bash + + make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk" + + This will give an error for a missing license file: + + .. code-block:: console + + [FATAL] Cannot find any license or opae-sdk.license file! + +#. Navigate to the package with build failures: + + .. code-block:: bash + + cd packages/opae-sdk + +#. Add a license: + + .. code-block:: bash + + echo "Intel Corporation" > opae-sdk.license + +#. Run autospec again: + + .. code-block:: bash + + make autospec + + This will result in a generic error: + + .. code-block:: console + + [FATAL] Build failed, aborting + +#. Open the build log to view the error details: + + .. code-block:: bash + + cat ./results/build.log + + In the build log, you will find details for the specific failures. In this + instance, there are missing dependencies: + + .. code-block:: console + + CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: + CJSON_LIBRARY + linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib + json-c_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + libuuid_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + +#. Search the spec files of upstream |CL| packages to see if the json-c library + is availabe. In this case, it does exist and we'll add the json-c 'dev' + package into the buildreq_add: + + .. code-block:: bash + + grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec + echo "json-c-dev" >> buildreq_add + + .. note:: + + This search step works only if the user cloned all of the upstream package + repos. In this example, upstream package repos were cloned in a previous + step. + +#. Search the spec files of upstream |CL| packages to see if the libuuid library + is available. In this case, it exists in the util-linux package, so we'll add + util-linux-dev package into the buildreq_add: + + .. code-block:: bash + + grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec + echo "util-linux-dev" >> buildreq_add + +#. Run autospec again and find the successfully-generated RPMs in the rpms + directory: + + .. code-block:: bash + + make autospec + +.. note:: + + If you need a dependency that does not exist in the |CL| repo, you must first + build it manually (see `Example 2: Build a new RPM`_), then add the repo so + that autospec knows the package exists. For example: + + .. code-block:: bash + + cd ~/clearlinux/packages/ + make repoadd + make repostatus + + You only need to add the dependency to the :file:`buildreq_add` control file + if autospec is not able to automatically find the correct dependency on its + own. + +References ********** -Now you can create a custom bundle with your new RPM and use it with |CL|: +Reference the `autospec README`_ for details regarding autospec commands and +options. -* Use the :ref:`Mixer tool ` to add a new bundle to your derivative of |CL|. -* Use the :ref:`Mixin tool ` to customize your upstream |CL| installation with a new bundle. +Setup environment to build source +================================= + +.. _install-tooling-after-header: + +Setup of the workspace and tooling used for building source in |CL| is mostly +automated for you with a setup script. It uses tools from the +:command:`os-clr-on-clr` bundle. + +The setup script creates a workspace in the :file:`clearlinux` folder, with the +subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The +:file:`projects` folder contains the main tools used for making packages in +|CL|: `autospec` and `common`. + +Follow these steps to setup the workspace and tooling for building source: + +#. Install the :command:`os-clr-on-clr` bundle: + + .. code-block:: bash + + sudo swupd bundle-add os-clr-on-clr + +#. Download the :file:`user-setup.sh` script: + + .. code-block:: bash + + curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh + +#. Make :file:`user-setup.sh` executable: + + .. code-block:: bash + + chmod +x user-setup.sh + +#. Run the script as an unprivileged user: + + .. code-block:: bash + + ./user-setup.sh + +#. After the script completes, log out and log in again to complete the setup + process. + +#. Set your Git user email and username for the repos on your system: + + .. code-block:: bash + + git config --global user.email "you@example.com" + git config --global user.name "Your Name" + + This global setting is used by |CL| tools that make use of Git. + +.. _install-tooling-end: Related topics ************** * :ref:`Mixer tool ` * :ref:`Mixin tool ` -* :ref:`autospec ` -* :ref:`Bundles ` - - -.. _rpm website: http://rpm.org - -.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ .. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh - -.. _autospec readme: https://github.com/clearlinux/autospec +.. _autospec README: https://github.com/clearlinux/autospec +.. _control files: https://github.com/clearlinux/autospec#control-files +.. _mock wiki: https://github.com/rpm-software-management/mock/wiki +.. _rpm website: http://rpm.org +.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ diff --git a/source/clear-linux/guides/maintenance/enable-user-space.rst b/source/clear-linux/guides/maintenance/enable-user-space.rst index 288ada71..1ed1fcb1 100644 --- a/source/clear-linux/guides/maintenance/enable-user-space.rst +++ b/source/clear-linux/guides/maintenance/enable-user-space.rst @@ -6,22 +6,15 @@ Create and enable a new user space This section provides steps to complete the following basic setup tasks for a newly installed |CL-ATTR| system: -* Create a new user. -* Update the OS to its most current version using `swupd`. -* Install the most common applications for system administrators and - developers using bundles. -* Set up a new user and add the new user to the `wheel` group. -* Install a GUI to test `sudo` privileges. - -.. note:: - Log in as the root user to complete the tasks in this - section. +.. contents:: + :local: + :depth: 1 Create a new user -****************** +***************** To create a new user and set a password for that user, enter the following -commands as a root user: +commands as a `root` user: .. code-block:: bash @@ -33,29 +26,8 @@ including the password for that user. The :command:`passwd` command prompts you to enter a new password. Retype the new password for the new user account just created. -Install and update the OS software to its current version -********************************************************* - -|CL| has a unique application and architecture to add and update applications -and to perform system updates called software update utility or -:command:`swupd`. Software applications are installed as bundles using the -sub-command :command:`bundle-add`. - -The `sysadmin-basic` bundle installs the vast majority of -applications useful to a system administrator. -Install the `sysadmin-basic` bundle: - -.. code-block:: bash - - swupd bundle-add sysadmin-basic - -We provide the full list of bundles and packages installed with the -`sysadmin-basic`_ bundle. Additionally, we have listed -`all bundles`_ for |CL|, active or deprecated. Click any bundle on the -list to view the manifest of the bundle. - -Set up a new user and add the new user to the `wheel` group -*********************************************************** +Add the new user to the `wheel` group +************************************* Before logging off as root and logging into your new user account, enable the :command:`sudo` command for your new ``. @@ -73,65 +45,50 @@ To be able to execute all applications with root privileges, add the To log off as root, enter :command:`exit`. - The command will bring you back to the `login:` prompt. - #. Enter the new `` and the password created earlier. - You will now be in the home directory of ``. The bundle - `sysadmin-basic`_ contains the majority of applications that a system - administrator would want, but it does not include a graphical user - interface. The `desktop` bundle includes the GNOME\* Display Manager and - additional supporting applications. + You will now be in the home directory of ``. -Install a GUI to test `sudo` privileges -======================================== -.. note:: +Install and update the OS software to its current version +********************************************************* - If you are following this sequence after just setting up the - pre-configured VMware\* virtual machine from the repo, you must - :ref:`increase virtual disk size` or the - following step will fail. +The |CL| software utility :ref:`swupd ` allows you to perform system updates while reaping the benefits of upstream development. -To test the :command:`sudo` command and ensure it is set up correctly, -install the GNOME Display Manager (gdm) and start it. +To update your newly installed OS, run: -#. To install the the GNOME Display Manager using :command:`swupd`, enter - the following command: +.. code-block:: bash - .. code-block:: bash + sudo swupd update - sudo swupd bundle-add desktop +Add a bundle +************ -#. To start the GNOME Display Manager, enter the following command: +Software applications are installed as bundles using the command +:command:`swupd bundle-add`. Experienced Linux* users might compare `swupd` +to running :command:`apt-get` or :command:`yum install` for package +management. Yet |CL| manages packages at the level of bundles, which +are integrated stacks of packages. - .. code-block:: bash +For example, the `sysadmin-basic` bundle installs the majority of applications useful to a system administrator. To install it, enter: - systemctl start gdm +.. code-block:: bash -#. The system prompts you to authenticate the user. Enter the password for - ``, and the GNOME Display Manager starts as shown in Figure - 1: + swupd bundle-add sysadmin-basic - .. figure:: figures/gnomedt.png - :scale: 50 % - :alt: Gnome Desktop +View a full list of bundles and packages installed with the `sysadmin-basic`_ bundle. You can also view `all bundles`_ for |CL|, active or deprecated. - Figure 1: :guilabel:`Gnome Desktop` +Expand your knowledge of :command:`swupd` and check out our developer resources: -#. To start the GNOME Display Manager each time you start your system, enter - the following command: - - .. code-block:: bash - - systemctl enable gdm +* :ref:`swupd-guide` +* :ref:`developer-workstation` Next steps -*********** +********** -With your system now running |CL|, many opportunities exist. +Check out our guides and tutorials. -Visit the :ref:`tutorials ` page for examples on using your |CL| -system. +* :ref:`guides` +* :ref:`tutorials` .. _`sysadmin-basic`: https://github.com/clearlinux/clr-bundles/blob/master/bundles/sysadmin-basic diff --git a/source/clear-linux/guides/maintenance/figures/architect-lifecycle-1.png b/source/clear-linux/guides/maintenance/figures/architect-lifecycle-1.png new file mode 100644 index 00000000..294d0f35 Binary files /dev/null and b/source/clear-linux/guides/maintenance/figures/architect-lifecycle-1.png differ diff --git a/source/clear-linux/guides/maintenance/figures/distro-factory-1.png b/source/clear-linux/guides/maintenance/figures/distro-factory-1.png new file mode 100644 index 00000000..49102bac Binary files /dev/null and b/source/clear-linux/guides/maintenance/figures/distro-factory-1.png differ diff --git a/source/clear-linux/guides/maintenance/increase-virtual-disk-size.rst b/source/clear-linux/guides/maintenance/increase-virtual-disk-size.rst index 287be71b..36344aec 100644 --- a/source/clear-linux/guides/maintenance/increase-virtual-disk-size.rst +++ b/source/clear-linux/guides/maintenance/increase-virtual-disk-size.rst @@ -166,4 +166,4 @@ Resize the filesytem Congratulations! You have resized the disk, partition, and filesystem. At this point, the increase in disk capacity is usable. -.. _releases: https://download.clearlinux.org/releases/ \ No newline at end of file +.. _releases: https://cdn.download.clearlinux.org/releases/ \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/ister.rst b/source/clear-linux/guides/maintenance/ister.rst new file mode 100644 index 00000000..4f29229a --- /dev/null +++ b/source/clear-linux/guides/maintenance/ister.rst @@ -0,0 +1,94 @@ +.. _ister: + +ister.py image builder +###################### + +The `ister.py tool`_ is a template based installer used by |CL-ATTR| to produce +images for each release. The same ister tool is available for use in |CL| to +create custom images based on an upstream image. + +.. contents:: + :local: + :depth: 1 + +Description +*********** + +|CL| is a rolling release and produces on average 10 releases per week using the +ister tool. With each release we produce multiple +`image types for different environments`_ and use cases such as installers, +Hyper-V, KVM, or VMWare. + +Each image has a JSON configuration file used by ister to generate the image. +These JSON configuration files describe the image type, partitions, version, +and which bundles will be preinstalled by default with the image. For each image +type we produce, the corresponding JSON configuration file for the image is also +published. + +The :ref:`mixer` tool also uses ister to build images for your custom +mix. Like upstream images, a JSON configuration file is defined for the image, +which ister uses to generate the image. Refer to the :ref:`mixer` guide +for instruction on using ister to build an image for a custom mix. + +Examples +******** + +Recreate an upstream image +========================== + +The published configuration files for upstream images may be used to recreate an +image, for example when you want to: + +* Use an older version of |CL| and the image is no longer available (only after + March 2017). +* Customize the partitions of an image. +* Customize the bundles preinstalled in an image. +* Run your own post installation script. + + +Follow these steps to recreate an upstream image based on the image's JSON +configuration file: + +#. Install the :command:`os-installer` bundle. Refer to `Install a bundle`_ for + more details. + +#. Download the `ister.py tool`_ and grant it sudo privileges. + +#. Download the JSON configuration file for the desired image: + + * `Configuration files for the current release`_ + * `Previous releases`_ (only after March 2017) + + For a previous release, navigate to `Previous releases`_, select the version + you want, and find the JSON configuration file under + :file:`/clear/config/image`. For example: + ``https://cdn.download.clearlinux.org/releases/15700/clear/config/image/`` + +#. Download “PostNonChroot” script (if applicable). + + The JSON configuration file for the image may have an accompanying + “PostNonChroot” script that is executed at the end of the image creation + process. If it does, download the script and make it executable. + +#. Edit the JSON configuration file as needed. + +#. If your configuration file has an accompanying "PostNonChroot" script, change + the default path of the script to match your path. + +#. Generate the new image with the following command: + + .. code-block:: bash + + sudo ister.py -t [JSON configuration] + +Related topics +************** + +* :ref:`mixer` +* :ref:`bulk-provision` + +.. _ister.py tool: https://github.com/bryteise/ister +.. _image types for different environments: https://cdn.download.clearlinux.org/image/README-IMAGES.html +.. _Configuration files for the current release: https://cdn.download.clearlinux.org/current/config/image/ +.. _Previous releases: https://cdn.download.clearlinux.org/releases/ +.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/kernel-development.rst b/source/clear-linux/guides/maintenance/kernel-development.rst new file mode 100644 index 00000000..57b5b4ff --- /dev/null +++ b/source/clear-linux/guides/maintenance/kernel-development.rst @@ -0,0 +1,447 @@ +.. _kernel-development: + +Kernel development +################## + +This document shows how to obtain and compile a Linux* kernel source +using |CL-ATTR| development tooling. + +The `kernels available`_ in |CL| aim to be performant and practical. In some +cases, it may be necessary to modify the kernel to suit your specific needs +or test new kernel code as a developer. + +.. contents:: + :local: + :depth: 1 + :backlinks: top + +Source RPM files (SRPM) are also available for all |CL| kernels, and can be +used for development instead. Select this link to view the latest `source RPM files`_. + +Request changes be included with the |CL| kernel +************************************************ + +If the kernel modification you need is already open source and likely to be +useful to others, consider submitting a request to include it in the +|CL| kernels.If your change request is accepted, you do not need to maintain your own modified kernel. + +Make enhancement requests to the |CL| `distribution on GitHub`_ . + +Set up kernel development environment +************************************* + +In some cases, it may be necessary to modify the kernel to suit your specific +needs or to test new kernel code. + +You can build and install a custom kernel; however you must: + +* Disable Secure Boot +* Maintain any updates to the kernel going forward + +To create a custom kernel, start with the |CL| development environment. +Then make changes to the kernel, build it, and install it. + + +Install the |CL| development tooling framework +============================================== + +.. include:: autospec.rst + :start-after: install-tooling-after-header: + :end-before: install-tooling-end: + +Clone the kernel package +======================== +Clone the existing kernel package repository from |CL| as a starting point. + +#. Clone the Linux kernel package from |CL|. Using the + :command:`make clone_` command in the + :file:`clearlinux/` directory clones the package from the + `clearlinux-pkgs GitHub`_. + + .. code-block:: bash + + cd ~/clearlinux + make clone_linux + +#. Navigate into the cloned package directory. + + .. code-block:: bash + + cd ~/clearlinux/packages/linux + + +The "linux" package is the kernel that comes with |CL| in the `kernel-native` +bundle. Alternatively, you can use a different kernel variant as the base for +modification. For a list of kernel package names which you can clone instead, +see the `clearlinux-pkgs GitHub`_. + +.. note:: + + The latest version of the |CL| kernel package is pulled as a starting + point. An older version can pulled by switching to different git tag by using :command:`git checkout tag/`. + +Change the kernel version +========================= + +|CL| tends to use the latest kernel available from `kernel.org`_, the Linux +upstream. The kernel version that will be built can be changed in the +RPM SPEC file. While most packages in Clear Linux are typically packaged +using :ref:`autospec-about`, the kernel is not. This means control files +provided by autospec are not available and changes must be made manually. + +#. Open the Linux kernel package RPM SPEC file in an editor. + + .. code-block:: bash + + $EDITOR linux.spec + +#. Modify the Version, Release, and Source0 URL entries at the top of the + file to change the version of Linux kernel that will be compiled. + + A list of current and available kernel release can be found on + `kernel.org`_. + + .. code-block:: bash + + Name: linux + Version: 4.20.8 + Release: 696 + License: GPL-2.0 + Summary: The Linux kernel + Url: http://www.kernel.org/ + Group: kernel + Source0: https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.20.8.tar.xz + Source1: config + Source2: cmdline + + %define ktarget native + + .. note:: + - Consider changing the Name from *linux* in the RPM spec file to easily identify a modified kernel. + + - Consider changing the ktarget from *native* in the RPM spec file to easily identify a modified kernel. + +#. Commit and save the changes to the file. + +Pull a copy of the Linux kernel source code +=========================================== + +Obtain a local copy of the source code to make modifications against. + +#. Run make sources to pull the kernel source code specified in the RPM + SPEC file. In the example, it downloads the :file:`linux-4.20.8.tar.xz` file. + + .. code-block:: bash + + make sources + + +#. Extract the kernel source code archive. This will create a working copy + of the Linux source that you can modify. + + .. code-block:: bash + + tar -xvf linux-4.20.8.tar.xz + +#. Navigate to the extracted directory. In this example, it has been + extracted into a :file:`linux-4.20.8` directory. + + .. code-block:: bash + + cd linux-4.20.8/ + + +Customize the Linux kernel source +********************************* + +After the kernel sources have been obtained, customizations to the kernel +configuration or source code can be made for inclusion with the kernel +build. These customizations are optional. + +Modify kernel configuration +=========================== + +The kernel source has many configuration options available to pick support for different hardware and software features. + +These configuration values must be provided in the :file:`.config` file at +compile time. You will need to make modifications to the :file:`.config` +file, and include it in the kernel package. + + +#. Make sure you have followed the steps to + `Pull a copy of the Linux kernel source code`_ and are in the kernel + source working directory. + + +#. If you have an existing :file:`.config` file from an old kernel, copy it + into the working directory as :file:`.config` for comparison. + Otherwise, use the |CL| kernel configuration file as template + + .. code-block:: bash + + cp ~/clearlinux/packages/linux/config .config + + +#. Make any desired changes to the :file:`.config` using a kernel + configuration tool. Below are some popular options: + + - :command:`$EDITOR .config` - the .config file can be directly edited + for simple changes with names that are already known. + + - :command:`make config` - a text-based tool that asks questions + one-by-one to decide configuration options. + + - :command:`make menuconfig` - a terminal user interface that provides + menus to decide configuration options. + + - :command:`make xconfig` - a graphical user interface that provides + tree views to decide configuration options. + + + More configuration tools can be found by looking at the make help: + :command:`make help | grep config` + +#. Commit and save the changes to the :file:`.config` file. + +#. Copy the :file:`.config` file from the kernel source directory into + the kernel package directory as :file:`config` for inclusion in the build. + + .. code-block:: bash + + cp .config ../config + +Modify kernel source code +========================= + +Changes to kernel code are applied with patch files. Patch files are +formatted git commits that can be applied to the main source code. + +You will need to obtain a copy of the source code, +make modifications, generate patch file(s), and add them to the RPM SPEC +file for inclusion during the kernel build. + +If you have a large number of patches or a more complex workflow, +consider using a patch management tool in addition to Git such as +`Quilt`_. + + +#. Make sure you have followed the steps to + `Pull a copy of the Linux kernel source code`_ and are in the kernel + source working directory. + + +#. Initialize the kernel source directory as a new git repo and create a + commit with all the existing source files to begin tracking changes. + + .. code-block:: bash + + git init + git add -A + git commit -m "Initial commit of Linux kernel source" + + +#. Apply patches provided by the |CL| kernel package to the kernel source + in the working directory. + + .. code-block:: bash + + git am ../*.patch + + +#. Make any of your desired code changes to the Linux source code files. + + +#. Track and commit your changes to the local git repo. + + .. code-block:: bash + + git add + git commit -m "My patch for driver A" + + +#. Generate a patch file based on your git commits. + represents the number of local commits to create patch file. + See the `git-format-patch Documentation`_ for detailed information + on using :command:`git format-patch` + + .. code-block:: bash + + git format-patch - + +#. Copy the patch files from the patches directory in the linux + source tree to the RPM build directory. + + .. code-block:: bash + + cp *.patch ~/clearlinux/packages/linux/ + + +#. Navigate back to the RPM build directory. + + .. code-block:: bash + + cd ~/clearlinux/packages/linux/ + +#. Open the Linux kernel package RPM SPEC file in an editor. + + .. code-block:: bash + + $EDITOR linux.spec + +#. Locate the section of the SPEC file that contains existing patch + variable definitions and append your patch file name. Ensure the + patch number does not collide with an existing patch. + In this example, the patch file is called + :file:`2001-my-patch-for-driver-A.patch` + + .. code-block:: bash + + # + # Small Clear Linux Tweaks + # + Patch0501: 0501-zero-extra-registers.patch + Patch0502: 0502-locking-rwsem-spin-faster.patch + + #Serie1.name WireGuard + #Serie1.git https://git.zx2c4.com/WireGuard + #Serie1.tag 00bf4f8c8c0ec006633a48fd9ee746b30bb9df17 + Patch1001: 1001-WireGuard-fast-modern-secure-kernel-VPN-tunnel.patch + #Serie1.end + + Patch2001: 2001-my-patch-for-driver-A.patch + + +#. Locate the section of the SPEC file further down that contains + patch application and append your patch file number used in the step above. + In this example, patch2001 is added. + + .. code-block:: bash + + # + # Small tweaks + # + %patch0501 -p1 + %patch0502 -p1 + + #Serie1.patch.start + %patch1001 -p1 + #Serie1.patch.end + + %patch2001 -p1 + + +#. Commit and save the changes to the RPM SPEC file. + +Modify kernel boot parameters +============================= +The kernel boot options are passed from the bootloader to the kernel with +command-line parameters. + +While temporary changes can be made to kernel parameters on a running +system or on a during boot, you can also modify the default parameters that +are persistent and distributed with a customized kernel. + + +#. Open the kernel :file:`cmdline` file in an editor. + + .. code-block:: bash + + $EDITOR cmdline + + +#. Make any desired change to the kernel parameters. + For example, you can remove the :command:`quiet` parameter to see more + verbose output of kernel log messages during the boot process. + +#. Commit and save the changes to the :file:`cmdline` file. + +See the `Kernel parameters documentation`_ for a list of available +parameters. + +Build and install the kernel +**************************** +After changes have been made to the kernel source and RPM SPEC file, +the kernel is ready to be compiled and packaged into an RPM. + +The |CL| development tooling makes use of :command:`mock` environments to +isolate building of packages in a sanitized workspace. + +#. Start the compilation process by issuing the :command:`make build` + command. This process is typically resource intensive and will take a while. + + .. code-block:: bash + + make build + + .. note:: + The `ccache plugin for mock`_ can be enabled to help speed up any future rebuilds of the kernel package by caching compiler outputs and reusing them. + + +#. The result will be multiple :file:`.rpm` files in the :file:`rpms` + directory as output. + + .. code-block:: bash + + ls rpms/ + + The kernel RPM will be named + :file:`linux--.x86_64.rpm` + + +#. The kernel RPM file can be input to the :ref:`mixer` to create a + custom bundle and mix of |CL|. + +Alternatively, the kernel RPM bundle can be installed manually on a local +machine for testing. This approach works well for individual development or +testing. For a more scalable and customizable approach, consider using the +:ref:`mixer` to provide a custom kernel with updates. + +1. Install the kernel onto the local system by extracting the RPM with the + :command:`rpm2cpio` command. + + .. code-block:: bash + + rpm2cpio linux--.x86_64.rpm | (cd /; sudo cpio -i -d -u -v); + + +#. Update the |CL| boot manager using :command:`clr-boot-manager` and reboot. + + .. code-block:: bash + + sudo clr-boot-manager list-kernels + sudo clr-boot-manager set-kernel org.clearlinux..- + + sudo reboot + + +#. After a reboot, verify the customized kernel is running. + + .. code-block:: bash + + uname -a + +Related topics +************** + +* :ref:`kernel-modules` +* :ref:`mixer` + +.. _kernels available: https://clearlinux.org/documentation/clear-linux/reference/compatible-kernels + +.. _distribution on GitHub: https://github.com/clearlinux/distribution/issues/new/choose + +.. _source RPM files: https://cdn.download.clearlinux.org/current/source/SRPMS/ + +.. _Quilt: http://savannah.nongnu.org/projects/quilt + +.. _clearlinux-pkgs GitHub: https://github.com/clearlinux-pkgs + +.. _kernel.org: https://www.kernel.org/ + +.. _Kernel parameters documentation: https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt + +.. _ccache plugin for mock: https://fedoraproject.org/wiki/Mock/Plugin/CCache?rd=Subprojects/Mock/Plugin/CCache + +.. _git-format-patch Documentation: https://git-scm.com/docs/git-format-patch + +.. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/kernel-modules-dkms.rst b/source/clear-linux/guides/maintenance/kernel-modules-dkms.rst new file mode 100644 index 00000000..a9718b94 --- /dev/null +++ b/source/clear-linux/guides/maintenance/kernel-modules-dkms.rst @@ -0,0 +1,319 @@ +.. _kernel-modules-dkms: + +Add kernel modules with DKMS +############################ + +Kernel modules are additional pieces of software capable of being inserted +into the Linux kernel to add functionality, such as a hardware driver. +Kernel modules may already be part of the Linux source tree (in-tree) or may +come from an external source, such as directly from a vendor (out-of-tree). + +In cases where drivers beyond those enabled by default in |CL-ATTR| are +needed it may be necessary to manually build out-of-tree modules. + +Out-of-tree kernel modules can be can be `manually built and maintained +`_. Out-of-tree kernel modules can also be managed with the +`Dynamic Kernel Module System (DKMS)`_ on |CL| using the instructions in this +document. + +:abbr:`DKMS (Dynamic Kernel Module System)` is a framework that facilitates +the building and installation of kernel modules. This allows |CL| to provide +hooks that automatically rebuild modules against new kernel versions. + + +.. contents:: :local: + :depth: 1 + :backlinks: top + + + +.. include:: kernel-modules.rst + :start-after: kernel-modules-availability-begin: + :end-before: kernel-modules-availability-end: + + + + +Install DKMS on |CL| +==================== + +.. _kernel-modules-dkms-install-begin: + +The *kernel-native-dkms* bundle provides the :command:`dkms` program and +Linux kernel headers, which are required for compiling kernel modules. + +The *kernel-native-dkms* bundle also: + +* Adds a systemd update trigger + (:file:`/usr/lib/systemd/system/dkms-new-kernel.service`) to automatically + run DKMS to rebuild modules after a kernel upgrade occurs with :ref:`swupd + update `. + +* Disables kernel modules signature verification by appending a kernel + command-line parameter (:command:`module.sig_unenforce`) from + :file:`/usr/share/kernel/cmdline.d/clr-ignore-mod-sig.conf`. + +* Adds a notification to the Message of the Day (MOTD) indicating kernel + modules signature verification is disabled. + +.. warning:: + + #. It is important to always review the output of :command:`swupd update` to + make sure kernel modules rebuilt against the new kernel successfully. This is + especially important for systems where a successful boot relies on a kernel + module. + + +Install the *kernel-native-dkms* or *kernel-lts-dkms* bundle: + +#. Determine which kernel variant is running on |CL|. Only the *native* + and *lts* kernels are enabled to build and load out-of-tree kernel modules + with DKMS. + + .. code-block:: bash + + $ uname -r + 5.XX.YY-ZZZZ.native + + Ensure *.native* or *.lts* is in the kernel name. + +#. Install the dkms bundle corresponding to the installed kernel. + *kernel-native-dkms* for the native kernel or *kernel-lts-dkms* for the + lts kernel. + + .. code-block:: bash + + sudo swupd bundle-add kernel-native-dkms + + or + + .. code-block:: bash + + sudo swupd bundle-add kernel-lts-dkms + + +#. Update the |CL| bootloader and reboot. + + .. code-block:: bash + + sudo clr-boot-manager update + reboot + + +.. _kernel-modules-dkms-install-end: + +Build, install, and load an out-of-tree module +============================================== +In some cases you may need an out-of-tree kernel module that is not available +through |CL|. + + +Prerequisites +------------- + +You can build and load out-of-tree kernel modules, however you must: + +* Disable Secure Boot in UEFI/BIOS. The loading of new out-of-tree modules + modifies the signatures Secure Boot relies on for trust. + +* Have a kernel module package in the form of source code and/or precompiled + binaries. + + +This approach works well for individual use or testing. For a more +scalable and customizable approach, consider using the `mixer tool`_ to +provide a custom kernel and updates. + +Obtain kernel module source +--------------------------- + +A :file:`dkms.conf` file inside of the kernel module's source code directory +is required to inform DKMS how the kernel module should be compiled. + +Kernel modules may come packaged as: + +- Source code without a dkms.conf +- Source code with a premade dkms.conf +- Source code with a premade dkms.conf and precompiled module binaries +- Precompiled module binaries only without source code + +Precompiled kernel module binaries will not work on |CL| because it requires +kernel modules to be built against the same kernel source tree before they can +be loaded. + +If you are only able to obtain source code without a dkms.conf, a +:file:`dkms.conf` file will need to be manually created. + +#. Download the kernel module's source code. + + - Review the available download options. Some kernel modules provide + separate archives which are specifically enabled for DKMS support. + + - Review the README documentation because it often provides required + information to build the module with DKMS support. + + .. code-block:: bash + + curl -O http://.tar.gz + tar -xvf .tar.gz + cd / + cat README + + +Build kernel module with an existing dkms.conf +---------------------------------------------- + +If the kernel module maintainer packaged the source archive with the +:command:`dkms mktarball` command, the entire archive can be passed to the +:command:`dkms ldtarball` which will complete many steps for you. + +The archive will contain the required :file:`dkms.conf` file, and may contain +a :file:`dkms_source_tree` directory and a :file:`dkms_binaries_only` +directory. + + +#. Run the :command:`dkms ldtarball` command against the kernel module archive. + + .. code-block:: bash + + dkms ldtarball .tar.gz + + + :command:`dkms ldtarball` will place the kernel module source under + :file:`/usr/src/-/`, build if necessary, and + add the module into the dkms tree. + + +#. Verify the kernel module is detected by checking the output of + :command:`dkms status`. + + .. code-block:: bash + + dkms status + + +#. Install the kernel module. + + .. code-block:: bash + + dkms install -m -v + + + +Build kernel module without an existing dkms.conf +------------------------------------------------- + +If the kernel module source does not contain a :file:`dkms.conf` file or the +:command:`dkms ldtarball` command encounters errors, it needs to be manually +created. + +Review the kernel module README documentation for guidance on what needs to be +in the :file:`dkms.conf` including special variables that may be required to +build successfully. + +Here are some additional resources that can be used for reference: + +* The DKMS manual page (:command:`man dkms`) shows detailed syntax in the + DKMS.CONF section + +* ``_ + (shows an example where a single package contains multiple modules) + +* ``_ + + +The instructions below show a generic example: + +#. Create or modify the :file:`dkms.conf` file inside of the extracted source + code directory. + + .. code-block:: bash + + $EDITOR dkms.conf + + MAKE="make -C src/ KERNELDIR=/lib/modules/${kernelver}/build" + CLEAN="make -C src/ clean" + BUILT_MODULE_NAME=custom_module + BUILT_MODULE_LOCATION=src/ + PACKAGE_NAME=custom_module + PACKAGE_VERSION=1.0 + DEST_MODULE_LOCATION=/kernel/drivers/other + + This example identifies a kernel module named *custom_module* with version + *1.0*. + +#. Copy the kernel module source code into the :file:`/usr/src/` directory. + + .. code-block:: bash + + sudo mkdir /usr/src/- + sudo cp -Rv . /usr/src/- + + .. note:: + ** and ** should match the entries in :file:`dkms.conf` + + +#. Add the kernel module to the DKMS tree so that it is tracked by DKMS. + + .. code-block:: bash + + sudo dkms add -m + +#. Build the kernel module using DKMS. If the build encounters errors, the + :file:`dkms.conf` may need to be adjusted. + + .. code-block:: bash + + sudo dkms build -m -v + + +#. Install the kernel module using DKMS. + + .. code-block:: bash + + sudo dkms install -m -v + + + +Load kernel module +------------------ + +By default, DKMS installs modules "in-tree" under :file:`/lib/modules` so the +:command:`modprobe` command can be used to load them. + +#. Load the installed module with the :command:`modprobe` command. + + .. code-block:: bash + + sudo modprobe + +#. Validate the kernel module is loaded. + + .. code-block:: bash + + lsmod | grep + + + + +.. include:: kernel-modules.rst + :start-after: kernel-modules-autoload-begin: + :end-before: kernel-modules-autoload-end: + + + + +Additional resources +==================== +* `Dynamic Kernel Module System (DKMS) project on GitHub `_ + +* `Dell Linux Engineering Dynamic Kernel Module Support: From Theory to Practice `_ + +* `Linux Journal: Exploring Dynamic Kernel Module Support `_ + + +.. _`on GitHub`: https://github.com/clearlinux/distribution + +.. _`mixer tool`: https://clearlinux.org/features/mixer-tool + +.. _`Dynamic Kernel Module System (DKMS)`: https://github.com/dell/dkms diff --git a/source/clear-linux/guides/maintenance/kernel-modules.rst b/source/clear-linux/guides/maintenance/kernel-modules.rst index f5f76a9b..2339453a 100644 --- a/source/clear-linux/guides/maintenance/kernel-modules.rst +++ b/source/clear-linux/guides/maintenance/kernel-modules.rst @@ -1,88 +1,125 @@ .. _kernel-modules: -Add kernel modules -################## +Add kernel modules manually +########################### -Kernel modules are additional pieces of software capable of being inserted -into the Linux kernel to add functionality, such as a hardware driver. -Kernel modules may already be part of the Linux source tree (in-tree) or may -come from an external source, such as directly from a vendor (out-of-tree). +Kernel modules are additional pieces of software capable of being inserted +into the Linux kernel to add functionality, such as a hardware driver. Kernel +modules may already be part of the Linux source tree (in-tree) or may come +from an external source, such as directly from a vendor (out-of-tree). In cases where drivers beyond those enabled by default in |CL-ATTR| are -needed it may be necessary to: +needed it may be necessary to manually build out-of-tree modules. + +Out-of-tree kernel modules can be managed by `Dynamic Kernel Module System +(DKMS) `_ on |CL| for automatic rebuilding upon kernel +updates. Out-of-tree kernel modules can also be manually built and maintained +using the instructions in this document. + + .. contents:: :local: :depth: 1 :backlinks: top -Check if the module is available through |CL| -============================================= -Using an existing module is significantly easier to maintain and retains -signature verification of the |CL| kernel. For more information on |CL| +.. _kernel-modules-availability-begin: + +Kernel module availability in |CL| +================================== + +Before continuing, check if the kernel module you're looking for is already +available in |CL| or can be requested. + + +Check if the module is already available +---------------------------------------- + +|CL| comes with many upstream kernel modules available for use. If you +require a kernel module, be sure to check whether it is already available in +|CL| first. + +Using an existing module is significantly easier to maintain and retains +signature verification of the |CL| kernel. For more information on |CL| security practices, see the :ref:`security` page. -|CL| comes with many upstream kernel modules available for use. If -you require a kernel module, be sure to check whether it is already available in |CL| first. +You can search for kernel module file names, which end with the :file:`.ko` +file extension, using the :command:`swupd search` command. For example: +:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-search` for +more information. -You can search for kernel module file names, which end with the :file:`.ko` -file extension, using the :command:`swupd search` command. For example: -:command:`sudo swupd search ${module_name}.ko`. -See :ref:`swupd-search` for more information. Request the module be added to |CL| -=================================== +----------------------------------- -If the kernel module you need is already open source -(e.g. in the Linux upstream) and likely to be useful to others, -consider submitting a request to add or enable in the |CL| kernel. +If the kernel module you need is already open source (e.g. in the Linux +upstream) and likely to be useful to others, consider submitting a request to +add or enable in the |CL| kernel. Make enhancement requests to the |CL| distribution `on GitHub`_ . -Build and load an out-of-tree module -==================================== +.. _kernel-modules-availability-end: -In some cases you may need an out-of-tree kernel module that is not -available through |CL|. + +Build, install, and load an out-of-tree module +============================================== + +In some cases you may need an out-of-tree kernel module that is not available +through |CL|. + + +Prerequisites +------------- You can build and load out-of-tree kernel modules, however you must: -* disable secure boot -* disable kernel module integrity checking -* build the module against new versions of the Linux kernel +* Disable Secure Boot. +* Disable kernel module integrity checking. +* Have a kernel module package in the form of source code. +* Rebuild the module against new versions of the Linux kernel. .. note:: Any time the kernel is upgraded on your Clear Linux system, you will need to rebuild your out-of-tree modules. -This approach works well for individual development or testing. -For a more scalable and customizable approach, consider using the -`mixer tool`_ to provide a custom kernel and updates. +This approach works well for individual development or testing. For a more +scalable and customizable approach, consider using the `mixer tool`_ to +provide a custom kernel and updates. -Build kernel module -------------------- -#. From a |CL| system, ensure you are running the *native* kernel. - Currently only the native kernel is enabled to build and load - out-of-tree modules. +Build and install kernel module +------------------------------- + +#. Determine which kernel variant is running on |CL|. In the example below, + the *native* kernel is in use. .. code-block:: bash $ uname -r - 4.XX.YY-ZZZZ.native + 5.XX.YY-ZZZZ.native - Ensure *.native* is in the kernel name - -#. Install the `linux-dev` bundle to obtain the kernel headers, which are - required for compiling kernel modules. +#. Install the kernel dev bundle corresponding to the installed kernel. The + kernel dev bundle contains the kernel headers, which are required for + compiling kernel modules.For example: + + * `linux-dev` for developing against the native kernel. + * `linux-lts-dev` for developing against the LTS kernel. .. code-block:: bash sudo swupd bundle-add linux-dev #. Follow instructions from the kernel module source code to compile the - kernel module. + kernel module. For example: + + .. code-block:: bash + + curl -O http://.tar.gz + tar -xvf .tar.gz + cd / + cat README + Load kernel module @@ -124,28 +161,30 @@ Load kernel module .. code-block:: bash - sudo insmod ${path_to_module} + sudo insmod -Optional: Use `modprobe` to specify module options and aliases --------------------------------------------------------------- -Use :command:`modprobe` to load a module and set options. +.. _kernel-modules-autoload-begin: -Because :command:`modprobe` can add or remove more than one module, due to -modules having dependencies, a method of specifying what options are -to be used with individual modules is useful. This can be done with -configuration files under the :file:`/etc/modprobe.d` directory. +Optional: Specify module options and aliases +============================================ + +Use the :command:`modprobe` command to load a module and set options. + +Because :command:`modprobe` may add or remove more than one module due to +modules having dependencies, a method of specifying what options are to be +used with individual modules is useful. This can be done with configuration +files under the :file:`/etc/modprobe.d` directory. .. code-block:: bash sudo mkdir /etc/modprobe.d -All files underneath the :file:`/etc/modprobe.d` directory -that end with the :file:`.conf` extension specify module options to use when -loading. This can also be used to create convenient aliases for modules or -they can override the normal loading behavior altogether for those with -special requirements. +All files underneath the :file:`/etc/modprobe.d` directory that end with the +:file:`.conf` extension specify module options to use when loading. This can +also be used to create convenient aliases for modules or they can override the +normal loading behavior altogether for those with special requirements. You can find more info on module loading in the modprobe.d manual page: @@ -154,18 +193,18 @@ You can find more info on module loading in the modprobe.d manual page: man modprobe.d Optional: Configure kernel modules to load at boot --------------------------------------------------- +================================================== -Use the :file:`/etc/modules-load.d` configuration directory to -specify kernel modules to load automatically at boot. +Use the :file:`/etc/modules-load.d` configuration directory to specify kernel +modules to load automatically at boot. .. code-block:: bash sudo mkdir /etc/modules-load.d -All files underneath the :file:`/etc/modules-load.d` directory -that end with the :file:`.conf` extension contain a list of module names -of aliases (one per line) to load at boot. +All files underneath the :file:`/etc/modules-load.d` directory that end with +the :file:`.conf` extension contain a list of module names of aliases (one per +line) to load at boot. You can find more info on module loading in the modules-load.d manual page: @@ -173,5 +212,9 @@ You can find more info on module loading in the modules-load.d manual page: man modules-load.d + +.. _kernel-modules-autoload-end: + + .. _`on GitHub`: https://github.com/clearlinux/distribution .. _`mixer tool`: https://clearlinux.org/features/mixer-tool diff --git a/source/clear-linux/guides/maintenance/mixin.rst b/source/clear-linux/guides/maintenance/mixin.rst index 75e2c488..37e47ef8 100644 --- a/source/clear-linux/guides/maintenance/mixin.rst +++ b/source/clear-linux/guides/maintenance/mixin.rst @@ -1,167 +1,191 @@ .. _mixin: -Create and add custom bundles to your upstream Clear Linux system -################################################################# +mixin +##### -|CL-ATTR| offers many curated bundles that you can install on your system to -create your desired capabilities. If the available upstream bundles do not -meet your needs, you can create and add your own custom bundles to your -system using one of two methods. Note: Upstream refers to the official -version of |CL|. +mixin is a tool provided in the |CL-ATTR| that allows users to add custom +content to their client systems and still receive updates from their upstream OS +vendor. -The first method is to use the :ref:`mixer tool` to create your own -|CL| image and add your bundles to it. Mixing your own |CL| image can -give you great control and flexibility; however, you must act as an -:abbr:`OSV (Operating System Vendor)` and maintain your releases and -updates because you have forked from upstream. +.. contents:: + :local: + :depth: 1 -The second method is to use the :command:`mixin` tool, which also -makes use of mixer to create custom bundles that you can add to your -upstream |CL| system. This simpler method provides a “light” forking from -upstream, which means you can continue to get upstream bundles and updates. -If needed, you can easily revert your system back to the upstream version. +Description +*********** -This guide shows you how to accomplish the second method by following these -steps: +mixin uses the mixer tool to generate a local update for client systems. With +the mixin tool, a user can add remote RPM repositories or local RPMs and mix +them into their update stream, while continuing to get upstream bundles and +updates. The metadata generated from the mixin tool is merged with the upstream +metadata to provide a single source of update content, which swupd uses to +perform updates. -#. Set up the workspace. -#. Copy your custom RPM package to the workspace. -#. Create a bundle with your custom RPM package. -#. Migrate your |CL| system to your custom mix. -#. Add your custom bundle to your system. -#. Optional: Revert your system back to 100% upstream. +The mixin tool is included in the :command:`mixer` bundle. -Set up the workspace -******************** +How to use +********** -#. Install the mixer bundle to enable mixer. +Learn the mixin tool set up and workflow. + +.. contents:: + :local: + :depth: 1 + +Prerequisites +============= + +Install the :command:`mixer` bundle to add the mixin tool. Refer to +`Install a bundle`_ for more details. + +Workflow +======== + +The following steps show how to create and add a custom bundle with the mixin +tool: + +#. Add or create a new repo(s) + + mixin pulls packages to build your custom bundle from locations referred to + as repos. There are two default repos for mixin: + + * upstream + * local + + Additional repos can be added, such as other locations on your local system + or remote repos. + + RPMs must be built specifically for |CL| in order for them to work properly. + Refer to :ref:`autospec` for instruction on creating RPMs for |CL|. + +#. Create a custom bundle with desired RPMs + + Add the desired packages to your new bundle and build the bundle. By default, + the bundle will be named after its parent repo. + + The first time you build the bundle, mixer will create a new OS version by + taking your current upstream |CL| version and multiplying it by 1000. For + example, if your upstream version is 27650, your custom version will be + 27650000. For each subsequent call to mixin, mixer will increment the version + by 10. + + View the `mixin man page`_ for more information on mixin commands. + +#. Update system to make custom bundle available + + Update your system using swupd to make your custom bundle accessible. + + When you first create your mix, you will have to do a one-time migration to + your custom mix as part of the update. After you migrate, the system version + switches over to your last custom version number as noted in the previous + step. As long as you remain on your custom version of |CL| you can continue + to create and add new bundles to your mix with no extra migration step. + +#. Install custom bundles + + Install your custom bundle using the normal swupd :command:`bundle-add` + command. + + View the `swupd man page`_ for more information on swupd commands. + +Examples +******** + +Complete all `Prerequisites`_ before using these examples. + +Example 1: Add custom helloclear bundle +======================================= + +This example shows the basic steps of adding a custom bundle from a local repo. + +#. Check that :command:`helloclear` does not exist on your system: + + .. code-block:: bash + + helloclear .. code-block:: console - $ sudo swupd bundle-add mixer + helloclear: command not found -#. Create the workspace. +#. Follow the "Build a new RPM" example from :ref:`autospec` to create a new + `helloclear` RPM. + + The resulting RPMs are in `~/clearlinux/packages/helloclear/rpms`. + +#. Create a new repo. + + #. Create a local repo folder and copy the new `helloclear` RPM files into + the repo: + + .. code-block:: bash + + mkdir ~/mixin-repo + cp ~/clearlinux/packages/helloclear/rpms/helloclear-v1.0-1.x86_64.rpm ~/mixin-repo + cp ~/clearlinux/packages/helloclear/rpms/helloclear-bin-v1.0-1.x86_64.rpm ~/mixin-repo + + #. Create the repo data: + + .. code-block:: bash + + cd ~/mixin-repo + createrepo_c . + + #. Add the repo name: + + .. code-block:: bash + + sudo mixin repo add mylocalrepo file://$HOME/mixin-repo/ + +#. Create custom bundle with the new `helloclear` RPM. Add `helloclear` to the + :command:`helloclear-bundle` bundle and build the bundle: + + .. code-block:: bash + + sudo mixin package add helloclear --bundle helloclear-bundle + sudo mixin build + +#. Migrate your |CL| to your custom mix. Check your version before and after the + update to see the switch to your custom mix: + + .. code-block:: bash + + sudo swupd check-update + sudo swupd update --migrate + sudo swupd check-update + +#. Install your custom bundle. Check that the `helloclear-bundle` is now + available and install it to your system: + + .. code-block:: bash + + sudo swupd bundle-list -a | grep helloclear-bundle + sudo swupd bundle-add helloclear-bundle + +#. Test for `helloclear` again to see that it is installed: + + .. code-block:: bash + + helloclear + +#. Revert your system back to upstream (optional). This example reverts back to + upstream version 27650: .. code-block:: console - $ sudo mkdir -p /usr/share/mix/local-rpms + sudo swupd verify --fix --picky --force -m 27650 -C /usr/share/clear/update-ca/Swupd_Root.pem + sudo swupd clean --all + sudo swupd check-update -Copy your custom RPM package to the workspace -********************************************* +Related topics +************** -.. note:: +* :ref:`About mixer ` +* :ref:`mixer` +* :ref:`autospec-about` +* :ref:`bundles-about` +* :ref:`swupd-about` - You cannot simply use RPMs from other Linux distros on |CL|. You must - build RPMs specifically for |CL| in order for them to work properly. - Follow the instructions on how to build RPMs found at the - `Developer tooling framework for Clear Linux`_. - -If you have a local RPM you want to add to your mix you can do so by copying -your RPM package to the workspace. - -.. code-block:: console - - $ sudo cp [RPM] /usr/share/mix/local-rpms - -Alternatively, you can add a remote RPM repository by running the following -command. - -.. code-block:: console - - $ sudo mixin repo add [repo-name] [repo-url] - -Create a bundle with your custom RPM package -******************************************** - -Use the :command:`mixin` command to create a bundle with the RPM -package. - -.. code-block:: console - - $ sudo mixin package add [package-name] [--bundle bundle-name] [--build] - -This command will add package-name to a bundle that is named after its parent -repository. For example, if the RPM was provided locally, it will be added to -the 'local' bundle. If it came from a repo that was added with -:command:`mixin repo add`, it will be added to a bundle named after the -repo-name. If the `--bundle bundle-name` flag is provided, the package will -be added to `bundle-name` instead. The `--build` flag tells :command:`mixin` -to run a `mixer` build after adding the package. - -To add more than one RPM to your previously-created bundle, repeat -the :command:`mixin package add` command and change the package name. Do not -add the `--build` flag until all packages have been added. Once done adding -packages, run the following to create your local mix. - -.. code-block:: console - - $ sudo mixin build - -.. note:: - - * The first time you run the :command:`mixin build` command, mixer - creates a new OS version by taking your current upstream |CL| version - and multiplying it by 1000. For example, if your upstream version is - 21530, your custom version will be 21530000. For each subsequent call - to mixin, mixer will increment the version by 10. For example, - 21530010, 21530020, etc. - -Migrate your Clear Linux system to your custom mix -************************************************** - -Before you can use your custom bundle, you must migrate your |CL| system -to your custom mix to make the bundle accessible. - -.. code-block:: console - - $ sudo swupd update --migrate - -After you migrate, the version of your |CL| system switches over to your -last custom version number as noted in the previous section. - -You can continue to create new bundles with :command:`mixin` -while you are in your custom version of |CL|. You do not need to migrate -again. However, you must run :command:`swupd update` again to update your -system in order to make those bundles visible. - -Add your custom bundle to your system -************************************* - -#. Get a listing of your newly-created bundle. - - .. code-block:: console - - $ sudo swupd bundle-list -a - - The listing includes all upstream bundles. - -#. Add your bundle. - - .. code-block:: console - - $ sudo swupd bundle-add [bundle-name] - -.. note:: - - You can also update your system to the latest upstream version using - this command: - - .. code-block:: console - - $ sudo swupd update - -Optional: Revert your system back to 100% upstream -************************************************** - -If you want to revert your |CL| system back to the official upstream -version, use this command: - -.. code-block:: console - - $ sudo swupd verify --fix --force --picky -m [upstream-version-number] -C /usr/share/clear/update-ca/Swupd_Root.pem - -After the command completes, all custom RPMs and bundles are unavailable -because :file:`/usr/share/mix` is deleted as part of the reversion process. - -.. _Developer tooling framework for Clear Linux: - https://github.com/clearlinux/common +.. _mixin man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixin.1.rst +.. _swupd man page: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst +.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/swupd-guide.rst b/source/clear-linux/guides/maintenance/swupd-guide.rst index 0200ba2a..686c7a1f 100644 --- a/source/clear-linux/guides/maintenance/swupd-guide.rst +++ b/source/clear-linux/guides/maintenance/swupd-guide.rst @@ -49,7 +49,7 @@ Current OS version and update server info: .. code-block:: console Installed version: 23330 - Version URL: https://download.clearlinux.org/update/ + Version URL: https://cdn.download.clearlinux.org/update/ Content URL: https://cdn.download.clearlinux.org/update/ Enable or disable automatic updates diff --git a/source/clear-linux/guides/maintenance/validate-signatures.rst b/source/clear-linux/guides/maintenance/validate-signatures.rst index 54bbfe4d..b506ba3a 100644 --- a/source/clear-linux/guides/maintenance/validate-signatures.rst +++ b/source/clear-linux/guides/maintenance/validate-signatures.rst @@ -26,11 +26,11 @@ used for illustrative purposes. You may use any image of |CL| you choose. .. code-block:: console # Image - curl -O https://download.clearlinux.org/current/clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz + curl -O https://cdn.download.clearlinux.org/current/clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz # Signature of SHA512 sum of image - curl -O https://download.clearlinux.org/current/clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig + curl -O https://cdn.download.clearlinux.org/current/clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig # Certificate - curl -O https://download.clearlinux.org/releases/$(curl https://download.clearlinux.org/latest)/clear/ClearLinuxRoot.pem + curl -O https://cdn.download.clearlinux.org/releases/$(curl https://cdn.download.clearlinux.org/latest)/clear/ClearLinuxRoot.pem #. Generate the SHA256 sum of the |CL| certificate. @@ -49,7 +49,7 @@ used for illustrative purposes. You may use any image of |CL| you choose. .. code-block:: console - sha512sum clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz > sha512sum.out + sha512sum clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz > sha512sum.out #. Ensure the signature of the SHA512 sum of the image was created using the |CL| certificate. This validates the image is trusted and it has not @@ -57,7 +57,7 @@ used for illustrative purposes. You may use any image of |CL| you choose. .. code-block:: console - openssl smime -verify -purpose any -in clear-$(curl https://download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig -inform der -content sha512sum.out -CAfile ClearLinuxRoot.pem + openssl smime -verify -purpose any -in clear-$(curl https://cdn.download.clearlinux.org/latest)-installer.img.xz-SHA512SUMS.sig -inform der -content sha512sum.out -CAfile ClearLinuxRoot.pem .. note:: @@ -83,11 +83,11 @@ these steps manually when performing a ``swupd update``. .. code-block:: console # MoM - curl -O https://download.clearlinux.org/update/$(curl https://download.clearlinux.org/latest)/Manifest.MoM + curl -O https://cdn.download.clearlinux.org/update/$(curl https://cdn.download.clearlinux.org/latest)/Manifest.MoM # Signature of MoM - curl -O https://download.clearlinux.org/update/$(curl https://download.clearlinux.org/latest)/Manifest.MoM.sig + curl -O https://cdn.download.clearlinux.org/update/$(curl https://cdn.download.clearlinux.org/latest)/Manifest.MoM.sig # Swupd certificate - curl -O https://download.clearlinux.org/releases/$(curl https://download.clearlinux.org/latest)/clear/Swupd_Root.pem + curl -O https://cdn.download.clearlinux.org/releases/$(curl https://cdn.download.clearlinux.org/latest)/clear/Swupd_Root.pem #. Generate the SHA256 sum of the Swupd certificate. diff --git a/source/clear-linux/guides/network/dpdk.rst b/source/clear-linux/guides/network/dpdk.rst index 112dfe13..51f9d202 100644 --- a/source/clear-linux/guides/network/dpdk.rst +++ b/source/clear-linux/guides/network/dpdk.rst @@ -262,7 +262,7 @@ machines control the NICs on the host. .. code-block:: bash - sudo curl -O https://download.clearlinux.org/image/start_qemu.sh + sudo curl -O https://cdn.download.clearlinux.org/image/start_qemu.sh #. Download a bare-metal image of |CL| and rename it as :file:`clear.img`. @@ -338,7 +338,7 @@ machines control the NICs on the host. #. Run the :file:`start_qemu.sh` script. -.. _13330: https://download.clearlinux.org/releases/13330/ +.. _13330: https://cdn.download.clearlinux.org/releases/13330/ .. _DPDK project: http://dpdk.org .. _dpdk.org NICs: http://dpdk.org/doc/nics .. _pktgen tar package: http://dpdk.org/browse/apps/pktgen-dpdk/refs diff --git a/source/clear-linux/guides/network/ipxe-install.rst b/source/clear-linux/guides/network/ipxe-install.rst index f16ca06b..1e12fd3c 100644 --- a/source/clear-linux/guides/network/ipxe-install.rst +++ b/source/clear-linux/guides/network/ipxe-install.rst @@ -103,8 +103,8 @@ To set up |CL| manually, perform the steps below. sudo mkdir -p $ipxe_root sudo curl -o /tmp/clear-pxe.tar.xz \ - https://download.clearlinux.org/current/clear-$(curl \ - https://download.clearlinux.org/latest)-pxe.tar.xz + https://cdn.download.clearlinux.org/current/clear-$(curl \ + https://cdn.download.clearlinux.org/latest)-pxe.tar.xz sudo tar -xJf /tmp/clear-pxe.tar.xz -C $ipxe_root sudo ln -sf $(ls $ipxe_root | grep 'org.clearlinux.*') $ipxe_root/linux diff --git a/source/clear-linux/tutorials/telemetry-backend/figures/telemetry-backend-1.png b/source/clear-linux/guides/telemetrics/figures/telemetry-backend-1.png similarity index 100% rename from source/clear-linux/tutorials/telemetry-backend/figures/telemetry-backend-1.png rename to source/clear-linux/guides/telemetrics/figures/telemetry-backend-1.png diff --git a/source/clear-linux/tutorials/telemetry-backend/figures/telemetry-backend-2.png b/source/clear-linux/guides/telemetrics/figures/telemetry-backend-2.png similarity index 100% rename from source/clear-linux/tutorials/telemetry-backend/figures/telemetry-backend-2.png rename to source/clear-linux/guides/telemetrics/figures/telemetry-backend-2.png diff --git a/source/clear-linux/guides/telemetrics/figures/telemetry-e2e.png b/source/clear-linux/guides/telemetrics/figures/telemetry-e2e.png new file mode 100644 index 00000000..75868291 Binary files /dev/null and b/source/clear-linux/guides/telemetrics/figures/telemetry-e2e.png differ diff --git a/source/clear-linux/guides/telemetrics/telem-guide.rst b/source/clear-linux/guides/telemetrics/telem-guide.rst new file mode 100644 index 00000000..21e6c68d --- /dev/null +++ b/source/clear-linux/guides/telemetrics/telem-guide.rst @@ -0,0 +1,629 @@ +.. _telem-guide: + +Telemetrics +########### + +Telemetrics in |CL-ATTR| is a client and server solution used to collect data from running |CL| systems to help quickly identify and fix bugs in the OS. Both client and server are customizable, and an API is available on the client side for instrumenting your code for debug and analysis. + +.. contents:: + :local: + :depth: 1 + + +Description +************* + +Telemetry, one of the key features of |CL|, enables developers to observe and proactively address issues in the OS before end users are impacted. + +Telemetrics is a combination word made from: + +* Telemetry, which is sensing and reporting data. +* Analytics, which is using visualization and statistical inferencing to make sense of the reported data. + +|CL| telemetry reports system-level debug/crash information using specialized probes. The probes monitor system tasks such as swupd, kernel oops, machine error checks, and the BIOS error report table for unhandled hardware failures. Telemetry enables real-time issue reporting to allow system developers to quickly focus on an issue and monitor corrective actions. + +|CL| telemetry is fully customizable and can be used during software development for debugging purposes. You can use the libtelemetry library in your code to create custom telemetry records. You can also use the telem-record-gen utility in script files for light touch record creation where instrumenting code files doesn't make sense. + +Architecture +============ + +|CL| telemetry has two fundamental components, which are shown in figure 1: + +* Client: generates and delivers records to the backend server via the network + +* Backend: receives records sent from the client and displays the cumulative content through a specialized web interface. + +.. figure:: figures/telemetry-e2e.png + :alt: Figure 1, Telemetry Architecture + + Figure 1: :guilabel:`|CL| Telemetry Architecture` + +The telemetry client provides the front end of the telemetrics solution and includes the following components: + +* telemprobd, a daemon that receives and prepares telemetry records from probes and spools them to disk. +* telempostd, a daemon that manages spooled telemetry records and delivers these records according to configurable settings. +* probes, that collect specific types of data from the operating system. +* libtelemetry, the API that telemetrics probes use to create records. + +The telemetry backend provides the server-side component of the telemetrics solution and consists of: + +* Nginx web server. +* Two Flask apps: + * Collector, an ingestion web app for records received from client probes. + * TelemetryUI, a web app that exposes different views to visualize the telemetry data. +* PostgreSQL as the underlying database server. + +.. note:: + + The default telemetry backend server is hosted by the Intel |CL| development team and is not viewable outside the Intel firewall. To collect your own records, you must set up your own telemetry backend server. + +.. note:: + + The telemetry functionality adheres to `Intel privacy policies`_ regarding the collection and use of :abbr:`PII (Personally Identifiable Information)` and is open source. + Specifically, no intentionally identifiable information about the user or system owner is collected. + +How To Use +********** + +From a workflow perspective, the |CL| telemetrics system is straightforward. On the client side, the main decisions after installation and enabling telemetry concern what to do with the record data generated by the probes. You can send the data to the default or a custom backend server, keep the data local to the system, or both. The backend server has a more complex setup, but once it's running, it is simple to use and configure. + +This section walks through some of the possible scenarios for configuring the |CL| telemetrics system, and suggests which make sense according to your needs. + +Scenarios +========= +#. Enable telemetry + + Before probes can generate records, the telemetry client daemons must be enabled. You can configure the client before enabling by creating a custom :file:`telemetrics.conf` file that you place in the :file:`/etc/telemetrics` directory. If you choose to use the default settings, records will be sent to the telemetrics backend server managed by the |CL| development team at Intel. + +#. Save record data locally + + You can configure the telemetry client to save records locally. This is convenient when you want instant feedback during a development cycle, or to track system issues if you believe there is a machine specific problem. The client can be set to not send records at all, or to both keep the records locally and send to the backend server. + +#. Set up a server to collect data + + Whether you are managing a network of |CL| systems or you don't want to send records to the default telemetry server, you can set up a backend server to collect your records. The backend server can be installed on any Linux system and will give you the same dashboard as the default server. + + +#. Instrument your code with the libtelemetry API + + The ``telemetrics`` bundle includes the libtelemetry C library, which exposes an API used by the telemprobd and telempostd daemons. You can use these in your applications as well. The API documentation is found in the :file:`telemetry.h` file in `Telemetrics client`_ repository. + + +Examples +******** + +.. contents:: + :local: + :depth: 1 + +Enable or Disable Telemetry +=========================== + +#. Enabling during installation + + During the initial installation of |CL|, you are requested to join the stability enhancement program and allow |CL| to collect anonymous reports to improve system stability. If you choose not to join this program, then the telemetry software bundle is not added to your system. Choosing to join will automatically enable telemetry on your system after installation is commplete. + +#. Enabling after install + + To start telemetry on your system, run the following command: + + .. code-block:: bash + + sudo telemctl start + + This enables and starts the :command:`telemprobd` and :command:`telempostd` daemons. Your system will begin to send telemetry data to the server defined in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not exist, the :command:`telemprobd` and :command:`telempostd` daemons will use the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. + +#. Disabling after install + + To disable both of the telemetry daemons, run the following command: + + .. code-block:: bash + + sudo telemctl stop + +#. Opt in to telemetry + + To opt-in to the telemetry services, simply enter the opt-in command and start the service: + + .. code-block:: bash + + sudo telemctl opt-in + + This removes the :file:`/etc/telemetrics/opt-out` file, if it exists, and starts the telemetry services. + + .. note:: + + To opt-in but not immediately start telemetry services, you will need to run the command :command:`sudo telemctl stop` after the :command:`opt-in` command is entered. Once you are ready to start the service, enter the command :command:`sudo telemctl start`. + +#. Opt out of telemetry + + To stop sending telemetrics data from your system, opt out of the telemetry service: + + .. code-block:: bash + + sudo telemctl opt-out + + This creates the file :file:`/etc/telemetrics/opt-out` and stops the telemetry services. + + +Saving Data Locally +=================== + +This example requires |CL| to be installed, and telemetry to be enabled on the system. + +To change how records are managed, we will be copying the default :file:`/usr/share/defaults/telemetrics/telemetrics.conf` file to :file:`/etc/telemetrics/telemetrics.conf` and editing it. The changes in the :file:`/etc/telemetrics/telemetrics.conf` file will override the defaults in the :file:`/usr/share/defaults/telemetrics/telemetrics.conf` file. You may need ``root`` permissions to create and edit files in :file:`/etc`. For each example, and for any time you make changes to the configuration file, you will need to restart the client daemons to pick up the changes: + +.. code-block:: bash + + sudo telemctl restart + + +The :command:`telemctl journal` command gives you access to features and options of the telemetry journal to assist with system analytics and debug. :command:`telemctl journal` has a number of options to help filter records. Use :command:`-h` or :command:`--help` to view usage options. + + +#. Keep local copy and send records to backend server + + To keep a local copy of the telemetry record and also send it on to the backend server, we will need to change the :guilabel:`record_retention_enabled` configuration key value to :guilabel:`true`. + + +#. Keep all records -- don't send to backend server + + To keep records on the system without sending them to a backend server, set the :guilabel:`record_server_delivery_enabled` key value to :guilabel:`false`. Note that you will also need to ensure the :guilabel:`record_retention_enabled` configuration key value is set to :guilabel:`true` or the system will not keep local copies. + +#. Keep and send records to custom server + + This assumes you have set up a custom server according to the next example. + + The server is identified by the :guilabel:`server` setting, and by default records will be sent to the |CL| server :guilabel:`server=https://clr.telemetry.intel.com/v2/collector`. To change this, you can use an IP address or fully qualified domain name. + + + +Set up a backend server to collect telemetry records +==================================================== + +For this example, start with a clean installation of |CL| on a new system using the :ref:`bare-metal-install` getting started guide and: + +#. Join the :guilabel:`Stability Enhancement Program` to install and + enable the telemetrics components. +#. Select the manual installation method with the following settings: + + * Set the hostname to :guilabel:`clr-telem-server`, + * Create an administrative user named :guilabel:`clear` and add this user + to sudoers + +#. Log in with your administrative user, from your :file:`$HOME` directory, run :command:`git` to clone the :guilabel:`telemetrics-backend` repository into the :file:`$HOME/telemetrics-backend` directory: + + .. code-block:: console + + git clone https://github.com/clearlinux/telemetrics-backend + + .. note:: + + You may need to set up the :envvar:`https_proxy` environment variable if you have issues reaching github.com. + +#. Change your current working directory to :file:`telemetrics-backend/scripts`. +#. We will install the telemetrics backend with the :file:`deploy.sh` script file. We will set the following options and leave the remainder as default: + + * *-a install* to perform an install + * *-d clr* to install to a |CL| distro + * *-H localhost* to set the domain to localhost + + .. caution:: + The :file:`deploy.sh` shell script has minimal error checking and makes + several changes to your system. Be sure that the options you define on the cmdline are correct before proceeding. + +#. Run the shell script from the :file:`$HOME/telemetrics-backend/scripts` directory: + + .. code-block:: console + + ./deploy.sh -H localhost -a install -d clr + + + + The script will start and list all the defined options and prompt you for the :guilabel:`PostgreSQL` database password + + .. code-block:: console + + Options: + host: localhost + distro: clr + action: install + repo: https://github.com/clearlinux/telemetrics-backend + source: master + type: git + DB password: (default: postgres): + +#. For the :guilabel:`DB password:`, press the :kbd:`Enter` key to accept the default password `postgres`. + + .. note:: + + The :file:`deploy.sh` script uses :command:`sudo` to run commands and you may be prompted to enter your user password at any time while the script is executing. If this occurs, enter your user password to execute the :command:`sudo` command. + + +#. Once all the server components have been installed you are prompted to enter the :guilabel:`PostgreSQL` database password to change it as illustrated below: + + .. code-block:: console + + Enter password for 'postgres' user: + New password: + Retype new password: + passwd: password updated successfully + + Enter `postgres` for the current value of the password and then enter a new password, retype it to verify the new password and the :guilabel:`PostgreSQL` database password will be updated. + +#. Once the installation is complete you can use your web browser to view the new server by opening the browser on the system and typing in ``localhost`` in the address bar. You should see a web page similar to the one shown in figure 1: + + .. figure:: figures/telemetry-backend-1.png + :alt: Telemetry UI + + Figure 1: :guilabel:`Telemetry UI` + + + +Create records with telem-record-gen +==================================== + +The telemetrics bundle provides a record generator tool called ``telem-record-gen``. This tool can be used to create records from shell scripts or the command line when writing a probe in C is not desirable. Records are sent to the backend server, and can also be echoed to stdout. + +There are three ways to supply the payload to the record. + +#. On the command line, use the :command:`-p ` option: + + .. code-block:: bash + + telem-record-gen -c a/b/c -n -o -p 'payload goes here' + + .. code-block:: console + + record_format_version: 4 + classification: a/b/c + severity: 1 + machine_id: FFFFFFFF + creation_timestamp: 1539023189 + arch: x86_64 + host_type: innotek GmbH|VirtualBox|1.2 + build: 25180 + kernel_version: 4.14.71-404.lts + payload_format_version: 1 + system_name: clear-linux-os + board_name: VirtualBox|Oracle Corporation + cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz + bios_version: VirtualBox + event_id: 2236710e4fc11e4a646ce956c7802788 + + payload goes here + +#. Specify a file that contains the payload with the option :command:`-P path/to/file`. + + .. code-block:: bash + + telem-record-gen -c a/b/c -n -o -P ./payload_file.txt + + .. code-block:: console + + record_format_version: 4 + classification: a/b/c + severity: 1 + machine_id: FFFFFFFF + creation_timestamp: 1539023621 + arch: x86_64 + host_type: innotek GmbH|VirtualBox|1.2 + build: 25180 + kernel_version: 4.14.71-404.lts + payload_format_version: 1 + system_name: clear-linux-os + board_name: VirtualBox|Oracle Corporation + cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz + bios_version: VirtualBox + event_id: d73d6040afd7693cccdfece479df9795 + + payload read from file + +#. If the :command:`-p` or :command:`-P` options are absent, the tool reads from stdin so you can use it in a :file:`heredoc` in scripts. + + .. code-block:: bash + + #telem-record-gen -c a/b/c -n -o << HEOF + payload read from stdin + HEOF + + .. code-block:: console + + record_format_version: 4 + classification: a/b/c + severity: 1 + machine_id: FFFFFFFF + creation_timestamp: 1539023621 + arch: x86_64 + host_type: innotek GmbH|VirtualBox|1.2 + build: 25180 + kernel_version: 4.14.71-404.lts + payload_format_version: 1 + system_name: clear-linux-os + board_name: VirtualBox|Oracle Corporation + cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz + bios_version: VirtualBox + event_id: 2f070e8e71679f2b1f28794e3a6c42ee + + payload read from stdin + + + +Instrument your code with the libtelemetry API +============================================== + +Prerequisites +------------- +Confirm that the telemetrics header file is located on the system at :file:`usr/include/telemetry.h` The `latest version`_ of the file can also be found on github for reference, but installing the `telemetry` bundle will install the header file that matches your |CL| version. + +#. Includes and variables + + You will need to include the following headers in your code to use the API: + + :: + + #define _GNU_SOURCE + #include + #include + #include + #include + + + Use the following code to create the variables we need to hold the data for the record we will be creating: + + :: + + uint32_t severity = 1; + uint32_t payload_version = 1; + char classification[30] = "org.clearlinux/hello/world"; + struct telem_ref *tm_handle = NULL; + char *payload; + int ret = 0; + + + + Severity: + | Type: uint32_t + | Value: Severity field value. Accepted values are in the range 1-4, with 1 being the lowest severity, and 4 being the highest severity. Values provided outside of this range are clamped to 1 or 4. [low, med, high, crit] + + Payload_version: + | Type: uint32_t + | Value: Payload format version. The only supported value right now is 1, which indicates that the payload is a freely-formatted (unstructured) string. Values greater than 1 are reserved for future use. + + Classification: + | Type: char array + | Value: It should have the form, DOMAIN/PROBENAME/REST: DOMAIN is the reverse domain to use as a namespace for the probe (e.g. org.clearlinux); PROBENAME is the name of the probe; and REST is an arbitrary value that the probe should use to classify the record. The maximum length for the classification string is 122 bytes. Each sub-category may be no longer than 40 bytes long. Two / delimiters are required. + + Tm_handle: + | Type: Telem_ref struct pointer + | Value: Struct pointer declared by the caller, The struct is initialized if the function returns success. + + Payload: + | Type: char pointer + | Value: The payload to set + + + +#. For this example, we'll set the payload to “hello” by using ``asprintf()`` + + :: + + if (asprintf(&payload, "hello\n") < 0) { + exit(EXIT_FAILURE); + } + + + + The functions ``asprintf()`` and ``vasprintf()`` are analogs of ``sprintf(3)`` and ``vsprintf(3)``, except that they allocate a string large enough to hold the output including the terminating null byte ('\0'), and return a pointer to it via the first argument. This pointer should be passed to ``free(3)`` to release the allocated storage when it is no longer needed. + +#. Create the new telemetry record + + The function ``tm_create_record()`` initializes a telemetry record and sets the severity and classification of that record, as well as the payload version number. The memory needed to store the telemetry record is allocated and should be freed with ``tm_free_record()`` when no longer needed. + + :: + + if ((ret = tm_create_record(&tm_handle, severity, classification, payload_version)) < 0) { + printf("Failed to create record: %s\n", strerror(-ret)); + ret = 1; + goto fail; + } + +#. Set the payload field of a telemetrics record + + The function ``tm_set_payload()`` attaches the provided telemetry record data to the telemetry record. The current maximum payload size is 8192b. + + :: + + if ((ret = tm_set_payload(tm_handle, payload)) < 0) { + printf("Failed to set record payload: %s\n", strerror(-ret)); + ret = 1; + goto fail; + } + free(payload); + + The ``free()`` function frees the memory space pointed to by ptr, which must have been returned by a previous call to ``malloc()``, ``calloc()``, or ``realloc()``. Otherwise, or if ``free(ptr)`` has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed. + +#. Send a record to the telemetrics daemon + + The function ``tm_send_record()`` delivers the record to the local ``telemprobd(1)`` service. Since the telemetry record was allocated by the program it should be freed with ``tm_free_record()`` when it is no longer needed. + + :: + + if ((ret = tm_send_record(tm_handle)) < 0) { + printf("Failed to send record to daemon: %s\n", strerror(-ret)); + ret = 1; + goto fail; + } else { + printf("Successfully sent record to daemon.\n"); + ret = 0; + } + fail: + tm_free_record(tm_handle); + tm_handle = NULL; + + return ret; + + +#. Full sample application with compiling flags + + Create a new file test.c add the following code. + + :: + + #define _GNU_SOURCE + #include + #include + #include + #include + + int main(int argc, char **argv) + { + uint32_t severity = 1; + uint32_t payload_version = 1; + char classification[30] = "org.clearlinux/hello/world"; + struct telem_ref *tm_handle = NULL; + char *payload; + + int ret = 0; + + if (asprintf(&payload, "hello\n") < 0) { + exit(EXIT_FAILURE); + } + + if ((ret = tm_create_record(&tm_handle, severity, classification, payload_version)) < 0) { + printf("Failed to create record: %s\n", strerror(-ret)); + ret = 1; + goto fail; + } + + if ((ret = tm_set_payload(tm_handle, payload)) < 0) { + printf("Failed to set record payload: %s\n", strerror(-ret)); + ret = 1; + goto fail; + } + + free(payload); + + if ((ret = tm_send_record(tm_handle)) < 0) { + printf("Failed to send record to daemon: %s\n", strerror(-ret)); + ret = 1; + goto fail; + } else { + printf("Successfully sent record to daemon.\n"); + ret = 0; + } + fail: + tm_free_record(tm_handle); + tm_handle = NULL; + + return ret; + } + + + + Compile with the gcc compiler, using this command: + + .. code-block:: bash + + gcc test.c -ltelemetry -o test_telem + + + Test to ensure the program is working: + + .. code-block:: bash + + ./test_telem + Successfully sent record to daemon. + + .. note:: + + A full example of the `heartbeat probe`_ in C is documented in the source code. + +Reference +********* + +.. contents:: + :local: + :depth: 1 + +The Telemetry API +================= + +Installing the ``telemetrics`` bundle includes the libtelemetry C library, which exposes an API used by the telemprobd and telempostd daemons. You can use these in your applications as well. The API documentation is found in the :file:`telemetry.h` file in `Telemetrics client`_ repository. + +Client Configuration +==================== + +The telemetry client will look for the configuration file located at :file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the file does not exist, the client will use the default configuration located at :file:`/usr/share/defaults telemetrics/telemetrics.conf`. To modify or customize the configuration, copy the file from :file:`/usr/share/defaults/telemetrics` to :file:`/etc/telemetrics` and edit it. + +Configuration Options +--------------------- +The client uses the following configuration options from the config file: + +* **server**: This specifies the web server to which telempostd sends the telemetry records. +* **socket_path**: This specifies the path of the unix domain socket that the telemprobd listens on for connections from the probes. +* **spool_dir**: This configuration option is related to spooling. If the daemon is not able to send the telemetry records to the backend server due to reasons such as the network availability, then it stores the records in a spool directory. This option specifies that path of the spool directory. This directory should be owned by the same user as the daemon. +* **record_expiry**: This is the time in minutes after which the records in the spool directory are deleted by the daemon. +* **spool_process_time**: This specifies the time interval in seconds that the daemon waits for before checking the spool directory for records. The daemon picks up the records in the order of modification date and tries to send the record to the server. It sends a maximum of 10 records at a time. If it was able to send a record successfully, it deletes the record from the spool. If the daemon finds a record older than the "record_expiry" time, then it deletes that record. The daemon looks at a maximum of 20 records in a single spool run loop. +* **rate_limit_enabled**: This determines whether rate-limiting is enabled or disabled. When enabled, there is a threshold on both records sent within a window of time, and record bytes sent within a window a time. +* **record_burst_limit**: This is the maximum amount of records allowed to be passed by the daemon within the record_window_length of time. If set to -1, the rate-limiting for record bursts is disabled. +* **record_window_length**: The time in minutes (0-59) that establishes the window length for the record_burst_limit. EX: if record_burst_window=1000 and record_window_length=15, then no more than 1000 records can be passed within any given fifteen minute window. +* **byte_burst_limit**: This is the maximum amount of bytes that can be + passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled. +* **byte_window_length**: This is the time, in minutes (0-59), that establishes the window length for the byte_burst_limit. +* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting threshold has been reached. Currently the options are 'drop' or 'spool', with spool being the default. If spool is chosen, records will be spooled and sent at a later time. +* **record_retention_enabled**: When this key is enabled (true) the daemon saves a copy of the payload on disk from all valid records. To avoid the excessive use of disk space only the latest 100 records are kept. The default value for this configuration key is false. +* **record_server_delivery_enabled**: This key controls the delivery of records to server; when enabled (default value), the record will be posted to the address in the configuration file. If this configuration key is disabled (false), records will not be spooled or posted to backend. This configuration key can be used in combination with record_retention_enabled to keep copies of telemetry records locally only. + + .. note:: + + Configuration options may change as the telemetry client evolves. + Please use the comments in the file itself as the most accurate + reference for configuration. + + +Client Run-time Options +======================= +The |CL| telemetry client provides an admin tool called :guilabel:`telemctl` for managing the telemetry services and probes. The tool is located in :file:`/usr/bin`. Running it with no argument results in the following: + +.. code-block:: bash + + sudo telemctl + +.. code-block:: console + + /usr/bin/telemctl - Control actions for telemetry services + stop Stops all running telemetry services + start Starts all telemetry services + restart Restarts all telemetry services + is-active Checks if telemprobd and telempostd are active + opt-in Opts in to telemetry, and starts telemetry services + opt-out Opts out of telemetry, and stops telemetry services + journal Prints telemetry journal contents. Use -h argument for more + options + +start/stop/restart +------------------ + +The commands to start, stop and restart the telemetry services manage all required services and probes on the system. There is no need to separately start/stop/restart the two client daemons **telemprobd** and **telempostd**. The **restart** command option will call **telemctl stop** followed by **telemctl start** . + +is-active +--------- + +The `is-active` option reports whether the two client daemons are active. This is useful to verify that the **opt-in** and **opt-out** options have taken effect, or to ensure that telemetry is functioning on the system. Note that both daemons are verified. + +.. code-block:: bash + + sudo telemctl is-active + +.. code-block:: console + + telemprobd : active + telempostd : active + + + +.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/ +.. _latest version: https://github.com/clearlinux/telemetrics-client/tree/master/src +.. _heartbeat probe: https://github.com/clearlinux/telemetrics-client/tree/master/src/probes/hello.c +.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html diff --git a/source/clear-linux/guides/telemetrics/telemctl.rst b/source/clear-linux/guides/telemetrics/telemctl.rst deleted file mode 100644 index 97db933a..00000000 --- a/source/clear-linux/guides/telemetrics/telemctl.rst +++ /dev/null @@ -1,65 +0,0 @@ -.. _telemctl: - -telemctl options -################ - -The |CL-ATTR| telemetry client provides an admin tool called ``telemctl`` -that is used for managing the telemetry services and probes. The tool is -located in :file:`/usr/bin`. Running it with no argument results in the -following: - -.. code-block:: bash - - sudo telemctl - -.. code-block:: console - - /usr/bin/telemctl - Control actions for telemetry services - stop Stops all running telemetry services - start Starts all telemetry services - restart Restarts all telemetry services - is-active Checks if telemprobd and telempostd are active - opt-in Opts in to telemetry, and starts telemetry services - opt-out Opts out of telemetry, and stops telemetry services - journal Prints telemetry journal contents. Use -h argument for more - options - -telemctl commands: -****************** - -start/stop/restart -================== - -The commands to start, stop and restart the telemetry services manage all -required services and probes on the system. There is no need to separately -start/stop/restart the two client daemons **telemprobd** and **telempostd**. -The **restart** command option will call **telemctl stop** followed by -**telemctl start** . - -is-active -========= - -The `is-active` option reports whether the two client daemons are active. -This is useful to verify that the **opt-in** and **opt-out** options have -taken effect, or to ensure that telemetry is functioning on the system. -Note that both daemons are verified. - -.. code-block:: bash - - sudo telemctl is-active - -.. code-block:: console - - telemprobd : active - telempostd : active - -.. include:: ./telemetry-enable.rst - :start-after: incl-opt-in-out-telemetry: - :end-before: incl-opt-in-out-telemetry-end: - -Next steps -========== - -Learn to read records: - -* :ref:`telemetry-journal` diff --git a/source/clear-linux/guides/telemetrics/telemetrics.rst b/source/clear-linux/guides/telemetrics/telemetrics.rst deleted file mode 100644 index 0a4d1cea..00000000 --- a/source/clear-linux/guides/telemetrics/telemetrics.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _telemetrics: - -Telemetry -######### - -The |CL-ATTR| telemetrics solution collects data from running |CL| systems -and helps to quickly identify and fix bugs in the OS. These guides will walk -you through setup, configuration, and customization of the telemetry client. The data collected from the client system is analyzed and presented by the -telemetry backend solution. For more details, learn how to -:ref:`telemetry-backend`. - -.. toctree:: - :maxdepth: 1 - - telemetry-enable - telemetry-config - telemctl - telemetry-journal - telemetry-api diff --git a/source/clear-linux/guides/telemetrics/telemetry-api.rst b/source/clear-linux/guides/telemetrics/telemetry-api.rst deleted file mode 100644 index d8a2fd0b..00000000 --- a/source/clear-linux/guides/telemetrics/telemetry-api.rst +++ /dev/null @@ -1,170 +0,0 @@ -.. _telemetry-api: - -Telemetry API -############# - -Installing the ``telemetrics`` bundle includes the libtelemetry C library, -which exposes an API used by the telemprobd and telempostd daemons. You -can use these in your applications as well. The API documentation is found -in the :file:`telemetry.h` file in `Telemetrics client`_ repository. - -Creating records with telem-record-gen -************************************** - -The telemetrics bundle also provides a record generator tool called -``telem-record-gen``. This tool can be used to create records from shell -scripts, etc., when writing a probe in C is not desirable. Records are sent -to the backend server, and can also be echoed to stdout. - -telem-record-gen usage -====================== - -.. code-block:: bash - - telem-record-gen [OPTIONS] - create and send a custom telemetry record - -.. code-block:: console - - Help Options: - -h, --help Show help options - - Application Options: - -V, --version Print the program version - -s, --severity Severity level (1-4) - (default 1) - -c, --class Classification level_1/level_2/level_3 (required) - -p, --payload Record body (max size = 8k) (required) - -P, --payload-file File to read payload from - -R, --record-version Version number for format of payload (default 1) - -e, --event-id Event id to use in the record - -o, --echo Echo record to stdout - -n, --no-post Do not post record just print - -The :command:`-c` and :command:`-p` options are required; defaults are -supplied for most other options. The maximum payload size is 8k -(8192 bytes). Excess is ignored, regardless of source (file/commandline/ -stdin). An empty payload is allowed, but even an empty payload must be -specified in one of the three ways shown below. - -telem-record-gen examples -========================= - -There are three ways to supply the payload to the record. - -#. On the command line, use the :command:`-p ` option: - - .. code-block:: bash - - telem-record-gen -c a/b/c -n -o -p 'payload goes here' - - .. code-block:: console - - record_format_version: 4 - classification: a/b/c - severity: 1 - machine_id: FFFFFFFF - creation_timestamp: 1539023189 - arch: x86_64 - host_type: innotek GmbH|VirtualBox|1.2 - build: 25180 - kernel_version: 4.14.71-404.lts - payload_format_version: 1 - system_name: clear-linux-os - board_name: VirtualBox|Oracle Corporation - cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz - bios_version: VirtualBox - event_id: 2236710e4fc11e4a646ce956c7802788 - - payload goes here - -#. Specify a file that contains the payload with the option - :command:'-P path/to/file'. - - .. code-block:: bash - - telem-record-gen -c a/b/c -n -o -P ./payload_file.txt - - .. code-block:: console - - record_format_version: 4 - classification: a/b/c - severity: 1 - machine_id: FFFFFFFF - creation_timestamp: 1539023621 - arch: x86_64 - host_type: innotek GmbH|VirtualBox|1.2 - build: 25180 - kernel_version: 4.14.71-404.lts - payload_format_version: 1 - system_name: clear-linux-os - board_name: VirtualBox|Oracle Corporation - cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz - bios_version: VirtualBox - event_id: d73d6040afd7693cccdfece479df9795 - - payload read from file - -#. If the :command:`-p` or :command:`-P` options are absent, the tool reads - from stdin so you can use it in a :file:`heredoc` in scripts. - - .. code-block:: bash - - telem-record-gen -c a/b/c -n -o << HEOF - payload read from stdin - HEOF - - .. code-block:: console - - record_format_version: 4 - classification: a/b/c - severity: 1 - machine_id: FFFFFFFF - creation_timestamp: 1539023621 - arch: x86_64 - host_type: innotek GmbH|VirtualBox|1.2 - build: 25180 - kernel_version: 4.14.71-404.lts - payload_format_version: 1 - system_name: clear-linux-os - board_name: VirtualBox|Oracle Corporation - cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz - bios_version: VirtualBox - event_id: 2f070e8e71679f2b1f28794e3a6c42ee - - payload read from stdin - - .. note:: - - Although only the classification and payload are specified, the tool supplies values for the remaining values. - -Telemetry records and the REST API -================================== - -If you have not configured the telemetry client to keep records locally, you -can view them using the Web UI of the server, or you can query them from the -server using the REST API provided by |CL| telemetrics. The API is -available at :file:`/api/records`, and when queried, returns a JSON -response that contains a list of records. There are several parameters for -filtering queries, similar to the filters available through the telemetryui Records view. - -* classification: The classification of the record -* severity: The severity of the record. Restricted to integer value -* machine_id: The id of the machine where this record was generated on -* build: The build on which the record was generated. Restricted to 256 - characters. -* created_in_days: causes the query to return records created after the last - given days -* created_in_sec: returns the records created after the last given seconds -* limit: The maximum number of records to be returned. - -Next Steps -========== - -* :ref:`telemetry-backend` -* `Telemetrics client`_ - -Related topics -============== - -* :ref:`telemetry-about` - -.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/ diff --git a/source/clear-linux/guides/telemetrics/telemetry-config.rst b/source/clear-linux/guides/telemetrics/telemetry-config.rst deleted file mode 100644 index d97b89bb..00000000 --- a/source/clear-linux/guides/telemetrics/telemetry-config.rst +++ /dev/null @@ -1,101 +0,0 @@ -.. _telemetry-config: - -Telemetry client configuration -############################## - -The telemetry client will look for the configuration file located at -:file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the -file does not exist, the client will use the default configuration located -at :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. To modify or -customize the configuration, copy the file from -:file:`/usr/share/defaults/telemetrics` to -:file:`/etc/telemetrics` and edit it. - -Configuration Options -********************* -The client uses the following configuration options from the config file: - -* **server**: This specifies the web server to which telempostd sends the - telemetry records. -* **socket_path**: This specifies the path of the unix domain socket that - the telemprobd listens on for connections from the probes. -* **spool_dir**: This configuration option is related to spooling. If the - daemon is not able to send the telemetry records to the backend server due - to reasons such as the network availability, then it stores the records in - a spool directory. This option specifies that path of the spool directory. - This directory should be owned by the same user as the daemon. - - - mkdir -p /var/spool/telemetry - - chown -R telemetry:telemetry /var/spool/telemetry - - systemctl restart telemprobd.service - -* **record_expiry**: This is the time in minutes after which the records in - the spool directory are deleted by the daemon. -* **spool_process_time**: This specifies the time interval in seconds that - the daemon waits for before checking the spool directory for records. The - daemon picks up the records in the order of modification date and tries to - send the record to the server. It sends a maximum of 10 records at a time. - If it was able to send a record successfully, it deletes the record from - the spool. If the daemon finds a record older than the "record_expiry" - time, then it deletes that record. The daemon looks at a maximum of 20 - records in a single spool run loop. -* **rate_limit_enabled**: This determines whether rate-limiting is enabled or - disabled. When enabled, there is a threshold on both records sent within a - window of time, and record bytes sent within a window a time. -* **record_burst_limit**: This is the maximum amount of records allowed to be - passed by the daemon within the record_window_length of time. If set to - -1, the rate-limiting for record bursts is disabled. -* **record_window_length**: The time in minutes (0-59) that - establishes the window length for the record_burst_limit. EX: if - record_burst_window=1000 and record_window_length=15, then no more than - 1000 records can be passed within any given fifteen minute window. -* **byte_burst_limit**: This is the maximum amount of bytes that can be - passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled. -* **byte_window_length**: This is the time, in minutes (0-59), that - establishes the window length for the byte_burst_limit. -* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting - threshold has been reached. Currently the options are 'drop' or 'spool', - with spool being the default. If spool is chosen, records will be spooled - and sent at a later time. -* **record_retention_enabled**: When this key is enabled (true) the daemon - saves a copy of the payload on disk from all valid records. To avoid the - excessive use of disk space only the latest 100 records are kept. The - default value for this configuration key is false. -* **record_server_delivery_enabled**: This key controls the delivery of - records to server; when enabled (default value), the record will be posted - to the address in the configuration file. If this configuration key is - disabled (false), records will not be spooled or posted to backend. This - configuration key can be used in combination with record_retention_enabled - to keep copies of telemetry records locally only. - - .. note:: - - Configuration options may change as the telemetry client evolves. - Please use the comments in the file itself as the most accurate - reference for configuration. - -Setting a static machine id -=========================== - -The machine id reported by the telemetry client is rotated every 3 days for -privacy reasons. If you wish to have a static machine id for testing -purposes, you can opt in by creating a static machine id file named -"opt-in-static-machine-id" under the directory :file:`/etc/telemetrics/`. -Where "unique machine ID" is your desired static machine ID. - -.. code-block:: bash - - sudo mkdir -p /etc/telemetrics - -.. code-block:: bash - - sudo echo "unique machine ID" > /etc/telemetrics/opt-in-static-machine-id - -.. note:: - - The machine id mentioned here is not the same as the system hostname. Learn how to :ref:`hostname`: - -Next steps -========== - -* :ref:`telemctl` diff --git a/source/clear-linux/guides/telemetrics/telemetry-enable.rst b/source/clear-linux/guides/telemetrics/telemetry-enable.rst deleted file mode 100644 index 7ae19481..00000000 --- a/source/clear-linux/guides/telemetrics/telemetry-enable.rst +++ /dev/null @@ -1,114 +0,0 @@ -.. _telemetry-enable: - -Enable telemetry -################ - -Telemetry enables developers to observe and proactively address issues on -|CL-ATTR| before end users are impacted. The telemetry functionality -is maintained in the ``telemetrics`` software bundle. - -.. note:: - - The telemetry functionality adheres to `Intel privacy policies`_ - regarding the collection and use of :abbr:`PII (Personally Identifiable - Information)` and is open source. Specifically, no intentionally - identifiable information about the user or system owner is collected. - -End users may enable or disable the telemetry component of |CL| or even -redirect where the records go if they wish to collect records for themselves. - -Install the telemetry software bundle -************************************* - -During the initial installation of |CL|, you are requested to join the -stability enhancement program and allow |CL| to collect anonymous reports -to improve system stability. If you choose not to join this program, then the -telemetry software bundle is not added to your system. - -To install the telemetry bundle, enter the following command as either the -root user or with :command:`sudo` privileges: - -.. code-block:: bash - - sudo swupd bundle-add telemetrics - -This adds the telemetrics-client to your system, and you will automatically -opt-in for the service. - -Enable telemetry -================ - -To start telemetry on your system, run the following command: - -.. code-block:: bash - - sudo telemctl start - -This enables and starts the :command:`telemprobd` and :command:`telempostd` -daemons. Your system will begin to send telemetry data to the server defined -in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not -exist, the :command:`telemprobd` and :command:`telempostd` daemons will use -the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. - -Disable telemetry -================= - -To disable both of the telemetry daemons, run the following command: - -.. code-block:: bash - - sudo telemctl stop - -.. _incl-opt-in-out-telemetry: - -Opt-in to telemetry -=================== - -To opt-in to the telemetry services, simply enter the opt-in -command and start the service: - -.. code-block:: bash - - sudo telemctl opt-in - -This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists, -and starts the telemetry services. - -.. note:: - - To opt-in but not immediately start telemetry services, you will need to - run the command :command:`sudo telemctl stop` after the :command:`opt-in` - command is entered. Once you are ready to start the service, enter the - command :command:`sudo telemctl start`. - -Opt-out of telemetry -==================== - -To stop sending telemetrics data from your system, opt out of the -telemetry service: - -.. code-block:: bash - - sudo telemctl opt-out - -This creates the file :file:`/etc/telemetrics/opt-out` and stops the -telemetry services. - -.. _incl-opt-in-out-telemetry-end: - -Remove the telemetry software bundle -==================================== - -To completely remove telemetrics from your system, use the :command:`swupd` -command to remove the telemetry software bundle: - -.. code-block:: bash - - sudo swupd bundle-remove telemetrics - -Next steps -========== - -* :ref:`telemetry-config` - -.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html diff --git a/source/clear-linux/guides/telemetrics/telemetry-journal.rst b/source/clear-linux/guides/telemetrics/telemetry-journal.rst deleted file mode 100644 index c0c4be80..00000000 --- a/source/clear-linux/guides/telemetrics/telemetry-journal.rst +++ /dev/null @@ -1,108 +0,0 @@ -.. _telemetry-journal: - -telemctl journal -################ - -The telemctl ``journal`` command gives you access to features and options of -the telemetry journal to assist with system analytics and debug. The -:command:`telemctl journal` has a number of options to help filter -records. Use :command:`-h` or :command:`--help` to view usage options. - -.. code-block:: bash - - sudo telemctl journal -h - -:: - - -r, --record_id Print record with specific record_id - -e, --event_id Print records with specific event_id - -c, --classification Print records with specific classification - -b, --boot_id Print records with specific boot_id - -i, --include_record Include record content - -V, --verbose Verbose output - -h, --help Display this help message - -Journal Output -************** - -To see the listing of records in the journal, run the command: - -.. code-block:: bash - - sudo telemctl journal -V - -This will produce output like the following: - -.. list-table:: **Table 1. Journal Output** - :widths: 10 30 20 20 20 - :header-rows: 1 - - * - Classification - - Time stamp - - Record ID - - Event ID - - Boot ID - - * - org.clearlinux/heartbeat/ping - - Fri 2018-09-21 00:00:57 UTC - - 269e8e4026e6aa440c4d2ed71e38efcd - - b3a51b5e62a008ed0b56d1740be67d48 - - 853a75aa-da3b-4356-a085-079abab3ffe1 - - * - org.clearlinux/hello/world - - Fri 2018-09-21 17:53:21 UTC - - b06c8d31adf5ccc7d5d3f8959d8d3e72 - - 57c64c79a9b911d68f4dab10a00267d7 - - 853a75aa-da3b-4356-a085-079abab3ffe1 - - * - org.clearlinux/crash/clr - - Fri 2018-09-21 17:57:59 UTC - - b62cd4278672ae3331cf121bc7a8e1c6 - - b6adb5751382c48eebb7ee007fe1790a - - 853a75aa-da3b-4356-a085-079abab3ffe1 - -Each line gives information about a distinct record. The :command:`-V` or -:command:`--verbose` option adds the header to identify the Classification, -Time Stamp, Record ID, Event ID and Boot ID for each record. The journal -feature can filter records according to the Classification, Record ID, Event -ID and Boot ID by using the :command:`-c`,:command:`-r`, :command:`-e` and -:command:`-b` options accordingly. - -Payload Information -******************** - -From the previous output, you may want to get more information about the -record with the "org.clearlinux/crash/clr" classification to help debug a -crash. You can use the :command:`-c` and :command:`-i` options to see the payload of the record, like this: - -.. code-block:: bash - - sudo telemctl journal -c org.clearlinux/crash/clr -i - -.. code-block:: console - - org.clearlinux/crash/clr Tue 2018-09-25 18:43:50 UTC 07ae583edbd13829965d67e9ba97d70c 69c600470769c841649266178375d67e d32c13d1-fda0-49c6-8431-e6c5b29cbefa - Process: /usr/bin/bash - PID: 685 - Signal: 11 - - Backtrace (TID 685): - #0 kill() - [libc.so.6] - #1 bash_tilde_expand() - [/usr/bin/bash] - #2 maybe_execute_file() - [/usr/bin/bash] - #3 main() - [/usr/bin/bash] - #4 __libc_start_main() - [libc.so.6] - #5 _start() - [/usr/bin/bash] - -If you have records of multiple crashes, you can use the :command:'-r' -option to specify the record more precisely, rather than going by -classification. You can also specify a classification of record and use the -:command:'-i' option to see the payload of each record with that -classification. - -Next steps -========== - -Adding telemetry to your applications: - -* :ref:`telemetry-api` diff --git a/source/clear-linux/reference/bundles/bundles.html.txt b/source/clear-linux/reference/bundles/bundles.html.txt index 9f44f5bd..297439d1 100644 --- a/source/clear-linux/reference/bundles/bundles.html.txt +++ b/source/clear-linux/reference/bundles/bundles.html.txt @@ -14,7 +14,7 @@ - Updated: 01/28/19 19:15 UTC + Updated: 02/20/19 17:05 UTC @@ -26,7 +26,6 @@ - @@ -112,13 +111,17 @@ - atom - The popular atom text editor
+ ark + No detailed summary available

Includes bundle(s):

    -
  • desktop-gnomelibs
    • +
  • kde-frameworks5
    • + +
  • qt-basic
    • + +
  • libstdcpp

@@ -128,9 +131,38 @@ - - audit - system audit tools + + atom + The popular atom text editor
+ +

Includes bundle(s): +

    + +
  • desktop-gnomelibs
    • + +
  • python3-basic
    • + +
+

+ + + + + + + + audit + system audit tools
+ +

Includes bundle(s): +

    + +
  • python3-basic
    • + +
+

+ + @@ -142,7 +174,7 @@

Includes bundle(s):

    -
  • python3-basic
    • +
  • python-extras
  • python2-basic
    • @@ -174,6 +206,27 @@
  • python3-basic
    • +
  • libstdcpp
    • + +
+

+ + + + + + + + baobab + No detailed summary available
+ +

Includes bundle(s): +

    + +
  • libX11client
    • + +
  • libglib
    • +

@@ -189,6 +242,29 @@ + + bcc + BPF Compiler Collection (BCC)
+ +

Includes bundle(s): +

    + +
  • BCC
    • + +
  • python3-basic
    • + +
  • libstdcpp
    • + +
  • llvm
    • + +
+

+ + + + + + big-data-basic Tools and frameworks for big data management
@@ -210,6 +286,22 @@ + + binutils + various development tools to manage binary executable code
+ + + + + + + + bison + bison development tool + + + + blender A 3D Modelling And Rendering Package
@@ -219,6 +311,8 @@
  • python3-basic
    • +
  • libstdcpp
    • +

    @@ -227,9 +321,21 @@ - - bmap-tools - Tools for creating the block map (bmap) for a file and copying files. Alternative to "dd" or "cp" + + bmap-tools + Tools for creating the block map (bmap) for a file and copying files. Alternative to "dd" or "cp"
    + +

    Includes bundle(s): +

      + +
    • bmap
      • + +
    • python3-basic
      • + +
    +

    + + @@ -269,9 +375,19 @@ - - borgbackup - Deduplicating archiver with compression and authenticated encryption. + + borgbackup + Deduplicating archiver with compression and authenticated encryption.
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + @@ -287,6 +403,12 @@
  • nasm
    • +
  • binutils
    • + +
  • bison
    • + +
  • flex
    • +
  • llvm
  • make
    • @@ -308,6 +430,12 @@

    Includes bundle(s):

      +
    • binutils
      • + +
    • bison
      • + +
    • flex
      • +
    • make
    • devpkg-zlib
      • @@ -341,9 +469,40 @@ - - clamav - Clam AntiVirus is an anti-virus toolkit for Unix + + cheese + Cheese webcam utilities
      + +

      Includes bundle(s): +

        + +
      • libX11client
        • + +
      • gnome-base-libs
        • + +
      • libglib
        • + +
      +

      + + + + + + + + clamav + Clam AntiVirus is an anti-virus toolkit for Unix
      + +

      Includes bundle(s): +

        + +
      • libstdcpp
        • + +
      +

      + + @@ -364,6 +523,8 @@
    • python3-basic
      • +
    • libstdcpp
      • +

    @@ -416,6 +577,8 @@
  • p11-kit
    • +
  • libstdcpp
    • +

    @@ -488,9 +651,21 @@ - - cockpit - Web based administration for GNU/Linux servers + + cockpit + Web based administration for GNU/Linux servers
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -512,6 +687,10 @@
  • python2-basic
    • +
  • openblas
    • + +
  • devpkg-opencv
    • +

    @@ -550,6 +729,8 @@
  • p11-kit
    • +
  • libstdcpp
    • +

    @@ -571,6 +752,12 @@
  • os-core-dev
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -579,10 +766,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • devpkg-gtk3
    • @@ -599,6 +790,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libsecret
  • devpkg-libxkbcommon
    • @@ -609,6 +802,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -619,12 +814,20 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • pygobject
    • +
  • devpkg-qtbase
    • +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -685,6 +888,12 @@
  • bc
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -693,10 +902,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • devpkg-gtk3
    • @@ -713,6 +926,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libpng
  • devpkg-libsecret
    • @@ -727,6 +942,8 @@
  • musl
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -739,12 +956,20 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • + +
  • pygobject
    • +
  • devpkg-qtbase
    • +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -825,6 +1050,12 @@
  • os-core-dev
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -833,10 +1064,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -847,6 +1082,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -855,6 +1092,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -865,8 +1104,14 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
    • +
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -900,16 +1145,40 @@ - - cve-bin - CVE scanning tool + + cve-bin-tool + CVE scanning tool
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + - - darktable - Photography workflow application + + darktable + Photography workflow application
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -923,6 +1192,8 @@
  • python3-basic
    • +
  • libstdcpp
    • +
  • postgresql
    • @@ -946,18 +1217,28 @@
  • os-core-dev
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -970,6 +1251,10 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • + +
  • devpkg-libjpeg-turbo
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -980,6 +1265,8 @@
  • nasm
    • +
  • devpkg-ncurses
    • +
  • devpkg-openmpi
  • openssl
    • @@ -992,13 +1279,19 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-qtbase
    • + +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • -
  • texinfo
    • +
  • devpkg-systemd
    • -
  • tzdata
    • +
  • texinfo
  • unzip
    • @@ -1053,12 +1346,18 @@
  • fuse
    • +
  • pygobject
    • +
  • tzdata
  • alsa-utils
  • fwupdate
    • +
  • gnome-characters
    • + +
  • thermal_daemon
    • +

    @@ -1082,10 +1381,54 @@
  • xemacs
    • +
  • baobab
    • + +
  • cheese
    • + +
  • eog
    • + +
  • evince
    • + +
  • evolution
    • + +
  • file-roller
    • +
  • firefox
    • +
  • gedit
    • + +
  • gnome-calculator
    • + +
  • gnome-characters
    • + +
  • gnome-color-manager
    • + +
  • gnome-disk-utility
    • + +
  • gnome-font-viewer
    • + +
  • gnome-logs
    • + +
  • gnome-music
    • + +
  • gnome-photos
    • + +
  • gnome-screenshot
    • + +
  • gnome-system-monitor
    • + +
  • gnome-todo
    • + +
  • gnome-weather
    • + +
  • gvim
    • +
  • libva-utils
    • +
  • seahorse
    • + +
  • totem
    • +
  • xemacs
    • @@ -1115,12 +1458,12 @@
  • darktable
    • +
  • gimp
    • +
  • thunderbird
  • vlc
    • -
  • gimp
    • -

    @@ -1138,6 +1481,8 @@
  • desktop-gnomelibs
    • +
  • fonts-basic
    • +
  • intended for all supported styles
  • current default
    • @@ -1201,12 +1546,28 @@
  • sysadmin-basic-dev
    • +
  • devpkg-R
    • + +
  • devpkg-SDL
    • + +
  • devpkg-SDL2
    • + +
  • devpkg-SDL2_image
    • + +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
  • devpkg-atk
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-bluez
    • +
  • devpkg-cairo
  • devpkg-cheese
    • @@ -1217,6 +1578,8 @@
  • devpkg-cogl
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -1227,6 +1590,8 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-evolution-data-server
    • @@ -1235,6 +1600,8 @@
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
  • fonttools
    • @@ -1243,6 +1610,8 @@
  • devpkg-fribidi
    • +
  • devpkg-fuse
    • +
  • gdb
  • devpkg-gdk-pixbuf
    • @@ -1269,24 +1638,40 @@
  • devpkg-icu4c
    • +
  • inkscape
    • +
  • intltool
  • iproute2
  • jq
    • +
  • devpkg-json-glib
    • +
  • kbd
  • less
  • devpkg-libX11
    • +
  • devpkg-libXv
    • + +
  • devpkg-libarchive
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • +
  • devpkg-libical
    • +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -1297,6 +1682,8 @@
  • devpkg-libsoup
    • +
  • devpkg-libva
    • +
  • devpkg-libxkbcommon
  • libxml2
    • @@ -1305,18 +1692,32 @@
  • libxslt
    • +
  • devpkg-libxslt
    • +
  • llvm
    • +
  • devpkg-llvm
    • + +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
    • +
  • devpkg-mozjs52
    • + +
  • devpkg-mozjs60
    • +
  • nasm
    • +
  • devpkg-ncurses
    • +
  • devpkg-nspr
  • devpkg-nss
    • +
  • devpkg-opencv
    • +
  • openssl
  • devpkg-openssl
    • @@ -1335,18 +1736,30 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • + +
  • pygobject
    • +
  • devpkg-qtbase
  • devpkg-qtdeclarative
    • +
  • devpkg-qtsvg
    • + +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -1365,6 +1778,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorg-server
  • devpkg-xorgproto
    • @@ -1394,6 +1809,8 @@
  • libX11client
    • +
  • perl-basic
    • +

    @@ -1428,8 +1845,6 @@
  • desktop-kde-libs
    • -
  • desktop-kde-apps
    • -
  • qt-basic
  • x11-tools
    • @@ -1438,10 +1853,16 @@
  • fuse
    • +
  • dolphin
    • +
  • gpgme
  • kdiff3
    • +
  • konsole
    • + +
  • ksysguard
    • +
  • libxslt
    • @@ -1456,6 +1877,43 @@ desktop-kde-apps Applications for the desktop from the KDE ecosystem
    +

    Includes bundle(s): +

      + +
    • desktop-kde-libs
      • + +
    • perl-basic
      • + +
    • qt-basic
      • + +
    • desktop-kde-games
      • + +
    • ark
      • + +
    • digikam
      • + +
    • dolphin
      • + +
    • gwenview
      • + +
    • kate
      • + +
    • konsole
      • + +
    • krita
      • + +
    +

    + + + + + + + + desktop-kde-games + Games for the desktop from the KDE ecosystem
    +

    Includes bundle(s):

      @@ -1482,6 +1940,8 @@
    • qt-basic
      • +
    • kde-frameworks5
      • +

    @@ -1499,6 +1959,27 @@
  • perl-basic
    • +
  • libstdcpp
    • + + +

    + + + + + + + + desktop-lxqt + Run the LXQt desktop environment
    + +

    Includes bundle(s): +

      + +
    • x11-server
      • + +
    • fonts-basic
      • +

    @@ -1542,6 +2023,8 @@
  • llvm
    • +
  • devpkg-llvm
    • +
  • patch
  • strace
    • @@ -1569,24 +2052,42 @@
  • perl-basic-dev
    • +
  • devpkg-SDL
    • + +
  • devpkg-alsa-lib
    • +
  • bc
    • +
  • binutils
    • + +
  • bison
    • +
  • devpkg-cairo
  • cpio
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • + +
  • devpkg-freetype
    • +
  • devpkg-glib
  • gzip
    • @@ -1601,6 +2102,14 @@
  • devpkg-libX11
    • +
  • devpkg-libXv
    • + +
  • devpkg-libarchive
    • + +
  • devpkg-libffi
    • + +
  • devpkg-libjpeg-turbo
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -1613,12 +2122,20 @@
  • libxslt
    • +
  • devpkg-libxslt
    • +
  • llvm
    • +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
    • +
  • devpkg-ncurses
    • + +
  • openblas
    • +
  • devpkg-openmpi
  • openssl
    • @@ -1631,16 +2148,26 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • +
  • devpkg-qtbase
    • +
  • devpkg-qttools
    • + +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
    • +
  • texlive
    • +
  • tzdata
  • unzip
    • @@ -1690,6 +2217,23 @@ + + devpkg-alsa-lib + All content for pkgconfig file alsa-lib.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + devpkg-at-spi2-atk All content for pkgconfig file at-spi2-atk.pc
    @@ -1768,6 +2312,23 @@ + + devpkg-bluez + All content for pkgconfig file bluez.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + devpkg-cairo All content for pkgconfig file cairo.pc
    @@ -1833,6 +2394,32 @@
  • devpkg-base
    • +
  • devpkg-atk
    • + +
  • devpkg-cairo
    • + +
  • devpkg-cogl
    • + +
  • devpkg-gdk-pixbuf
    • + +
  • devpkg-glib
    • + +
  • devpkg-gtk3
    • + +
  • devpkg-json-glib
    • + +
  • devpkg-libX11
    • + +
  • devpkg-libinput
    • + +
  • devpkg-libxkbcommon
    • + +
  • devpkg-pango
    • + +
  • devpkg-systemd
    • + +
  • devpkg-wayland
    • +

    @@ -1872,6 +2459,35 @@ devpkg-cogl All content for pkgconfig file cogl.pc
    +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-cairo
      • + +
    • devpkg-gdk-pixbuf
      • + +
    • devpkg-glib
      • + +
    • devpkg-libX11
      • + +
    • devpkg-mesa
      • + +
    • devpkg-wayland
      • + +
    +

    + + + + + + + + devpkg-curl + All content for pkgconfig file curl.pc
    +

    Includes bundle(s):

      @@ -1902,6 +2518,23 @@ + + devpkg-e2fsprogs + All content for pkgconfig file e2fsprogs.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-elfutils All content for pkgconfig file elfutils.pc
      @@ -2122,6 +2755,23 @@ + + devpkg-glm + All content for pkgconfig file glm.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-gnome-desktop All content for pkgconfig file gnome-desktop.pc
      @@ -2219,6 +2869,33 @@ + + devpkg-gtk + All content for pkgconfig file gtk+.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      • devpkg-atk
        • + +
      • devpkg-cairo
        • + +
      • devpkg-gdk-pixbuf
        • + +
      • devpkg-glib
        • + +
      • devpkg-pango
        • + +
      +

      + + + + + + devpkg-gtk3 All content for pkgconfig file gtk3.pc
      @@ -2298,6 +2975,42 @@ + + devpkg-json-glib + All content for pkgconfig file json-glib.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      • devpkg-glib
        • + +
      +

      + + + + + + + + devpkg-libarchive + All content for pkgconfig file libarchive.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-libdrm All content for pkgconfig file libdrm.pc
      @@ -2334,6 +3047,23 @@ + + devpkg-libffi + All content for pkgconfig file libffi.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-libical All content for pkgconfig file libical.pc
      @@ -2353,6 +3083,57 @@ + + devpkg-libinput + All content for pkgconfig file libinput.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + + + devpkg-libjpeg-turbo + All content for pkgconfig file libjpeg-turbo.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + + + devpkg-libogg + All content for pkgconfig file libogg.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-libpciaccess All content for pkgconfig file libpciaccess.pc
      @@ -2446,6 +3227,23 @@ + + devpkg-libva + All content for pkgconfig file libva.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-libX11 All content for pkgconfig file libX11.pc
      @@ -2744,6 +3542,46 @@ + + devpkg-libxslt + All content for pkgconfig file libxslt.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      • devpkg-libxml2
        • + +
      +

      + + + + + + + + devpkg-libXv + All content for pkgconfig file libXv.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      • devpkg-libX11
        • + +
      • devpkg-xorgproto
        • + +
      +

      + + + + + + devpkg-libXxf86vm All content for pkgconfig file libXxf86vm.pc
      @@ -2767,6 +3605,40 @@ + + devpkg-llvm + All content for pkgconfig file llvm.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + + + devpkg-lua + All content for pkgconfig file lua.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-mesa All content for pkgconfig file mesa.pc
      @@ -2788,6 +3660,57 @@ + + devpkg-mozjs52 + All content for pkgconfig file mozjs52.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + + + devpkg-mozjs60 + All content for pkgconfig file mozjs60.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + + + devpkg-ncurses + All content for pkgconfig file ncurses.pc
      + +

      Includes bundle(s): +

        + +
      • devpkg-base
        • + +
      +

      + + + + + + devpkg-nspr All content for pkgconfig file nspr.pc
      @@ -2833,6 +3756,8 @@
    • devpkg-base
      • +
    • openblas
      • +

    @@ -2972,6 +3897,23 @@ + + devpkg-popt + All content for pkgconfig file popt.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + devpkg-qt3d All content for pkgconfig file qt3d.pc
    @@ -3029,6 +3971,384 @@ + + devpkg-qtlocation + All content for pkgconfig file qtlocation.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-qtbase
      • + +
    +

    + + + + + + + + devpkg-qtsvg + All content for pkgconfig file qtsvg.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-qtbase
      • + +
    +

    + + + + + + + + devpkg-qttools + All content for pkgconfig file qttools.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-qtbase
      • + +
    +

    + + + + + + + + devpkg-qtwebchannel + All content for pkgconfig file qtwebchannel.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-qtbase
      • + +
    • devpkg-qtdeclarative
      • + +
    +

    + + + + + + + + devpkg-qtwebengine + All content for pkgconfig file qtwebengine.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-qtbase
      • + +
    • devpkg-qtdeclarative
      • + +
    • devpkg-qtlocation
      • + +
    • devpkg-qtwebchannel
      • + +
    +

    + + + + + + + + devpkg-R + All content for pkgconfig file R.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + + + devpkg-readline + All content for pkgconfig file readline.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + + + devpkg-SDL + All content for pkgconfig file SDL.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + + + devpkg-SDL2 + All content for pkgconfig file SDL2.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + + + devpkg-SDL2_gfx + All content for pkgconfig file SDL2_gfx.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL2
      • + +
    +

    + + + + + + + + devpkg-SDL2_image + All content for pkgconfig file SDL2_image.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL2
      • + +
    +

    + + + + + + + + devpkg-SDL2_mixer + All content for pkgconfig file SDL2_mixer.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL2
      • + +
    +

    + + + + + + + + devpkg-SDL2_net + All content for pkgconfig file SDL2_net.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL2
      • + +
    +

    + + + + + + + + devpkg-SDL2_ttf + All content for pkgconfig file SDL2_ttf.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL2
      • + +
    +

    + + + + + + + + devpkg-SDL_gfx + All content for pkgconfig file SDL_gfx.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL
      • + +
    +

    + + + + + + + + devpkg-SDL_image + All content for pkgconfig file SDL_image.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL
      • + +
    +

    + + + + + + + + devpkg-SDL_mixer + All content for pkgconfig file SDL_mixer.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL
      • + +
    +

    + + + + + + + + devpkg-SDL_net + All content for pkgconfig file SDL_net.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL
      • + +
    +

    + + + + + + + + devpkg-SDL_ttf + All content for pkgconfig file SDL_ttf.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    • devpkg-SDL
      • + +
    +

    + + + + + + + + devpkg-shared-mime-info + All content for pkgconfig file shared-mime-info.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + devpkg-sqlite-autoconf All content for pkgconfig file sqlite-autoconf.pc
    @@ -3046,6 +4366,23 @@ + + devpkg-systemd + All content for pkgconfig file systemd.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + devpkg-util-linux All content for pkgconfig file util-linux.pc
    @@ -3120,6 +4457,23 @@ + + devpkg-xcb-proto + All content for pkgconfig file xcb-proto.pc
    + +

    Includes bundle(s): +

      + +
    • devpkg-base
      • + +
    +

    + + + + + + devpkg-xorg-server All content for pkgconfig file xorg-server.pc
    @@ -3212,16 +4566,51 @@ - - diffoscope - In-depth comparison of files, archives, and directories + + diffoscope + In-depth comparison of files, archives, and directories
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + - - diffutils - Tools for file comparisons and differences + + diffutils + Tools for file comparisons and differences
    + + + + + + + + digikam + An advanced digital photo management application
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -3243,16 +4632,97 @@ - - docbook-utils - docbook tools + + dnf + The dnf tool for mixer use
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • pygobject
      • + +
    +

    + + - - docutils - Tools for converting text to other formats (e.g. HTML, XML, LaTeX) + + docbook-utils + docbook tools
    + +

    Includes bundle(s): +

      + +
    • perl-basic
      • + +
    +

    + + + + + + + + docker-compose + Compose is a tool for defining and running multi-container Docker applications.
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • containers-basic
      • + +
    +

    + + + + + + + + docutils + Tools for converting text to other formats (e.g. HTML, XML, LaTeX)
    + +

    Includes bundle(s): +

      + +
    • e.g. HTML, XML, LaTeX
      • + +
    • python3-basic
      • + +
    +

    + + + + + + + + dolphin + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -3268,6 +4738,8 @@
  • python3-basic
    • +
  • libglib
    • +

    @@ -3278,7 +4750,7 @@ dosfstools - + (V)FAT manipulation tools @@ -3290,9 +4762,19 @@ - - doxygen - documentation tooling + + doxygen + documentation tooling
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -3304,9 +4786,19 @@ - - duperemove - Tool for finding duplicated extents and submitting them for deduplication + + duperemove + Tool for finding duplicated extents and submitting them for deduplication
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    +

    + + @@ -3345,18 +4837,28 @@
  • os-core-dev
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -3369,14 +4871,20 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
  • libxslt
    • +
  • devpkg-lua
    • +
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -3385,15 +4893,19 @@
  • patch
    • -
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-qtbase
    • + +
  • devpkg-readline
  • devpkg-sqlite-autoconf
  • strace
    • -
  • texinfo
    • +
  • devpkg-systemd
    • -
  • tzdata
    • +
  • texinfo
  • unzip
    • @@ -3555,6 +5067,35 @@ enterprise-login Enable enterprise login systems like Active Directory
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + + + + + + + eog + The GNOME Image Viewer
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • gnome-base-libs
      • + +
    +

    + @@ -3567,6 +5108,50 @@ + + evince + GNOME document viewer view library
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • gnome-base-libs
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + evolution + libraries needed for Evolution shell components
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • gnome-base-libs
      • + +
    • libglib
      • + +
    +

    + + + + + + file program for recognizing the type of data contained in a computer file @@ -3574,6 +5159,25 @@ + + file-roller + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + findutils basic directory searching utilities e.g. find, xargs, and locate @@ -3588,9 +5192,21 @@ - - firefox - Web Browser + + firefox + The firefox web browser
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • desktop-gnomelibs
      • + +
    +

    + + @@ -3616,9 +5232,19 @@ - - flac - FLAC audio format tools + + flac + FLAC audio format tools
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -3632,6 +5258,10 @@
  • git
    • +
  • libstdcpp
    • + +
  • libglib
    • +

    @@ -3641,8 +5271,48 @@ - fonttools - tools to manipulate system and user fonts + flex + flex development tool + + + + + + fonts-basic + fonts and fonts tooling
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • libglib
      • + +
    • python3-basic
      • + +
    +

    + + + + + + + + fonttools + tools to manipulate system and user fonts
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -3716,16 +5386,46 @@
  • perl-basic-dev
    • +
  • devpkg-SDL
    • + +
  • devpkg-SDL2
    • + +
  • devpkg-SDL2_gfx
    • + +
  • devpkg-SDL2_image
    • + +
  • devpkg-SDL2_net
    • + +
  • devpkg-SDL2_ttf
    • + +
  • devpkg-SDL_gfx
    • + +
  • devpkg-SDL_image
    • + +
  • devpkg-SDL_net
    • + +
  • devpkg-SDL_ttf
    • + +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
  • devpkg-atk
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-bluez
    • +
  • devpkg-cairo
  • devpkg-clutter
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -3734,10 +5434,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
  • devpkg-freetype
    • @@ -3766,16 +5470,28 @@
  • intltool
    • +
  • devpkg-json-glib
    • +
  • kbd
  • devpkg-libX11
    • +
  • devpkg-libXv
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • +
  • devpkg-libical
    • +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -3792,14 +5508,22 @@
  • libxslt
    • +
  • devpkg-libxslt
    • +
  • llvm
    • +
  • devpkg-llvm
    • + +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
  • nasm
    • +
  • devpkg-ncurses
    • +
  • devpkg-nspr
  • devpkg-nss
    • @@ -3820,16 +5544,26 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • + +
  • pygobject
    • +
  • devpkg-qtbase
    • +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -3848,6 +5582,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -3873,16 +5609,63 @@ - - gftp - Text mode and GUI FTP Client + + gedit + gedit
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + - - gimp - GNU Image Manipulation Program + + gftp + Text mode and GUI FTP Client
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + gimp + GNU Image Manipulation Program
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + @@ -3908,9 +5691,21 @@ - - glmark2 - OpenGL 2.0 and ES 2.0 benchmark + + glmark2 + OpenGL 2.0 and ES 2.0 benchmark
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -3922,6 +5717,267 @@ + + gnome-base-libs + Common libraries used by gnome and related apps
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libX11client
      • + +
    • desktop-gnomelibs
      • + +
    +

    + + + + + + + + gnome-calculator + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libX11client
      • + +
    +

    + + + + + + + + gnome-characters + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libX11client
      • + +
    +

    + + + + + + + + gnome-color-manager + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • gnome-base-libs
      • + +
    • libglib
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + gnome-disk-utility + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + gnome-font-viewer + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • gnome-base-libs
      • + +
    +

    + + + + + + + + gnome-logs + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libX11client
      • + +
    +

    + + + + + + + + gnome-music + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + gnome-photos + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + gnome-screenshot + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • gnome-base-libs
      • + +
    • libglib
      • + +
    +

    + + + + + + + + gnome-system-monitor + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + gnome-todo + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + gnome-weather + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • libX11client
      • + +
    +

    + + + + + + go-basic Build and run go language programs
    @@ -3954,6 +6010,12 @@
  • os-core-dev
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -3962,10 +6024,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -3976,6 +6042,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -3984,6 +6052,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -3994,10 +6064,16 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • unzip
    • @@ -4029,9 +6105,67 @@ - - gpgme - GPG encryption tools + + gpgme + GPG encryption tools
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + gvim + Vi Improved, a highly configurable, improved version of the vi text editor (with advanced features, such as a GUI)
    + +

    Includes bundle(s): +

      + +
    • with advanced features, such as a GUI
      • + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + gwenview + A fast and easy to use image viewer
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -4047,6 +6181,35 @@ hardware-bluetooth Software to enable use of bluetooth hardware
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • libglib
      • + +
    +

    + + + + + + + + hardware-printing + Firmware and tools for printer support
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • python3-basic
      • + +
    +

    + @@ -4059,6 +6222,10 @@

    Includes bundle(s):

      +
    • libstdcpp
      • + +
    • libglib
      • +
    • wpa_supplicant
    @@ -4105,6 +6272,8 @@

    Includes bundle(s):

      +
    • libstdcpp
      • +
    • perl-basic
    @@ -4129,9 +6298,19 @@ - - hugo - Static Web site generator + + hugo + Static Web site generator
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + @@ -4150,16 +6329,36 @@ - - icdiff - Improved Color Differences between similar files + + icdiff + Improved Color Differences between similar files
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + - - icecream - Distributed network compilation + + icecream + Distributed network compilation
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -4171,6 +6370,25 @@ + + igt-gpu-tools + Intel graphics performance diagnostics
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + ImageMagick ImageMagick
    @@ -4188,9 +6406,23 @@ - - inkscape - Inkscape SVG drawing program + + inkscape + Inkscape SVG drawing program
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -4230,9 +6462,23 @@ - - irssi - Internet Relay Chat (IRC) client program + + irssi + Internet Relay Chat (IRC) client program
    + +

    Includes bundle(s): +

      + +
    • IRC
      • + +
    • perl-basic
      • + +
    • libglib
      • + +
    +

    + + @@ -4279,6 +6525,29 @@ + + java11-basic + Build and run java11 language programs
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • p11-kit
      • + +
    • python3-basic
      • + +
    • which
      • + +
    +

    + + + + + + java9-basic Build and run java9 language programs
    @@ -4316,6 +6585,44 @@ + + jupyter + Jupyter metapackage. Install all the Jupyter components in one go.
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + + + + + + + kate + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + kbd library to manage the Linux keymaps @@ -4323,9 +6630,57 @@ - - kdiff3 - Feature-Rich Code Comparison Utility + + kde-frameworks5 + Common KDE Frameworks 5 libraries
    + +

    Includes bundle(s): +

      + +
    • qt-basic
      • + +
    +

    + + + + + + + + kdenlive + A non-linear video editor for Linux using the MLT video framework
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + kdiff3 + Feature-Rich Code Comparison Utility
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    +

    + + @@ -4337,9 +6692,23 @@ - - keepassxc - KeePassXC is a password manager + + keepassxc + KeePassXC is a password manager
    + +

    Includes bundle(s): +

      + +
    • qt-basic
      • + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -4513,6 +6882,25 @@ + + kernel-lts-dkms + Support module building/loading via DKMS in LTS kernel
    + +

    Includes bundle(s): +

      + +
    • c-basic
      • + +
    • linux-lts-dev
      • + +
    +

    + + + + + + kernel-native Run the native kernel
    @@ -4643,6 +7031,71 @@ + + konsole + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + krita + Edit and paint images
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + ksysguard + Track and control the processes running in your system
    + +

    Includes bundle(s): +

      + +
    • kde-frameworks5
      • + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + kvm-host Run virtual machines
    @@ -4681,8 +7134,14 @@
  • devpkg-atk
    • +
  • binutils
    • + +
  • bison
    • +
  • devpkg-cairo
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -4693,16 +7152,22 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
  • devpkg-freetype
  • devpkg-fribidi
    • +
  • devpkg-fuse
    • +
  • gdb
  • devpkg-gdk-pixbuf
    • @@ -4721,14 +7186,24 @@
  • intltool
    • +
  • devpkg-json-glib
    • +
  • kbd
  • devpkg-libX11
    • +
  • devpkg-libXv
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • + +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -4745,12 +7220,20 @@
  • llvm
    • +
  • devpkg-llvm
    • + +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
    • +
  • devpkg-mozjs52
    • +
  • nasm
    • +
  • devpkg-ncurses
    • +
  • nfs-utils
  • devpkg-nspr
    • @@ -4773,14 +7256,22 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -4797,6 +7288,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -4860,9 +7353,35 @@ + + lftp + File transfer client for http, ftp, sftp and more
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + + + + + - lftp - File transfer client for http, ftp, sftp and more + libarchive + various archiving tools + + + + + + libglib + Common C routines used by Gtk+ and other libs
    + + @@ -4874,9 +7393,32 @@ - - libva-utils - collection of utilities and examples to exercise VA-API (Video Acceleration API) + + libstdcpp + C++ runtime suppot
    + + + + + + + + libva-utils + collection of utilities and examples to exercise VA-API (Video Acceleration API)
    + +

    Includes bundle(s): +

      + +
    • Video Acceleration API
      • + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -4885,14 +7427,34 @@ libX11client Grouping only bundle for use in X using bundles
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • libglib
      • + +
    +

    + - - libxml2 - XML processing tools + + libxml2 + XML processing tools
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + @@ -4904,9 +7466,19 @@ - - libzmq - ZMQ tools and libs + + libzmq + ZMQ tools and libs
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -4915,6 +7487,16 @@ lightdm LightDM login manager with GTK greeter
    +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + @@ -4928,15 +7510,44 @@ - linux-tools - Tools for use in Linux + linux-lts + programs for developing Linux LTS version - - llvm - LLVM/Clang compiler toolchain + + linux-tools + Tools for use in Linux
    + +

    Includes bundle(s): +

      + +
    • perl-basic
      • + +
    • binutils
      • + +
    +

    + + + + + + + + llvm + LLVM/Clang compiler toolchain
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -4964,6 +7575,13 @@ + + ltp + Linux Test Project + + + + lua-basic Lua programming language
    @@ -4999,8 +7617,16 @@
  • machine-learning-mycroft
    • +
  • machine-learning-pytorch
    • +
  • machine-learning-tensorflow
    • +
  • devpkg-libjpeg-turbo
    • + +
  • openblas
    • + +
  • devpkg-opencv
    • +

    @@ -5026,6 +7652,25 @@ + + machine-learning-pytorch + Build machine learning applications with mycroft
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + machine-learning-tensorflow Build machine learning applications using Tensorflow
    @@ -5060,6 +7705,8 @@
  • python-data-science
    • +
  • jupyter
    • +

    @@ -5075,10 +7722,10 @@

    Includes bundle(s):

      -
    • python2-basic
      • -
    • mutt
      • +
    • offlineimap
      • +

    @@ -5100,8 +7747,14 @@
  • os-core-dev
    • +
  • binutils
    • + +
  • bison
    • +
  • devpkg-cairo
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -5110,10 +7763,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
  • devpkg-freetype
    • @@ -5138,6 +7795,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libpng
  • devpkg-libpthread-stubs
    • @@ -5148,8 +7807,14 @@
  • libxslt
    • +
  • devpkg-lua
    • +
  • make
    • +
  • devpkg-ncurses
    • + +
  • openblas
    • +
  • openssl
  • devpkg-openssl
    • @@ -5160,12 +7825,18 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • +
  • devpkg-qtbase
    • +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -5178,6 +7849,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -5203,6 +7876,25 @@ + + maker-3dprinting + 3D printing tools for Makers
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • qt-basic
      • + +
    +

    + + + + + + maker-cnc CNC carving tools for Makers
    @@ -5216,6 +7908,8 @@
  • python2-basic
    • +
  • pygobject
    • +

    @@ -5241,9 +7935,19 @@ - - meld - visual file difference and merging tool, targeted at developers + + meld + visual file difference and merging tool, targeted at developers
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + @@ -5288,6 +7992,8 @@
  • zstd
    • +
  • dnf
    • +

    @@ -5296,16 +8002,38 @@ - - mkosi - generate disk images + + mkosi + generate disk images
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + - - mpg123 - play and decode mp3 files + + mpg123 + play and decode mp3 files
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • libglib
      • + +
    +

    + + @@ -5402,20 +8130,32 @@
  • perl-basic-dev
    • +
  • binutils
    • + +
  • bison
    • +
  • devpkg-cairo
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -5430,6 +8170,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libpng
  • devpkg-libxkbcommon
    • @@ -5438,8 +8180,12 @@
  • libxslt
    • +
  • devpkg-lua
    • +
  • make
    • +
  • devpkg-ncurses
    • +
  • devpkg-nss
  • openssl
    • @@ -5452,10 +8198,18 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-qtbase
    • + +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -5482,6 +8236,23 @@ + + network-monitor-node + Run network security monitoring node
    + +

    Includes bundle(s): +

      + +
    • suricata
      • + +
    +

    + + + + + + network-security-monitoring Various tools to test and improve network security
    @@ -5495,9 +8266,11 @@
  • qt-basic
    • +
  • network-monitor-node
    • +
  • nmap
    • -
  • suricata
    • +
  • wireshark

    • @@ -5540,16 +8313,36 @@ - - nload - network load monitor + + nload + network load monitor
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + - - nmap - security scanner for network device discovery + + nmap + security scanner for network device discovery
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -5558,14 +8351,85 @@ nodejs-basic Run javascript server side
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + - - octave - high-level programming language, primarily intended for numerical computations + + ocaml-basic + Build and run ocaml language programs
    + + + + + + + + octave + high-level programming language, primarily intended for numerical computations
    + +

    Includes bundle(s): +

      + +
    • qt-basic
      • + +
    • libX11client
      • + +
    • openblas
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + offlineimap + locally copy imap email
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    • libglib
      • + +
    • python2-basic
      • + +
    +

    + + + + + + + + openblas + The OpenBLAS linear algebra package
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -5577,9 +8441,19 @@ - - openmpi - HPC tools for running on jobs on clusters + + openmpi + HPC tools for running on jobs on clusters
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -5634,7 +8508,13 @@
  • perl-basic
    • -
  • python3-basic
    • +
  • python-extras
    • + +
  • openblas
    • + +
  • libstdcpp
    • + +
  • libX11client
  • keepalived
    • @@ -5760,6 +8640,14 @@ os-cloudguest-vmware Run any initialization processes required for VMWare VMs
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + @@ -5796,6 +8684,8 @@
  • network-basic-dev
    • +
  • nodejs-basic
    • +
  • openssh-server
  • os-core
    • @@ -5824,6 +8714,8 @@
  • diffoscope
    • +
  • dnf
    • +
  • devpkg-fribidi
  • devpkg-gdk-pixbuf
    • @@ -5832,8 +8724,16 @@
  • devpkg-gtk3
    • +
  • devpkg-gtk
    • + +
  • libarchive
    • + +
  • devpkg-libarchive
    • +
  • devpkg-libdrm
    • +
  • devpkg-libva
    • +
  • devpkg-libxcb
  • devpkg-libXdamage
    • @@ -5848,6 +8748,8 @@
  • scons
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
    • @@ -5889,8 +8791,14 @@
  • sysadmin-basic-dev
    • +
  • devpkg-R
    • + +
  • devpkg-SDL
    • +
  • acpica-unix2
    • +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
    • @@ -5899,6 +8807,12 @@
  • bc
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-bluez
    • +
  • devpkg-cairo
  • ccache
    • @@ -5909,6 +8823,8 @@
  • curl
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -5921,18 +8837,26 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
  • file
    • +
  • flex
    • +
  • devpkg-fontconfig
    • +
  • fonttools
    • +
  • devpkg-freetype
  • devpkg-fribidi
    • +
  • devpkg-fuse
    • +
  • gdb
  • devpkg-gdk-pixbuf
    • @@ -5959,18 +8883,32 @@
  • iproute2
    • +
  • devpkg-json-glib
    • +
  • kbd
  • less
  • devpkg-libX11
    • +
  • devpkg-libXv
    • + +
  • devpkg-libarchive
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • +
  • devpkg-libical
    • +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -5989,20 +8927,32 @@
  • libxslt
    • +
  • devpkg-libxslt
    • +
  • llvm
    • +
  • devpkg-llvm
    • + +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
    • +
  • devpkg-mozjs52
    • +
  • nasm
    • +
  • devpkg-ncurses
    • +
  • nfs-utils
  • devpkg-nspr
  • devpkg-nss
    • +
  • openblas
    • +
  • devpkg-openmpi
  • openssl
    • @@ -6023,18 +8973,32 @@
  • plzip
    • +
  • devpkg-popt
    • + +
  • pygobject
    • +
  • devpkg-qtbase
  • devpkg-qtdeclarative
    • +
  • devpkg-qtsvg
    • + +
  • devpkg-qttools
    • + +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • texlive
    • @@ -6053,6 +9017,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -6091,6 +9057,12 @@
  • dev-utils
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -6099,10 +9071,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • gdb
  • devpkg-glib
    • @@ -6115,6 +9091,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -6123,6 +9101,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -6131,8 +9111,14 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
    • +
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • unzip
    • @@ -6189,6 +9175,12 @@
  • os-core-update
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -6199,10 +9191,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • gdb
  • devpkg-glib
    • @@ -6217,6 +9213,10 @@
  • devpkg-libX11
    • +
  • devpkg-libarchive
    • + +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -6225,6 +9225,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -6235,8 +9237,14 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
    • +
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -6272,6 +9280,8 @@
  • os-core
    • +
  • libglib
    • +

    @@ -6328,12 +9338,12 @@
  • make
    • -
  • openssh-server
    • -
  • perl-basic
  • sudo
    • +
  • binutils
    • +

    @@ -6446,28 +9456,74 @@
  • devpkg-qtbase
    • +
  • devpkg-SDL
    • + +
  • devpkg-SDL2_gfx
    • + +
  • devpkg-SDL2_image
    • + +
  • devpkg-SDL2_mixer
    • + +
  • devpkg-SDL2_net
    • + +
  • devpkg-SDL2_ttf
    • + +
  • devpkg-SDL_gfx
    • + +
  • devpkg-SDL_image
    • + +
  • devpkg-SDL_mixer
    • + +
  • devpkg-SDL_net
    • + +
  • devpkg-SDL_ttf
    • +
  • bc
    • +
  • bison
    • +
  • devpkg-cairo
    • +
  • devpkg-curl
    • + +
  • devpkg-curl
    • +
  • fio
    • +
  • flex
    • +
  • gimp
    • +
  • devpkg-gtk
    • + +
  • devpkg-libXv
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpng
    • +
  • devpkg-lua
    • +
  • openmpi
  • devpkg-openmpi
  • nasm
    • +
  • devpkg-ncurses
    • + +
  • openblas
    • +
  • devpkg-openssl
  • patch
  • devpkg-pcre
    • +
  • devpkg-popt
    • +
  • redis-native
  • scons
    • @@ -6521,24 +9577,70 @@
  • yasm
    • +
  • devpkg-SDL
    • + +
  • devpkg-SDL2_gfx
    • + +
  • devpkg-SDL2_image
    • + +
  • devpkg-SDL2_mixer
    • + +
  • devpkg-SDL2_net
    • + +
  • devpkg-SDL2_ttf
    • + +
  • devpkg-SDL_gfx
    • + +
  • devpkg-SDL_image
    • + +
  • devpkg-SDL_mixer
    • + +
  • devpkg-SDL_net
    • + +
  • devpkg-SDL_ttf
    • +
  • bc
    • +
  • bison
    • +
  • devpkg-cairo
    • +
  • devpkg-curl
    • + +
  • devpkg-curl
    • +
  • fio
    • +
  • flex
    • +
  • gimp
    • +
  • devpkg-gtk
    • + +
  • devpkg-libXv
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpng
    • +
  • devpkg-lua
    • +
  • nasm
    • +
  • devpkg-ncurses
    • + +
  • openblas
    • +
  • devpkg-openssl
  • patch
  • devpkg-pcre
    • +
  • devpkg-popt
    • +
  • scons
  • devpkg-sqlite-autoconf
    • @@ -6588,22 +9690,40 @@
  • bc
    • +
  • bison
    • + +
  • devpkg-curl
    • +
  • fio
    • +
  • flex
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpng
    • +
  • devpkg-lua
    • +
  • openmpi
  • devpkg-openmpi
  • nasm
    • +
  • devpkg-ncurses
    • + +
  • openblas
    • +
  • devpkg-openssl
  • patch
  • devpkg-pcre
    • +
  • devpkg-popt
    • +
  • redis-native
  • scons
    • @@ -6633,6 +9753,8 @@
  • cryptography
    • +
  • fonts-basic
    • +
  • openssl
  • perl-basic
    • @@ -6655,8 +9777,12 @@
  • darktable
    • +
  • eog
    • +
  • firefox
    • +
  • gvim
    • +
  • thunderbird
  • xemacs
    • @@ -6684,6 +9810,12 @@
  • perl-basic-dev
    • +
  • devpkg-R
    • + +
  • devpkg-SDL
    • + +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
    • @@ -6692,10 +9824,20 @@
  • bc
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-bluez
    • +
  • devpkg-cairo
  • devpkg-clutter
    • +
  • devpkg-cogl
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -6706,16 +9848,26 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
    • +
  • devpkg-evolution-data-server
    • +
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
    • +
  • fonttools
    • +
  • devpkg-freetype
  • devpkg-fribidi
    • +
  • devpkg-fuse
    • +
  • gdb
  • devpkg-gdk-pixbuf
    • @@ -6742,18 +9894,36 @@
  • devpkg-icu4c
    • +
  • inkscape
    • +
  • intltool
  • jq
    • +
  • devpkg-json-glib
    • +
  • kbd
  • devpkg-libX11
    • +
  • devpkg-libXv
    • + +
  • devpkg-libarchive
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • + +
  • devpkg-libical
    • + +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -6764,6 +9934,8 @@
  • devpkg-libsoup
    • +
  • devpkg-libva
    • +
  • devpkg-libxkbcommon
  • libxml2
    • @@ -6772,16 +9944,34 @@
  • libxslt
    • +
  • devpkg-libxslt
    • + +
  • lightdm
    • +
  • llvm
    • +
  • devpkg-llvm
    • + +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
    • +
  • devpkg-mozjs52
    • + +
  • devpkg-mozjs60
    • +
  • nasm
    • +
  • devpkg-ncurses
    • +
  • nfs-utils
    • +
  • devpkg-nspr
    • + +
  • devpkg-nss
    • +
  • openssl
  • devpkg-openssl
    • @@ -6800,16 +9990,26 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • + +
  • pygobject
    • +
  • devpkg-qtbase
    • +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -6824,8 +10024,12 @@
  • devpkg-wayland-protocols
    • +
  • devpkg-webkitgtk
    • +
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorg-server
  • devpkg-xorgproto
    • @@ -6876,6 +10080,8 @@
  • zstd
    • +
  • binutils
    • +

    @@ -6897,6 +10103,8 @@
  • python2-basic
    • +
  • dnf
    • +

    @@ -6919,16 +10127,38 @@ - - pdns - PowerDNS authoritative name server (DNSSEC enabled) + + pdns + PowerDNS authoritative name server (DNSSEC enabled)
    + +

    Includes bundle(s): +

      + +
    • DNSSEC enabled
      • + +
    • libstdcpp
      • + +
    +

    + + - - pdns-recursor - PowerDNS recursive caching name server + + pdns-recursor + PowerDNS recursive caching name server
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -6948,7 +10178,7 @@
  • python3-basic
    • -
  • tcl-basic
    • +
  • bcc
  • htop
    • @@ -6990,6 +10220,12 @@
  • perl-basic
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -7002,6 +10238,8 @@
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -7010,6 +10248,8 @@
  • kbd
    • +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -7018,6 +10258,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -7026,8 +10268,14 @@
  • patch
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
    • +
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • unzip
    • @@ -7073,6 +10321,14 @@ php-basic Run php language programs
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + @@ -7082,14 +10338,34 @@ pidgin Multi-protocol instant messagging client
    +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + - - plzip - Parallel LZMA Data Compressor + + plzip + Parallel LZMA Data Compressor
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -7117,9 +10393,19 @@ - - powertop - diagnose issues with power consumption and power management + + powertop + diagnose issues with power consumption and power management
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -7175,6 +10461,27 @@ + + pygobject + The pygobject ecosystem
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + python-basic Run python language programs
    @@ -7209,20 +10516,30 @@
  • python-basic
    • +
  • binutils
    • + +
  • bison
    • +
  • devpkg-cairo
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
  • devpkg-freetype
    • @@ -7245,6 +10562,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libpng
  • devpkg-libpthread-stubs
    • @@ -7257,6 +10576,10 @@
  • make
    • +
  • devpkg-ncurses
    • + +
  • openblas
    • +
  • openssl
  • devpkg-openssl
    • @@ -7267,10 +10590,18 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-qtbase
    • + +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -7283,6 +10614,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -7329,6 +10662,8 @@
  • python3-basic
    • +
  • openblas
    • +
  • cloud-api
    • @@ -7339,6 +10674,25 @@ + + python-testing + Improve user experience with a common set of prebuilt python libraries for testing
    + +

    Includes bundle(s): +

      + +
    • cloud-api
      • + +
    • python3-basic
      • + +
    +

    + + + + + + python2-basic Run legacy python language programs
    @@ -7348,6 +10702,10 @@
  • p11-kit
    • +
  • libstdcpp
    • + +
  • openblas
    • +

    @@ -7375,11 +10733,13 @@ qt-basic - Run programs that use the QT runtime
    + Run programs that use the Qt runtime

    Includes bundle(s):

      +
    • fonts-basic
      • +
    • libX11client
    @@ -7403,30 +10763,50 @@
  • qt-basic
    • +
  • devpkg-SDL
    • + +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
  • devpkg-atk
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-bluez
    • +
  • devpkg-cairo
  • devpkg-clutter
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
    • +
  • fonttools
    • +
  • devpkg-freetype
  • devpkg-fribidi
    • @@ -7453,15 +10833,25 @@
  • intltool
    • +
  • devpkg-json-glib
    • +
  • kbd
  • devpkg-libX11
    • +
  • devpkg-libXv
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • -
  • devpkg-libical
    • +
  • devpkg-libffi
    • + +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
  • devpkg-libpciaccess
    • @@ -7479,12 +10869,16 @@
  • llvm
    • +
  • devpkg-llvm
    • +
  • make
  • devpkg-mesa
  • nasm
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -7501,20 +10895,36 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • +
  • devpkg-qt3d
  • devpkg-qtbase
  • devpkg-qtdeclarative
    • +
  • devpkg-qtlocation
    • + +
  • devpkg-qtsvg
    • + +
  • devpkg-qttools
    • + +
  • devpkg-qtwebchannel
    • + +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -7531,6 +10941,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -7584,6 +10996,8 @@
  • libX11client
    • +
  • devpkg-R
    • +
  • plotting, help navigation, etc
  • less
    • @@ -7624,6 +11038,8 @@
  • R-datasets
    • +
  • python3-basic
    • +

    @@ -7646,6 +11062,32 @@ + + rhythmbox + Music playback and management application
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + rsyslog + Rocket-fast system for log processing + + + + ruby-basic Run ruby language programs
    @@ -7655,6 +11097,8 @@
  • p11-kit
    • +
  • libstdcpp
    • +

    @@ -7667,6 +11111,14 @@ runtime-libs-boost Basic C++ / Boost runtime libraries
    +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + @@ -7683,6 +11135,8 @@
  • p11-kit
    • +
  • libstdcpp
    • +

    @@ -7700,14 +11154,14 @@
  • git
    • +
  • nginx
    • +
  • openssh-server
  • perl-basic
  • python3-basic
    • -
  • nginx
    • -

    @@ -7723,6 +11177,25 @@ + + seahorse + No detailed summary available
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libX11client
      • + +
    +

    + + + + + + service-os Provides the needed software for Service OS
    @@ -7732,6 +11205,8 @@
  • bootloader
    • +
  • thermal_daemon
    • +
  • python3-basic
  • iasimage
    • @@ -7757,18 +11232,30 @@
  • service-os
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
  • docbook-utils
    • +
  • docutils
    • +
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-glib
  • gzip
    • @@ -7781,6 +11268,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libpciaccess
  • devpkg-libxkbcommon
    • @@ -7791,6 +11280,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -7801,10 +11292,18 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • + +
  • devpkg-qtbase
    • + +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -7838,6 +11337,8 @@

    Includes bundle(s):

      +
    • libstdcpp
      • +
    • zsh
    @@ -7848,9 +11349,19 @@ - - smartmontools - hard disk diagnostic utilities + + smartmontools + hard disk diagnostic utilities
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -7891,16 +11402,32 @@
  • software-defined-cockpit
    • +
  • devpkg-SDL
    • + +
  • devpkg-SDL2
    • + +
  • devpkg-SDL2_image
    • + +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
  • devpkg-atk
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-bluez
    • +
  • devpkg-cairo
  • devpkg-clutter
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -7911,12 +11438,18 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
    • +
  • fonttools
    • +
  • devpkg-freetype
  • devpkg-fribidi
    • @@ -7943,14 +11476,26 @@
  • intltool
    • +
  • devpkg-json-glib
    • +
  • kbd
  • devpkg-libX11
    • +
  • devpkg-libXv
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • + +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • + +
  • devpkg-libogg
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -7959,6 +11504,8 @@
  • devpkg-libsoup
    • +
  • devpkg-libva
    • +
  • devpkg-libxkbcommon
  • devpkg-libxml2
    • @@ -7967,12 +11514,18 @@
  • llvm
    • +
  • devpkg-llvm
    • +
  • make
  • devpkg-mesa
  • nasm
    • +
  • devpkg-ncurses
    • + +
  • devpkg-opencv
    • +
  • openssl
  • devpkg-openssl
    • @@ -7989,18 +11542,30 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • +
  • devpkg-qtbase
  • devpkg-qtdeclarative
    • +
  • devpkg-qtsvg
    • + +
  • devpkg-qttools
    • + +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -8017,6 +11582,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -8054,9 +11621,38 @@ - - squid - HTTP caching proxy + + squid + HTTP caching proxy
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + + + + + + + stellarium + Software for astronomers
    + +

    Includes bundle(s): +

      + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -8091,6 +11687,8 @@
  • smartmontools
    • +
  • libglib
    • +
  • dosfstools
  • nfs-utils
    • @@ -8122,6 +11720,12 @@
  • storage-utils
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -8132,10 +11736,16 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • + +
  • devpkg-fuse
    • +
  • gdb
  • devpkg-glib
    • @@ -8150,6 +11760,10 @@
  • devpkg-libX11
    • +
  • devpkg-libarchive
    • + +
  • devpkg-libffi
    • +
  • devpkg-libxkbcommon
  • libxml2
    • @@ -8158,10 +11772,14 @@
  • libxslt
    • +
  • devpkg-libxslt
    • +
  • llvm
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -8172,12 +11790,18 @@
  • devpkg-pcre
    • +
  • devpkg-popt
    • +
  • devpkg-qtbase
    • +
  • devpkg-readline
    • +
  • devpkg-sqlite-autoconf
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -8206,9 +11830,11 @@ - - strace - System call diagnostic, instructional, and debugging tool + + strace + System call diagnostic, instructional, and debugging tool
    + + @@ -8236,9 +11862,21 @@ - - suricata - network threat detection engine + + suricata + network threat detection engine
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -8281,6 +11919,8 @@
  • acpica-unix2
    • +
  • libglib
    • +
  • diffutils
  • file
    • @@ -8338,6 +11978,12 @@
  • sysadmin-basic
    • +
  • binutils
    • + +
  • bison
    • + +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -8346,10 +11992,14 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • gdb
  • devpkg-glib
    • @@ -8366,6 +12016,8 @@
  • devpkg-libX11
    • +
  • devpkg-libffi
    • +
  • devpkg-libpng
  • devpkg-libxkbcommon
    • @@ -8376,6 +12028,8 @@
  • make
    • +
  • devpkg-ncurses
    • +
  • openssl
  • devpkg-openssl
    • @@ -8386,10 +12040,14 @@
  • devpkg-pcre
    • -
  • devpkg-qtbase
    • +
  • devpkg-popt
    • + +
  • devpkg-readline
  • strace
    • +
  • devpkg-systemd
    • +
  • texinfo
  • tzdata
    • @@ -8513,9 +12171,59 @@ - - texlive - major TeX-related typesetting programs, macro packages, and fonts. + + texlive + major TeX-related typesetting programs, macro packages, and fonts.
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + texmaker + TeX editor
    + +

    Includes bundle(s): +

      + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + + + + + + + thermal_daemon + thermal management
    + +

    Includes bundle(s): +

      + +
    • libglib
      • + +
    • libstdcpp
      • + +
    +

    + + @@ -8543,6 +12251,27 @@ + + totem + Totem Movie Player plugin API
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    • gnome-base-libs
      • + +
    +

    + + + + + + tzdata timezone database @@ -8550,9 +12279,38 @@ - - unbundle - Recursively resolves a package list from a bundle definition + + uget + GTK+ download manager featuring download classification and HTML import
    + +

    Includes bundle(s): +

      + +
    • libX11client
      • + +
    • libglib
      • + +
    +

    + + + + + + + + unbundle + Recursively resolves a package list from a bundle definition
    + +

    Includes bundle(s): +

      + +
    • python3-basic
      • + +
    +

    + + @@ -8635,8 +12393,12 @@
  • user-basic
    • +
  • devpkg-SDL
    • +
  • acpica-unix2
    • +
  • devpkg-alsa-lib
    • +
  • devpkg-at-spi2-atk
  • devpkg-at-spi2-core
    • @@ -8645,10 +12407,16 @@
  • bc
    • +
  • binutils
    • + +
  • bison
    • +
  • devpkg-cairo
  • cpio
    • +
  • devpkg-curl
    • +
  • devpkg-dbus
  • diffutils
    • @@ -8661,16 +12429,22 @@
  • doxygen
    • +
  • devpkg-e2fsprogs
    • +
  • devpkg-elfutils
  • devpkg-expat
    • +
  • flex
    • +
  • devpkg-fontconfig
  • devpkg-freetype
  • devpkg-fribidi
    • +
  • devpkg-fuse
    • +
  • gdb
  • devpkg-gdk-pixbuf
    • @@ -8693,16 +12467,28 @@
  • iproute2
    • +
  • devpkg-json-glib
    • +
  • kbd
  • less
  • devpkg-libX11
    • +
  • devpkg-libXv
    • + +
  • devpkg-libarchive
    • +
  • devpkg-libdrm
  • devpkg-libepoxy
    • +
  • devpkg-libffi
    • + +
  • devpkg-libinput
    • + +
  • devpkg-libjpeg-turbo
    • +
  • devpkg-libpciaccess
  • devpkg-libpng
    • @@ -8719,20 +12505,32 @@
  • libxslt
    • +
  • devpkg-libxslt
    • +
  • llvm
    • +
  • devpkg-llvm
    • + +
  • devpkg-lua
    • +
  • make
  • devpkg-mesa
    • +
  • devpkg-mozjs52
    • +
  • nasm
    • +
  • devpkg-ncurses
    • +
  • nfs-utils
  • devpkg-nspr
  • devpkg-nss
    • +
  • openblas
    • +
  • devpkg-openmpi
  • openssl
    • @@ -8751,18 +12549,30 @@
  • devpkg-pixman
    • +
  • devpkg-popt
    • +
  • devpkg-qtbase
    • +
  • devpkg-qttools
    • + +
  • devpkg-readline
    • +
  • scons
    • +
  • devpkg-shared-mime-info
    • +
  • devpkg-sqlite-autoconf
  • strace
  • subversion
    • +
  • devpkg-systemd
    • +
  • texinfo
    • +
  • texlive
    • +
  • tzdata
  • unzip
    • @@ -8777,6 +12587,8 @@
  • which
    • +
  • devpkg-xcb-proto
    • +
  • devpkg-xorgproto
  • xz
    • @@ -8795,16 +12607,38 @@ - - v4l-utils - video4linux tools + + v4l-utils + video4linux tools
    + +

    Includes bundle(s): +

      + +
    • qt-basic
      • + +
    • libstdcpp
      • + +
    +

    + + - - valgrind - suite of tools for debugging and profiling software code + + valgrind + suite of tools for debugging and profiling software code
    + +

    Includes bundle(s): +

      + +
    • libstdcpp
      • + +
    +

    + + @@ -8835,6 +12669,8 @@

    Includes bundle(s):

      +
    • python3-basic
      • +
    • desktop-gnomelibs
    • libX11client
      • @@ -8866,9 +12702,19 @@ - - vlc - VideoLAN Client, multimedia player and framework + + vlc + The vlc media player
      + +

      Includes bundle(s): +

        + +
      • libX11client
        • + +
      +

      + + @@ -8911,9 +12757,21 @@ - - weechat - Internet Relay Chat (IRC) client + + weechat + Internet Relay Chat (IRC) client
      + +

      Includes bundle(s): +

        + +
      • IRC
        • + +
      • libstdcpp
        • + +
      +

      + + @@ -8932,6 +12790,25 @@ + + wine + run PE32 executables with Linux
      + +

      Includes bundle(s): +

        + +
      • libX11client
        • + +
      • libstdcpp
        • + +
      +

      + + + + + + wireshark Network packet capture and analysis
      @@ -8998,9 +12875,19 @@ - - xemacs - text editor and application development system + + xemacs + The emacs editor, gui build
      + +

      Includes bundle(s): +

        + +
      • libX11client
        • + +
      +

      + + @@ -9012,6 +12899,8 @@

      Includes bundle(s):

        +
      • fonts-basic
        • +
      • libX11client
      • which
        • @@ -9024,9 +12913,19 @@ - - xfsprogs - XFS tools + + xfsprogs + XFS tools
        + +

        Includes bundle(s): +

          + +
        • libstdcpp
          • + +
        +

        + + @@ -9059,9 +12958,19 @@ - - znc - IRC network bouncer + + znc + IRC network bouncer
        + +

        Includes bundle(s): +

          + +
        • libstdcpp
          • + +
        +

        + + diff --git a/source/clear-linux/reference/bundles/openssh-server.rst b/source/clear-linux/reference/bundles/openssh-server.rst new file mode 100644 index 00000000..02e65cb4 --- /dev/null +++ b/source/clear-linux/reference/bundles/openssh-server.rst @@ -0,0 +1,169 @@ +.. _openssh-server: + +openssh-server +############## + +The **openssh-server** bundle provides the OpenSSH\* package needed to enable +an SSH service in |CL-ATTR|. Remote users require an SSH service to be able +to use an encrypted login shell. + +|CL| enables the `sshd.socket` unit, which will listen on port 22 by default +and start the OpenSSH service as required. The first time OpenSSH starts, it +generates the server SSH keys needed for the service. + +Prerequisites +************* + +Assure the bundle :file:`openssh-server` is installed. + +To check it it's on your host, enter: + +.. code-block:: bash + + sudo swupd bundle-list + +To add it, enter: + +.. code-block:: bash + + sudo swupd bundle-add openssh-server + +Change default port +******************* + +Perform the following steps to change the default listening port for the +OpenSSH service: + +#. Open the sshd.socket file: + + .. code-block:: bash + + sudo systemctl edit sshd.socket + +#. Add the `[Socket]` section and `ListenStream` option to the sshd.socket + file as shown below. The first `ListenStream` entry removes the |CL| + default listen port value. The second `ListenStream` entry sets the new + default listen port value. In this example, we set the new default port + to 4200: + + .. code-block:: console + + [Socket] + ListenStream= + ListenStream=4200 + + + Make sure to include a new line after the last line of text in the sshd.socket file. + +#. Verify your changes: + + .. code-block:: bash + + cat /etc/systemd/system/sshd.socket.d/override.conf + + You should see the following output: + + .. code-block:: console + + [Socket] + ListenStream= + ListenStream=4200 + +#. Reload the systemd daemon configurations: + + .. code-block:: bash + + sudo systemctl daemon-reload + +#. Restart the sshd.socket unit: + + .. code-block:: bash + + sudo systemctl restart sshd.socket + +#. Confirm the the sshd.socket unit is listening on your new port: + + .. code-block:: bash + + sudo systemctl status sshd.socket + + .. note:: + + Output should show :guilabel:`Active:` as `active(listening)`. + +Enable SFTP +*********** + +|CL| *disables* the :abbr:`SFTP (SSH File Transfer Protocol)` subsystem by +default due to security considerations. To enable the SFTP subsystem, perform +the following configuration of the :abbr:`SSHD (SSH Daemon)` service file: + +#. Create a systemd drop-in directory for the SSHD service: + + .. code-block:: bash + + sudo mkdir -p /etc/systemd/system/sshd@.service.d + +#. Create the following file: + :file:`/etc/systemd/system/sshd@.service.d/sftp.conf` + +#. Add the OPTIONS environment variable to the sftp.conf file. + + .. code-block:: console + + [Service] + Environment="OPTIONS=-o Subsystem=\"sftp /usr/libexec/sftp-server\"" + +#. Reload systemd configuration: + + .. code-block:: bash + + sudo systemctl daemon-reload + +Congratulations! The SFTP subsystem is enabled. + +Enable root login +***************** + +To enable root login via SSH, perform the following steps: + +#. Create a *ssh* directory in :file:`/etc`, if it does not already exist. + + .. code-block:: bash + + mkdir /etc/ssh + +#. Create the following file, if it does not already exist: + :file:`/etc/ssh/sshd_config` + +#. Set the configuration variable in /etc/ssh/sshd_config + + .. code-block:: console + + PermitRootLogin yes + +Enable X11-forwarding +********************* + +X11 forwarding allows you to securely run graphical applications +(i.e., X clients) over the ssh conection. This will alow for remote gui apps +without the need for full VNC/remotedesktop. To enable X11-forwarding via +SSH, perform the following steps: + +#. Create a *ssh* directory in :file:`/etc`, if it does not already exist. + + .. code-block:: bash + + mkdir /etc/ssh + +#. Create the following file, if it does not already exist: + :file:`/etc/ssh/sshd_config` + +#. Set the configuration variables. + + .. code-block:: bash + + AllowTcpForwarding yes + X11UseLocalhost yes + X11DisplayOffset 10 + X11Forwarding yes \ No newline at end of file diff --git a/source/clear-linux/reference/image-types.rst b/source/clear-linux/reference/image-types.rst index 5cb734c3..a87fbd56 100644 --- a/source/clear-linux/reference/image-types.rst +++ b/source/clear-linux/reference/image-types.rst @@ -76,6 +76,6 @@ Table 2 lists the currently available images that are platform specific. * - vmware.vmdk - Virtual Machine Disk for VMware\* platforms inclduing Player, Workstation, and ESXi. -.. _images: https://download.clearlinux.org/image +.. _images: https://cdn.download.clearlinux.org/image .. _`optimized kernel`: https://clearlinux.org/documentation/clear-linux/reference/compatible-kernels diff --git a/source/clear-linux/reference/reference.rst b/source/clear-linux/reference/reference.rst index a08ffda6..d0674f57 100644 --- a/source/clear-linux/reference/reference.rst +++ b/source/clear-linux/reference/reference.rst @@ -12,8 +12,10 @@ features. compatible-hardware bundle-commands bundles/bundles + bundles/openssh-server how-to-clear-overview collaboration/collaboration compatible-kernels system-requirements image-types + \ No newline at end of file diff --git a/source/clear-linux/tutorials/dlrs.rst b/source/clear-linux/tutorials/dlrs.rst index 113f3c8e..fdf894e3 100644 --- a/source/clear-linux/tutorials/dlrs.rst +++ b/source/clear-linux/tutorials/dlrs.rst @@ -4,38 +4,45 @@ Deep Learning Reference Stack ############################# This tutorial shows you how to run benchmarking workloads in |CL-ATTR| using -TensorFlow\* and Kubeflow with the Deep Learning Reference Stack. +TensorFlow\* or PyTorch\* with the Deep Learning Reference Stack. We also +cover using Kubeflow for multi-node benchmarking. -The Deep Learning Reference Stack is available in two versions. -The first is `Eigen`_, which includes `TensorFlow`_ optimized for Intel® -architecture. The second is `Intel MKL-DNN`_, which includes the TensorFlow -framework optimized using Intel® Math Kernel Library for Deep Neural -Networks (Intel® MKL-DNN) primitives. - -.. contents:: :local: +.. contents:: + :local: :depth: 1 +The Deep Learning Reference Stack is available in four versions: + +* `Eigen`_, which includes `TensorFlow`_ optimized for Intel® architecture. +* `Intel MKL-DNN`_, which includes the TensorFlow framework optimized using + Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) primitives. +* `PyTorch with OpenBLAS`_, which includes PyTorch with OpenBlas. +* `PyTorch with Intel MKL-DNN`_, which includes PyTorch optimized using + Intel® Math Kernel Library (Intel® MKL)and Intel MKL-DNN. + Release notes -============= +************* -View current `release notes`_ for the Deep Learning Reference Stack. - -View current `benchmark results`_ for the Deep Learning Reference Stack. +* View current `release notes`_ for the Deep Learning Reference Stack. +* View current `TensorFlow benchmark results`_ for the Deep Learning + Reference Stack with TensorFlow. +* View current `PyTorch benchmark results`_ for the Deep Learning Reference + Stack with PyTorch. .. note:: Performance test numbers in the Deep Learning Reference Stack were obtained using `runc` as the runtime. Prerequisites -============= +************* -* |CL| installed on host system. If not installed, :ref:`bare-metal-install` +* |CL| installed on host system. :ref:`Install ` * `containers-basic` bundle * `cloud-native-basic` bundle In |CL|, `containers-basic` provides Docker\*, which is required for -TensorFlow benchmarking. Use the :command:`swupd` utility to check if -`containers-basic` and `cloud-native-basic` are present: +TensorFlow and PyTorch benchmarking. Use the :command:`swupd` utility to +check if `containers-basic` and `cloud-native-basic` are present: .. code-block:: bash @@ -47,7 +54,7 @@ If you need to install the `containers-basic` or `cloud-native-basic`, enter: sudo swupd bundle-add containers-basic cloud-native-basic -To ensure that kubernetes is correctly installed and configured, +To ensure that Kubernetes is correctly installed and configured, follow :ref:`kubernetes`. We have validated these steps against the following software package @@ -59,7 +66,7 @@ versions: * Go 1.11.12 TensorFlow single and multi-node benchmarks -============================================ +******************************************* This section describes running the `TensorFlow benchmarks`_ in single node. For multi-node testing, replicate these steps for each node. These steps @@ -73,13 +80,14 @@ TensorFlow. .. code-block:: bash - docker run --name --rm -i -t bash + docker run --name --rm -i -t bash .. note:: - You will enter the following commands in the running container. - - Replace with the you specified above. + Launching the docker image with the :command:`-i` argument will put + you into interactive mode within the container. You will enter the + following commands in the running container. #. Clone the benchmark repository: @@ -98,28 +106,66 @@ TensorFlow. You can replace the model with one of your choice supported by the TensorFlow benchmarks. +PyTorch single and multi-node benchmarks +**************************************** + +This section describes running the `PyTorch benchmarks`_ for Caffe2 in +single node. We will be looking at validating the Caffe2 APIs with the +official benchmarks, but the same process applies for other cases. + +#. Download either the `PyTorch with OpenBLAS`_ or the `PyTorch with Intel + MKL-DNN`_ docker image + from `Docker Hub`_. + +#. Run the image with Docker: + + .. code-block:: bash + + docker run --name --rm -i -t bash + + .. note:: + + Launching the docker image with the :command:`-i` argument will put + you into interactive mode within the container. You will enter the + following commands in the running container. + +#. Clone the benchmark repository: + + .. code-block:: bash + + git clone https://github.com/pytorch/pytorch.git + +#. Next, execute the benchmark script to run the benchmark. + + .. code-block:: bash + + cd pytorch/caffe2/python + python convnet_benchmarks.py --batch_size 32 \ + --cpu \ + --model AlexNet + Kubeflow multi-node benchmarks -============================== +****************************** The benchmark workload will run in a Kubernetes cluster. We will use `Kubeflow`_ for the Machine Learning workload deployment on three nodes. Kubernetes setup -**************** +================ Follow the instructions in the :ref:`kubernetes` tutorial to get set up on |CL|. The kubernetes community also has `instructions for creating a cluster`_. Kubernetes networking -********************* +===================== We used `flannel`_ as the network provider for these tests. If you are comfortable with another network layer, refer to the Kubernetes `networking documentation`_ for setup. Images -****** +====== We need to add `launcher.py` to our docker image to include the Deep Learning Reference Stack and put the benchmarks repo in the correct @@ -139,12 +185,14 @@ kubeflow. We are working to create these images as part of our release cycle. ksonnet\* -********* +========= -Kubeflow uses ksonnet* to manage deployments, so we need to install that before setting up Kubeflow. +Kubeflow uses ksonnet\* to manage deployments, so we need to install that +before setting up Kubeflow. -Since Clear Linux version 27550, the ksonnet was added to the bundle cloud-native-basic. But if using -old versions (not recommended), please manually install the ksonnet as below. +Since Clear Linux version 27550, the ksonnet was added to the bundle +cloud-native-basic. But if using old versions (not recommended), please +manually install the ksonnet as below. On |CL|, follow these steps: @@ -161,7 +209,7 @@ After the ksonnet installation is complete, ensure that binary `ks` is accessible across the environment. Kubeflow -******** +======== Once you have Kubernetes running on your nodes, you can setup `Kubeflow`_ by following these instructions from their `quick start guide`_. @@ -194,7 +242,7 @@ Now you have all the required kubeflow packages, and you can deploy the primary This creates the CustomResourceDefinition(CRD) endpoint to launch a TFJob. Run a TFJob -=========== +*********** #. Select this link for the `ksonnet registries for deploying TFJobs`_. @@ -229,29 +277,47 @@ Run a TFJob This will replicate and deploy three test setups in your Kubernetes cluster. Results of Running this Tutorial -================================ +******************************** + You need to parse the logs of the Kubernetes pod to get the performance numbers. The pods will still be around post completion and will be in ‘Completed’ state. You can get the logs from any of the pods to inspect the -benchmark results. More information about `Kubernetes logging`_ is available from the Kubernetes community. +benchmark results. More information about `Kubernetes logging`_ is available +from the Kubernetes community. .. _TensorFlow: https://www.tensorflow.org/ + .. _Kubeflow: https://www.kubeflow.org/ + .. _Docker Hub: https://hub.docker.com/ + .. _TensorFlow benchmarks: https://www.tensorflow.org/guide/performance/benchmarks + +.. _PyTorch benchmarks: https://github.com/pytorch/pytorch/blob/master/caffe2/python/convnet_benchmarks.py + .. _instructions for creating a cluster: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ + .. _flannel: https://github.com/coreos/flannel + .. _networking documentation: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network + .. _quick start guide: https://www.kubeflow.org/docs/started/getting-started/ .. _Eigen: https://hub.docker.com/r/clearlinux/stacks-dlrs-oss/ + .. _Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-dlrs-mkl/ +.. _PyTorch with OpenBLAS: https://hub.docker.com/r/clearlinux/stacks-pytorch-oss + +.. _PyTorch with Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-pytorch-mkl + .. _release notes: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs .. _ksonnet registries for deploying TFJobs: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow/dlrs-tfjob - .. _Kubernetes logging: https://kubernetes.io/docs/concepts/cluster-administration/logging/ -.. _benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack +.. _TensorFlow benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack + +.. _PyTorch benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack-pytorch + diff --git a/source/clear-linux/tutorials/greengrass.rst b/source/clear-linux/tutorials/greengrass.rst index f0db1859..0ae516d4 100644 --- a/source/clear-linux/tutorials/greengrass.rst +++ b/source/clear-linux/tutorials/greengrass.rst @@ -1,40 +1,48 @@ .. _greengrass: -Enable AWS Greengrass* and OpenVINO™ on |CL-ATTR| -################################################# +Enable AWS Greengrass\* and OpenVINO™ toolkit +############################################# -Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers -to deploy inference functionalities [1] on Intel® IoT edge devices with -accelerators (CPU, Integrated GPU, Intel® FPGA, and Intel® Movidius™). These -functions provide a great developer experience and seamless migration of -visual analytics from cloud to edge in a secure manner using a containerized -environment. Hardware-accelerated FaaS provides the best-in-class +Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers to +deploy inference functionalities [1] on Intel® IoT edge devices with +accelerators (CPU, Integrated GPU, Intel® FPGA, and Intel® Movidius™ +technology). These functions provide a great developer experience and seamless +migration of visual analytics from cloud to edge in a secure manner using a +containerized environment. Hardware-accelerated FaaS provides the best-in-class performance by accessing optimized deep learning libraries on Intel® IoT edge devices with accelerators. -This tutorial will demonstrate how to: +This tutorial demonstrates how to: * Set up the Intel® edge device with |CL-ATTR| -* Install the OpenVINO™ and AWS Greengrass* software stacks -* Use AWS Greengrass and lambdas to deploy the FaaS samples from the cloud +* Install the OpenVINO™ toolkit and Amazon Web Services\* (AWS\*) + Greengrass\* software stacks +* Use AWS Greengrass\* and AWS Lambda\* to deploy the FaaS samples from the cloud -Supported Platforms +Refer to the following topics: + +.. contents:: :local: + :depth: 1 + + +Supported platforms ******************* * Operating System: |CL| latest release -* Hardware: Intel® core platforms (Tutorial supports inference on CPU only) +* Hardware: Intel® core platforms (This tutorial supports inference on CPU only.) -Description of Samples -********************** +Sample description +================== -The AWS Greengrass samples are located at the `Edge-Analytics-FaaS`_. For this tutorial we will use the 1.0 version of the source code. +The AWS Greengrass samples are located at `Edge-Analytics-FaaS`_. This +tutorial uses the 1.0 version of the source code. -We provide the following AWS Greengrass samples: +|CL| provides the following AWS Greengrass samples: * `greengrass_classification_sample.py`_ This AWS Greengrass sample classifies a video stream using classification - networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS* + networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS\* IoT Cloud every second. * `greengrass_object_detection_sample_ssd.py`_ @@ -46,8 +54,8 @@ We provide the following AWS Greengrass samples: coordinates on AWS IoT Cloud every second. -Installing |CL| on the edge device -********************************** +Install the OS on the edge device +********************************* Start with a clean installation of |CL| on a new system, using the :ref:`bare-metal-install`, found in :ref:`get-started`. @@ -56,8 +64,8 @@ Create user accounts ==================== After |CL| is installed, create two user accounts. Create an administrative -user in |CL|. You will also create a user account for the Greengrass -services to use (see Greengrass user below). +user in |CL| and create a user account for the Greengrass services to use (see +Greengrass user below). #. Create a new user and set a password for that user. Enter the following commands as ``root``: @@ -68,7 +76,7 @@ services to use (see Greengrass user below). passwd #. Next, enable the :command:`sudo` command for your new ````. Add - ```` to the ``wheel`` group: + ```` to the *wheel* group: .. code-block:: bash @@ -82,14 +90,13 @@ services to use (see Greengrass user below). .. note:: - By default |CL| does not create an :file:`/etc/fstab` file. - The Greengrass service needs to have the file created before - it will run. + By default, |CL| does not create an :file:`/etc/fstab` file. + You must create this file before the Greengrass service runs. Add required bundles ==================== -Use the ``swupd`` software updater utility to add the prerequisite bundles +Use the :command:`swupd` software updater utility to add the prerequisite bundles for the OpenVINO software stack: .. code-block:: bash @@ -100,20 +107,20 @@ for the OpenVINO software stack: Learn more about how to :ref:`swupd-guide`. -The ``computer-vision-basic`` bundle will install the OpenVINO software, -along with the edge device models needed. +The :command:`computer-vision-basic` bundle installs the OpenVINO™ toolkit, +and the sample models optimized for Intel® edge platforms. -Converting Deep Learning Models -=============================== +Convert deep learning models +============================ -Locate Sample Models +Locate sample models -------------------- -There are two types of provided models that can be used in conjunction with AWS Greengrass -for this tutorial: classification or object detection. +There are two types of provided models that can be used in conjunction with +AWS Greengrass for this tutorial: classification or object detection. To complete this tutorial using an image classification model, -download the BVLC Alexnet model files `bvlc_alexnet.caffemodel`_ and `deploy.prototxt`_ +download the BVLC AlexNet model files `bvlc_alexnet.caffemodel`_ and `deploy.prototxt`_ to the default model_location at :file:`/usr/share/openvino/models`. Any custom pre-trained classification models can be used with the classification sample. @@ -123,12 +130,12 @@ are included with the computer-vision-basic bundle installation at :file:`/usr/s These models are provided as an example; however, you may also use a custom SSD model with the Greengrass object detection sample. -Running Model Optimizer ------------------------ +Run model optimizer +------------------- -Follow these instructions for `converting deep learning models to Intermediate Representation using Model Optimizer`_. To optimize either of the afformentioned sample models, run one of the following commands. +Follow these instructions for `converting deep learning models to Intermediate Representation using Model Optimizer`_. To optimize either of the sample models described above, run one of the following commands. -For classification using BVLC Alexnet model: +For classification using BVLC AlexNet model: .. code-block:: bash @@ -148,40 +155,40 @@ For object detection using SqueezeNetSSD-5Class model: In these examples: -* ```` is :file:`/usr/share/openvino/models` +* ```` is :file:`/usr/share/openvino/models`. * ```` is FP32 or FP16, depending on target device. -* ```` is the directory where the user wants to store the - Intermediate Representation (IR). IR contains .xml format corresponding - to the network structure and .bin format corresponding to weights. This - .xml file should be passed to . +* ```` is the directory where the Intermediate Representation + (IR) is stored. IR contains .xml format corresponding to the network + structure and .bin format corresponding to weights. This .xml file should be + passed to :command:``. -* In the BVLC Alexnet model, the prototxt defines the input shape with +* In the BVLC AlexNet model, the prototxt defines the input shape with batch size 10 by default. In order to use any other batch size, the - entire input shape needs to be provided as an argument to the model - optimizer. For example, to use batch size 1, you can provide - “--input_shape [1,3,227,227]”. + entire input shape must be provided as an argument to the model + optimizer. For example, to use batch size 1, you must provide: + ``--input_shape [1,3,227,227]`` -Configuring an AWS Greengrass group -=================================== +Configure AWS Greengrass group +****************************** -For each Intel® edge platform, we need to create a new AWS Greengrass group +For each Intel® edge platform, you must create a new AWS Greengrass group and install AWS Greengrass core software to establish the connection between cloud and edge. -#. To create an AWS Greengrass group, follow the - `AWS Greengrass developer guide`_ +#. To create an AWS Greengrass group, follow the instructions in + `Configure AWS IoT Greengrass on AWS IoT`_. #. To install and configure AWS Greengrass core on edge platform, follow - the instructions at `Start AWS Greengrass on the Core Device`_. In - step 8(b), download the x86_64 Ubuntu configuration of the AWS Greengrass + the instructions in `Start AWS Greengrass on the Core Device`_. In + step 8(b), download the x86_64 Ubuntu\* configuration of the AWS Greengrass core software. .. note:: - You will not need to run the ``cgroupfs-mount.sh`` script in step #6 + You do not need to run the :file:`cgroupfs-mount.sh` script in step #6 of Module 1 of the `AWS Greengrass developer guide`_ because this is enabled already in |CL|. @@ -190,13 +197,13 @@ cloud and edge. .. note:: - Security certificates are linked to your AWS* account. + Security certificates are linked to your AWS account. -Creating and Packaging Lambda Functions -======================================= +Create and package Lambda function +********************************** -#. Complete steps 1-4 of the tutorial at `Create and Package Lambda Function`_ . +#. Complete steps 1-4 of the AWS Greengrass tutorial at `Create and Package a Lambda Function`_. .. note:: @@ -204,7 +211,7 @@ Creating and Packaging Lambda Functions environment on the edge device. -#. In step 5, replace greengrassHelloWorld.py with the classification or object detection +#. In step 5, replace :file:`greengrassHelloWorld.py` with the classification or object detection Greengrass sample from `Edge-Analytics-Faas`_: * Classification: `greengrass_classification_sample.py`_ @@ -227,36 +234,35 @@ Creating and Packaging Lambda Functions zip -r greengrass_lambda.zip greengrasssdk greengrass_object_detection_sample_ssd.py -#. Return to the AWS Documentation and follow steps 6-11 to `complete creating lambdas`_. +#. Return to the AWS documentation section called `Create and Package a Lambda Function`_ + and complete the procedure. .. note:: In step 9(a) of the AWS documentation, while uploading the zip file, - make sure to name the handler as below depending on the AWS Greengrass - sample you are using: + make sure to name the handler to one of the following, depending on the + AWS Greengrass sample you are using: - * greengrass_object_detection_sample_ssd.function_handler (or) + * greengrass_object_detection_sample_ssd.function_handler * greengrass_classification_sample.function_handler -Deploying Lambdas -================= -Configuring the Lambda function -------------------------------- +Configure Lambda function +************************* -After creating the Greengrass group and the lambda function, start -configuring the lambda function for AWS Greengrass. +After creating the Greengrass group and the Lambda function, start +configuring the Lambda function for AWS Greengrass. -#. Follow steps 1-8 in `Configure the Lambda Function`_ of the AWS +#. Follow steps 1-8 in `Configure the Lambda Function for AWS IoT Greengrass`_ in the AWS documentation. #. In addition to the details mentioned in step 8, change the Memory limit - to 2048MB to accommodate large input video streams. + to 2048 MB to accommodate large input video streams. #. Add the following environment variables as key-value pairs when editing - the lambda configuration and click on update: + the Lambda configuration and click on update: - .. list-table:: **Table 1. Environment Variables: Lambda Configuration** + .. list-table:: **Table 1. Environment variables: Lambda configuration** :widths: 20 80 :header-rows: 1 @@ -282,94 +288,100 @@ configuring the lambda function for AWS Greengrass. (e.g. 1 for top-1 result, 5 for top-5 results) #. Add subscription to subscribe, or publish messages from AWS Greengrass - lambda function by following the steps 10-14 in `Configure the Lambda Function`_ + Lambda function by completing the procedure in `Configure the Lambda Function for AWS IoT Greengrass`_. .. note:: - The “Optional topic filter” field should be the topic - mentioned inside the lambda function. + The optional topic filter field is the topic mentioned inside the Lambda + function. In this tutorial, sample topics include the following: + :command:`openvino/ssd` or :command:`openvino/classification` - For example, openvino/ssd or openvino/classification +Add local resources +=================== -Local Resources ---------------- -#. Select `this link to add local resources and access privileges`_. +Refer to the AWS documentation for details about `local resources and access privileges`_. - Following are the local resources needed for the CPU: +The following table describes the local resources needed for the CPU: - .. list-table:: **Local Resources** - :widths: 20, 20, 20, 20 - :header-rows: 1 +.. list-table:: **Local resources** + :widths: 20, 20, 20, 20 + :header-rows: 1 - * - Name - - Resource type - - Local path - - Access + * - Name + - Resource type + - Local path + - Access - * - ModelDir - - Volume - - to be specified by user - - Read-Only + * - ModelDir + - Volume + - to be specified by user + - Read-Only - * - Webcam - - Device - - /dev/video0 - - Read-Only + * - Webcam + - Device + - /dev/video0 + - Read-Only - * - DataDir - - Volume - - to be specified by user. Holds both input and output - data. - - Read and Write + * - DataDir + - Volume + - to be specified by user. Holds both input and output + data. + - Read and Write -Deploy ------- +Deploy Lambda function +********************** -To `deploy the lambda function to AWS Greengrass core device`_, select -“Deployments” on group page and follow the instructions. +Refer to the AWS documentation for instructions on how to +`deploy the lambda function to AWS Greengrass core device`_. Select +*Deployments* on the group page and follow the instructions. -Output Consumption ------------------- +Output consumption +================== There are four options available for output consumption. These options are used to report, stream, upload, or store inference output at an interval -defined by the variable ``reporting_interval`` in the AWS Greengrass samples. +defined by the variable :command:`reporting_interval` in the AWS Greengrass samples. -a. IoT Cloud Output: - This option is enabled by default in the AWS Greengrass samples using a - variable ``enable_iot_cloud_output``. We can use it to verify the lambda +a. IoT cloud output: + + This option is enabled by default in the AWS Greengrass samples using the + :command:`enable_iot_cloud_output` variable. You can use it to verify the lambda running on the edge device. It enables publishing messages to IoT cloud - using the subscription topic specified in the lambda (For example, - ‘openvino/classification’ for classification and ‘openvino/ssd’ for - object detection samples). For classification, top-1 result with class + using the subscription topic specified in the lambda. (For example, topics + may include :command:`openvino/classification` for classification and :command:`openvino/ssd` + for object detection samples.) For classification, top-1 result with class label are published to IoT cloud. For SSD object detection, detection - results such as bounding box co-ordinates of objects, class label, and + results such as bounding box coordinates of objects, class label, and class confidence are published. - Follow the instructions here to `view the output on IoT cloud`_ + Follow the instructions here to `view the output on IoT cloud`_. -b. Kinesis Streaming: +b. Kinesis streaming: This option enables inference output to be streamed from the edge device - to cloud using Kinesis [3] streams when ‘enable_kinesis_output’ is set + to cloud using Kinesis [3] streams when :command:`enable_kinesis_output` is set to True. The edge devices act as data producers and continually push - processed data to the cloud. The users need to set up and specify + processed data to the cloud. You must set up and specify Kinesis stream name, Kinesis shard, and AWS region in the AWS Greengrass samples. -c. Cloud Storage using AWS S3 Bucket: +c. Cloud storage using AWS S3 bucket: - When the ‘enable_s3_jpeg_output’ variable is set to True, it enables uploading and storing processed frames (in JPEG format) in an AWS S3 bucket. The users need to set up and specify the S3 bucket name in the - AWS Greengrass samples to store the JPEG images. The images are named using the timestamp and uploaded to S3. + When the :command:`enable_s3_jpeg_output` variable is set to True, it enables + uploading and storing processed frames (in jpeg format) in an AWS S3 + bucket. You must set up and specify the S3 bucket name in the AWS + Greengrass samples to store the JPEG images. The images are named using the + timestamp and uploaded to S3. -d. Local Storage: +d. Local storage: - When the ‘enable_s3_jpeg_output’ variable is set to True, it enables storing processed frames (in JPEG format) on the edge device. The - images are named using the timestamp and stored in a directory specified - by ‘PARAM_OUTPUT_DIRECTORY’. + When the :command:`enable_s3_jpeg_output` variable is set to True, it enables + storing processed frames (in jpeg format) on the edge device. The images + are named using the timestamp and stored in a directory specified by + :command:`PARAM_OUTPUT_DIRECTORY`. References ------------ +********** 1. AWS Greengrass: https://aws.amazon.com/greengrass/ 2. AWS Lambda: https://aws.amazon.com/lambda/ @@ -387,17 +399,15 @@ References .. _converting deep learning models to Intermediate Representation using Model Optimizer: https://software.intel.com/en-us/articles/OpenVINO-ModelOptimizer -.. _AWS Greengrass developer guide: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html +.. _AWS Greengrass Developer Guide: https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html + +.. _Configure AWS IoT Greengrass on AWS IoT: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html .. _Start AWS Greengrass on the Core Device: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-device-start.html -.. _AWS Greengrass Core SDK: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html +.. _Configure the Lambda Function for AWS IoT Greengrass: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html -.. _complete creating lambdas: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html - -.. _Configure the Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html - -.. _Add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html +.. _local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html .. _deploy the lambda function to AWS Greengrass core device: https://docs.aws.amazon.com/greengrass/latest/developerguide/configs-core.html @@ -407,4 +417,4 @@ References .. _this link to add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html -.. _Create and Package Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html +.. _Create and Package a Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html diff --git a/source/clear-linux/tutorials/kubernetes/kubernetes-bp.rst b/source/clear-linux/tutorials/kubernetes/kubernetes-bp.rst new file mode 100644 index 00000000..a7126e2b --- /dev/null +++ b/source/clear-linux/tutorials/kubernetes/kubernetes-bp.rst @@ -0,0 +1,92 @@ +.. _kubernetes-bp: + +Kubernetes Best Practices on |CL| +################################# + +Use swupd to update clusters +**************************** + +This tutorial shows you how to manage your Kubernetes cluster while using +:command:`swupd` to update |CL-ATTR|. + +In our tutorial :ref:`kubernetes`, we explain how to set up a Kubernetes +cluster on |CL| using `kubeadm`. `Kubeadm documentation`_ often builds on the +assumption that the distribution uses a traditional package manager (e.g., +RPM/DEB). + +In contrast, |CL| uses `swupd` to update the OS, which in this case updates +all of the kubernetes node and client binaries simultaneously, as part of +the `cloud-native-basic` bundle (e.g., kubectl, kubeadm, kubelet). Running +:command:`sudo swupd update` requires special care to ensure the OS +incorporates the latest Kubernetes upgrades. + +This document describes best practices to manage cluster upgrades with +`kubeadm` on a |CL|-based cluster. + +Prerequisites +************* + +Assure that you: + +* Completed :ref:`kubernetes` +* Installed the bundle `cloud-native-basic` + +.. note:: + + Other Linux\* distros shown in Kubernetes upgrade documentation reflect + `apt-get update`, `apt-mark hold kubeadm`, and similar commands; however, such commands **are not valid** on |CL|. + +Update the control plane +************************ + +#. Read kubernetes documentation `before you begin`_. + +#. On your master node, run the command: + + .. code-block:: bash + + sudo swupd update + + .. note:: + + If the minor version of Kubernetes changes, |CL| shows a message-of-the-day, or `motd`. When the `motd` appears, you **must postpone** a kubelet restart on master and nodes until the control plane is properly updated. :command:`swupd update` does not restart services automatically unless explicitly configured to do so. + +#. Now follow these instructions in kubernetes documentation. + + * `Upgrade control plane`_ + * `Drain control plane node`_ + * `Restart Kubelet and undrain node`_ + +Update worker nodes +******************* + +#. On each worker node, run the command: + + .. code-block:: bash + + sudo swupd update + +#. Now follow these instructions in kubernetes documentation: + + * `Drain node`_ + * `Update kubelet configuration`_ + * `Restart Kubelet and undrain node`_ + +.. _Kubeadm documentation: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/ + +.. _Restart Kubelet and undrain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#restart-the-kubelet-for-all-nodes + +.. _Update kubelet configuration: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#upgrade-the-kubelet-config-on-worker-nodes + +.. _Drain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes + + .. _Restart kubelet and undrain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#restart-the-kubelet-for-all-nodes + +.. _Upgrade control plane: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#upgrade-the-control-plane-node + +.. _Drain control plane node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes + +.. _Kubeadmn documentation: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/ + +.. _before you begin: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#before-you-begin + diff --git a/source/clear-linux/tutorials/kubernetes.rst b/source/clear-linux/tutorials/kubernetes/kubernetes.rst similarity index 90% rename from source/clear-linux/tutorials/kubernetes.rst rename to source/clear-linux/tutorials/kubernetes/kubernetes.rst index f05d4109..c8c65a3a 100644 --- a/source/clear-linux/tutorials/kubernetes.rst +++ b/source/clear-linux/tutorials/kubernetes/kubernetes.rst @@ -1,7 +1,7 @@ .. _kubernetes: -Run Kubernetes\* on |CL-ATTR| -############################# +Run Kubernetes\* +################ This tutorial describes how to install, configure, and run the `Kubernetes container orchestration system`_ on |CL-ATTR| using CRI+O and @@ -43,17 +43,16 @@ Kubernetes, a set of supported :abbr:`CRI (Container Runtime Interface)` runtimes, and networking plugins, are included in the `cloud-native-basic`_ bundle. -.. note:: - - CRI-O’s default plugin_dir is :file:`/opt/bin/cni`. - CNI plugins are installed as part of ``cloud-native-basic``. - To install this framework, enter the following command: .. code-block:: bash sudo swupd bundle-add cloud-native-basic +.. note:: + + For more on networking plugins, see `Install pod network add-on`_. + Configure Kubernetes ******************** @@ -150,7 +149,6 @@ Configure and run CRI-O + kata-runtime sudo systemctl daemon-reload sudo systemctl restart crio - sudo systemctl restart kubelet #. Initialize the master control plane with the command: @@ -158,7 +156,6 @@ Configure and run CRI-O + kata-runtime sudo kubeadm init --cri-socket=/run/crio/crio.sock - Install pod network add-on ************************** @@ -166,47 +163,49 @@ You must choose and install a `pod network add-on`_ to allow your pods to communicate. Check whether or not your add-on requires special flags when you initialize the master control plane. +The CRI-O default plugin_dir is :file:`/opt/cni/bin`. This must be a +writable directory because third-party networking add-ons will install +themselves there. + +.. note:: + + CNI plugins provided by |CL| are installed as part of *cloud-native-basic* + in :file:`/usr/libexec/cni/` and are currently *not* found by CRI-O by + default. These separate directories are required because `swupd` controls + the content of :file:`/usr` and leaves :file:`/opt` unchanged. + +When using third-party network add-ons that rely on those plugins, such as +Weave or Flannel do, make them available by creating symlinks: + +.. code-block:: bash + + sudo mkdir -p /opt/cni/bin + +.. code-block:: bash + + for i in /usr/libexec/cni/*; do sudo ln -sf $i /opt/cni/bin; done + +**Notes about Weave Net add-on** + +The Weave Net add-on works by default when the above configuration is done. + **Notes about flannel add-on** If you choose the `flannel` add-on, then you must add the following to the `kubeadm init` command: -.. code-block:: bash +.. code-block:: bash - --pod-network-cidr 10.244.0.0/16 + --pod-network-cidr 10.244.0.0/16 -If you are using CRI-O and `flannel` and you want to use Kata Containers, edit the :file:`/etc/crio/crio.conf` file to add: +If you are using CRI-O and `flannel` and you want to use Kata Containers, +edit the :file:`/etc/crio/crio.conf` file to add: .. code-block:: bash [crio.runtime] manage_network_ns_lifecycle = true -Create a symlink for the network overlays: - -.. code-block:: bash - - sudo ln -s /usr/libexec/cni /opt/cni/bin - -.. note:: - - |CL| installs CNI plugins that are part of the `cloud-native-basic` - bundle to :file:`/usr/libexec/cni`. The directory is required because ` - swupd verify` may use it to repair a system to a known good state. - -**Notes about Weave Net add-on** - -If you choose the `Weave Net` add-on, you must make the following -changes because it installs itself in the :file:`/opt/cni/bin` directory. - -For using CRI-O and ``Weave Net``, complete the following step. - -Add the `loopback` CNI plugin to the plugin path with the command: - -.. code-block:: bash - - sudo ln -s /usr/libexec/cni/loopback /opt/bin/cni/loopback - Use your cluster **************** @@ -240,8 +239,8 @@ Read the Kubernetes documentation to learn more about: * `Joining your nodes`_ -Package configuration customization in |CL| (optional) -****************************************************** +Package configuration customization (optional) +********************************************** |CL| is a stateless system that looks for user-defined package configuration files in the :file:`/etc/` directory to be used as default. If @@ -289,6 +288,12 @@ commands as a shell script to configure all of these services in one step: EOF done +Next steps +********** + +:ref:`kubernetes-bp` + + Troubleshooting *************** diff --git a/source/clear-linux/tutorials/multi-boot/multi-boot.rst b/source/clear-linux/tutorials/multi-boot/multi-boot.rst index 778729d0..8a8c8f80 100644 --- a/source/clear-linux/tutorials/multi-boot/multi-boot.rst +++ b/source/clear-linux/tutorials/multi-boot/multi-boot.rst @@ -55,7 +55,7 @@ installation of the tested operating systems. .. csv-table:: Table 1: OS specific installation information :header: # , OS, Version, Partition Size [#]_, Swap Size [#]_, EFI Partition Size [#]_, Download Link - 1,Clear Linux,16140,50 GB,8 GB,1 GB,https://download.clearlinux.org/releases/16140/clear/ + 1,Clear Linux,16140,50 GB,8 GB,1 GB,https://cdn.download.clearlinux.org/releases/16140/clear/ 2,Windows,Server 2016,50 GB,N/A,Shared with #1,https://www.microsoft.com/en-us/cloud-platform/windows-server 3,Red Hat\*,Server 7.4 Beta,45 GB,Shared with #1,Shared with #1,https://access.redhat.com/downloads/ 4,SUSE\*,Server 12 SP2,45 GB,Shared with #1,Shared with #1,https://www.suse.com/download-linux/ diff --git a/source/clear-linux/tutorials/nvidia.rst b/source/clear-linux/tutorials/nvidia.rst new file mode 100644 index 00000000..0f3850f7 --- /dev/null +++ b/source/clear-linux/tutorials/nvidia.rst @@ -0,0 +1,180 @@ +.. _nvidia: + +Install NVIDIA Drivers +###################### + +NVIDIA is a manufacture of graphics processing units (GPU), also known as +graphics cards. + +NVIDIA devices on Linux have two popular device driver options: the opensource +drivers from the `nouveau project`_ or the proprietary drivers published by +NVIDIA. The nouveau drivers are built into the |CL-ATTR| kernel and are loaded +automatically at system boot if a compatible card is detected. + +These instructions show how to use the proprietary NVIDIA drivers which +require a manual installation. + + +.. contents:: :local: + :depth: 2 + + + +Prerequisites +************* + +* A |CL| system with a desktop installed +* A NVIDIA device installed + + +Install the LTS kernel and DKMS +******************************* + +The Long Term Support (LTS) kernel variant is most likely to remain +compatible with NVIDIA drivers. The `Dynamic Kernel Module System (DKMS) +`_ allows the NVIDIA kernel modules to be automatically +integrated when kernel updates occur in |CL|. Install both using the +instructions below: + + +.. include:: ../guides/maintenance/kernel-modules-dkms.rst + :start-after: kernel-modules-dkms-install-begin: + :end-before: kernel-modules-dkms-install-end: + + +Download and install the NVIDIA Linux Driver +******************************************** + + +Download the NVIDIA Linux Driver +================================ + +#. Identify the model of NVIDIA GPU that is installed. + + .. code-block:: bash + + lshw -C display + + +#. Go to the `NVIDIA Driver Downloads website`_ . Search for and download the + appropriate driver based on the model of NVIDIA GPU you have with *Linux + 64-bit* selected as the Operating System . + + +#. Open a terminal and navigate to where the + :file:`NVIDIA-Linux-x86_64-.run` file was saved. In this + example, it was saved in the Downloads folder. + + .. code-block:: bash + + cd ~/Downloads/ + + +#. Extract the contents of the .run file. + + .. code-block:: bash + + sh NVIDIA-Linux-x86_64-.run --extract-only + + +Disable the nouveau driver +========================== + +#. The proprietary NVIDIA driver is incompatible with the nouveau driver and + needs to be disabled before installation can continue. + + Disable the nouveau driver by creating a file under :file:`/etc/modprobe.d` + and reboot. + + .. code-block:: bash + + sudo mkdir /etc/modprobe.d + + echo "blacklist nouveau" | sudo tee --append /etc/modprobe.d/nvidia-disable-nouveau.conf + echo "options nouveau modeset=0" | sudo tee --append /etc/modprobe.d/nvidia-disable-nouveau.conf + + +#. Reboot the system and log back in. It is normal for the graphical + environment to not start with no NVIDIA driver loaded. + + +Install the NVIDIA Linux Driver +=============================== + +#. Navigate into the directory where the NVIDIA installer was extracted. + + .. code-block:: bash + + cd ~/Downloads/NVIDIA-Linux-x86_64-/ + + +#. Run the installer with the advanced options below. + + .. code-block:: bash + + sudo ./nvidia-installer --no-nvidia-modprobe --no-distro-scripts --no-opengl-files --no-libglx-indirect --no-install-libglvnd --no-install-compat32-libs --dkms --ui=none + +#. The installer will prompt to register the kernel module sources with + DKMS. Enter Y for yes. + + .. code-block:: bash + + Welcome to the NVIDIA Software Installer for Unix/Linux + + + + Would you like to register the kernel module sources with DKMS? This will allow DKMS to automatically build a new + module, if you install a different kernel later. + [default: (Y)es]: Y + + + +#. The graphical interface may automatically start after the NVIDIA driver + is loaded. If it does restart, log back in. + + +#. Validate the nvidia kernel modules are loaded. + + .. code-block:: bash + + lsmod | grep ^nvidia + + +.. note:: + + The NVIDIA installer places files under the :file:`/usr` subdirectory which + are not managed by |CL| updates. The :command:`swupd verify --fix` command + should be avoided with the proprietary NVIDIA drivers in use. + + +Uninstalling the NVIDIA driver +****************************** + +The NVIDIA drivers and associated software can be uninstalled and nouveau +driver restored by: + +#. Remove the previously created file :file:`etc/modprobe.d` that is + preventing nouveau from loading. + + .. code-block:: bash + + sudo rm /etc/modprobe.d/nvidia-disable-nouveau.conf + + +#. Run the :command:`sudo nvidia-uninstall` + +#. Follow the prompts on the screen and reboot the system. + + + +Additional resources +******************** + +* `Why aren't the NVIDIA Linux drivers open source? `_ + +* `Where can I get support for NVIDIA Linux drivers? `_ + + +.. _`nouveau project`: https://nouveau.freedesktop.org/wiki/ + +.. _`NVIDIA Driver Downloads website`: https://www.nvidia.com/download/index.aspx diff --git a/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst b/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst deleted file mode 100644 index d0293d96..00000000 --- a/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst +++ /dev/null @@ -1,477 +0,0 @@ -.. _telemetry-backend: - -Create a telemetry backend server in |CL-ATTR| -############################################## - -|CL-ATTR| includes a telemetry and analytics solution, also known as -telemetrics, as part of the OS that records events of interest and reports -them back to the development team using the telemetrics daemons that are -running on the |CL| client system. - -End users can enable or disable the telemetry client component of |CL| and -also redirect where records go if they wish to collect records for themselves -by setting up and using their own telemetry backend server. - -A telemetry backend server consists of two Flask applications: - -* The :guilabel:`collector` is an ingestion app for records received from the - :guilabel:`telemetrics-client` probes. -* The :guilabel:`telemetryui` web app exposes several visualizations of the - telemetry data. The :guilabel:`telemetryui` app also provides a - REST API to perform queries on the data. - -The applications run within a web stack, using the :guilabel:`nginx` web -server, the :guilabel:`uWSGI` application server, and -:guilabel:`PostgreSQL` as the underlying database server. For a detailed -description, visit the |CL| `telemetrics backend`_ server overview. - -This tutorial walks you through creating a telemetrics backend server on -your local |CL| machine. The tutorial uses the :command:`deploy.sh` bash -shell script that is maintained in a the `telemetrics backend`_ GitHub repository. -Once the backend server isup and running, we show you how to redirect telemetry -records from the system you set up to this new server location. - -.. note:: - The telemetrics functionality adheres to `Intel privacy policies`_ - regarding the collection and use of :abbr:`PII (Personally Identifiable - Information)` and is open Source. Specifically, no intentionally - identifiable information about the user or system owner is collected. - -Prerequisites -************* - -For this tutorial, start with a clean installation of |CL| on a new system -using the :ref:`bare-metal-install` getting started guide: - -#. Choose to install |CL|. -#. Join the :guilabel:`Stability Enhancement Program` to install and - enable the telemetrics components. -#. Select the manual installation method with the following settings: - - * Set the hostname to :guilabel:`clr-telem-server`, - * Create an administrative user named :guilabel:`clear` and add this user - to sudoers, - * Add all additional software bundles. - -Download the clearlinux/telemetrics-backend Git repository -********************************************************** - -With all prerequisite software bundles installed and logged in with your -administrative user, from your :file:`$HOME` directory, run :command:`git` -to clone the :guilabel:`telemetrics-backend` repository into the -:file:`$HOME/telemetrics-backend` directory: - -.. code-block:: console - - git clone https://github.com/clearlinux/telemetrics-backend - -.. note:: - - You may need to set up the :envvar:`https_proxy` environment variable - if you have issues reaching github.com. - -Run the deploy.sh script to install the backend server -****************************************************** - -#. Change your current working directory to :file:`telemetrics-backend/scripts`. -#. Run the :command:`./deploy.sh -h` to see the list of options for the - :command:`deploy.sh` script: - -.. code-block:: console - - cd telemetrics-backend/scripts - ./deploy.sh -h - - Deploy snapshot of the telemetrics-backend - - -a Perform specified action (deploy, install, migrate, resetdb, - restart, uninstall; default: deploy) - -d Distro to deploy to (ubuntu, centos or clr; default: ubuntu) - -h Print these options - -H Set domain for deployment (only accepted value is "localhost" for - now) - -r Set repo location to deploy from - (default: https://github.com/clearlinux/telemetrics-backend) - -s Set source location (default: "master" branch from git repo) - -t Set source type (tarball, or git; default: git) - -u Perform complete uninstallation - -The :command:`deploy.sh` is a bash shell script that allows you to perform the -following actions: - -* *deploy* - install a complete instance of the telemetrics backend - server and all required components. This is the default action if no - *-a* argument is given on the command line. -* *install* - installs and enables all required components for the - telemetrics backend server. -* *migrate* - migrate database to new schema. -* *resetdb* - reset the database. -* *restart* - restart the nginx and uWSGI services. -* *uninstall* - uninstall all packages. - - .. note:: - - The *uninstall* option does not perform any actions if the - distro is set to |CL| and will only uninstall packages if the distro is - Ubuntu - -Next, we install the telemetrics backend server with the following options: - -* *-a install* to perform an install -* *-d clr* to install to a |CL| distro -* *-H localhost* to set the domain to localhost - -We do not need to set the following options since the values are set to the -correct values we want by default: - -* *-r https://github.com/clearlinux/telemetrics-backend* sets the - repo location for :command:`git` to clone from. -* *-s master* to set the location, or branch. -* *-t git* to set the source type to git. - -.. caution:: - The :file:`deploy.sh` shell script has minimal error checking and makes - several changes to your system. Be sure that the options you define on the - cmdline are correct before proceeding. - -To begin the installation with the options defined: - -#. Run the shell script from the :file:`$HOME/telemetrics-backend/scripts` - directory: - - .. code-block:: console - - ./deploy.sh -H localhost -a install -d clr - - The script will start and list all the defined options and prompt you for the - :guilabel:`PostgreSQL` database password as shown below: - - .. code-block:: console - - Options: - host: localhost - distro: clr - action: install - repo: https://github.com/clearlinux/telemetrics-backend - source: master - type: git - DB password: (default: postgres): - -#. For the :guilabel:`DB password:`, press the :kbd:`Enter` key to accept the - default password `postgres`. - -The :command:`swupd` begins installing the required software bundles to set -up the telemetrics backend server. The output will look similar to what is -shown below: - -.. code-block:: console - - swupd-client bundle adder 3.12.7 - Copyright (C) 2012-2017 Intel Corporation - - Downloading packs... - - Extracting application-server pack for version 18740 - ...5% - Extracting database-basic-dev pack for version 18670 - ...10% - Extracting database-basic pack for version 18670 - ...15% - Extracting os-clr-on-clr pack for version 18740 - ...21% - Extracting sysadmin-basic-dev pack for version 18740 - ...26% - Extracting storage-utils-dev pack for version 18770 - ...31% - Extracting os-core-update-dev pack for version 18760 - ...36% - Extracting network-basic-dev pack for version 18760 - ...42% - Extracting mixer pack for version 18790 - ...47% - Extracting os-installer pack for version 18800 - ...52% - Extracting mail-utils-dev pack for version 18760 - ...57% - Extracting koji pack for version 18800 - ...63% - Extracting go-basic pack for version 18800 - ...68% - Extracting dev-utils-dev pack for version 18820 - ...73% - Extracting python-basic-dev pack for version 18750 - ...78% - Extracting perl-basic-dev pack for version 18610 - ...84% - Extracting c-basic pack for version 18800 - ...89% - Extracting os-core-dev pack for version 18800 - ...94% - Extracting web-server-basic pack for version 18680 - ...100% - Installing bundle(s) files... - ...100% - Calling post-update helper scripts. - Possible filedescriptor leak : 8 (socket:[30833]) - Bundle(s) installation done. - -.. note:: - - This script uses :command:`sudo` to run commands and you may be prompted to - enter your user password at any time while the script is executing. If this - occurs, enter your user password to execute the :command:`sudo` command. - - .. code-block:: console - - Password: - - You may also see an informational message about setting the - :envvar:`https_proxy` environment variable if this variable isn't set. - -Once the :command:`swupd` command is complete, the script begins processing -the requirements to install and implement the telemetrics server. Finally, -the script enables the server and provides output similar to: - -.. code-block:: console - - Collecting uwsgitop - Downloading uwsgitop-0.10.tar.gz - Requirement already satisfied: simplejson in /usr/lib/python3.6/site-packages (from uwsgitop) - Collecting argparse (from uwsgitop) - Downloading argparse-1.4.0-py2.py3-none-any.whl - Building wheels for collected packages: uwsgitop - Running setup.py bdist_wheel for uwsgitop ... done - Stored in directory: /root/.cache/pip/wheels/8a/99/e9/accc80bcaa989218da65daaae4205dc4f6288d3551655aa638 - Successfully built uwsgitop - Installing collected packages: argparse, uwsgitop - Successfully installed argparse-1.4.0 uwsgitop-0.10 - mkdir: created directory '/var/www' - mkdir: created directory '/var/www/telemetry' - Already using interpreter /usr/bin/python3 - Using base prefix '/usr' - New python executable in /var/www/telemetry/venv/bin/python3 - Also creating executable in /var/www/telemetry/venv/bin/python - Installing setuptools, pip, wheel...done. - Collecting alembic==0.9.5 (from -r /tmp/requirements.txt.KDI3uU (line 1)) - Downloading alembic-0.9.5.tar.gz (990kB) - 100% |████████████████████████████████| 993kB 2.1MB/s - Collecting click==6.7 (from -r /tmp/requirements.txt.KDI3uU (line 2)) - Downloading click-6.7-py2.py3-none-any.whl (71kB) - 100% |████████████████████████████████| 71kB 8.3MB/s - Collecting Flask==0.12.2 (from -r /tmp/requirements.txt.KDI3uU (line 3)) - Downloading Flask-0.12.2-py2.py3-none-any.whl (83kB) - 100% |████████████████████████████████| 92kB 10.2MB/s - Collecting Flask-Migrate==2.1.0 (from -r /tmp/requirements.txt.KDI3uU (line 4)) - Downloading Flask-Migrate-2.1.0.tar.gz - Collecting Flask-SQLAlchemy==2.2 (from -r /tmp/requirements.txt.KDI3uU (line 5)) - Downloading Flask_SQLAlchemy-2.2-py2.py3-none-any.whl - Collecting Flask-WTF==0.14.2 (from -r /tmp/requirements.txt.KDI3uU (line 6)) - Downloading Flask_WTF-0.14.2-py2.py3-none-any.whl - Collecting itsdangerous==0.24 (from -r /tmp/requirements.txt.KDI3uU (line 7)) - Downloading itsdangerous-0.24.tar.gz (46kB) - 100% |████████████████████████████████| 51kB 12.4MB/s - Collecting Jinja2==2.9.6 (from -r /tmp/requirements.txt.KDI3uU (line 8)) - Downloading Jinja2-2.9.6-py2.py3-none-any.whl (340kB) - 100% |████████████████████████████████| 348kB 3.5MB/s - Collecting Mako==1.0.7 (from -r /tmp/requirements.txt.KDI3uU (line 9)) - Downloading Mako-1.0.7.tar.gz (564kB) - 100% |████████████████████████████████| 573kB 1.9MB/s - Collecting MarkupSafe==1.0 (from -r /tmp/requirements.txt.KDI3uU (line 10)) - Downloading MarkupSafe-1.0.tar.gz - Collecting psycopg2==2.7.3 (from -r /tmp/requirements.txt.KDI3uU (line 11)) - Downloading psycopg2-2.7.3.tar.gz (425kB) - 100% |████████████████████████████████| 430kB 4.0MB/s - Collecting python-dateutil==2.6.1 (from -r /tmp/requirements.txt.KDI3uU (line 12)) - Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB) - 100% |████████████████████████████████| 194kB 6.8MB/s - Collecting python-editor==1.0.3 (from -r /tmp/requirements.txt.KDI3uU (line 13)) - Downloading python-editor-1.0.3.tar.gz - Collecting six==1.10.0 (from -r /tmp/requirements.txt.KDI3uU (line 14)) - Downloading six-1.10.0-py2.py3-none-any.whl - Collecting SQLAlchemy==1.1.13 (from -r /tmp/requirements.txt.KDI3uU (line 15)) - Downloading SQLAlchemy-1.1.13.tar.gz (5.2MB) - 100% |████████████████████████████████| 5.2MB 394kB/s - Collecting uWSGI==2.0.15 (from -r /tmp/requirements.txt.KDI3uU (line 16)) - Downloading uwsgi-2.0.15.tar.gz (795kB) - 100% |████████████████████████████████| 798kB 1.5MB/s - Collecting Werkzeug==0.12.2 (from -r /tmp/requirements.txt.KDI3uU (line 17)) - Downloading Werkzeug-0.12.2-py2.py3-none-any.whl (312kB) - 100% |████████████████████████████████| 317kB 2.2MB/s - Collecting WTForms==2.1 (from -r /tmp/requirements.txt.KDI3uU (line 18)) - Downloading WTForms-2.1.zip (553kB) - 100% |████████████████████████████████| 563kB 1.7MB/s - Skipping bdist_wheel for psycopg2, due to binaries being disabled for it. - Building wheels for collected packages: alembic, Flask-Migrate, itsdangerous, Mako, MarkupSafe, python-editor, SQLAlchemy, uWSGI, WTForms - Running setup.py bdist_wheel for alembic ... done - Stored in directory: /root/.cache/pip/wheels/d1/0e/b9/fb570150b350298e1d8f1ff38a400ae709580b36e43bc3ac91 - Running setup.py bdist_wheel for Flask-Migrate ... done - Stored in directory: /root/.cache/pip/wheels/3d/29/d4/66747eca8b8a28973aa639f39e96a402b3dcab335e608048dd - Running setup.py bdist_wheel for itsdangerous ... done - Stored in directory: /root/.cache/pip/wheels/fc/a8/66/24d655233c757e178d45dea2de22a04c6d92766abfb741129a - Running setup.py bdist_wheel for Mako ... done - Stored in directory: /root/.cache/pip/wheels/33/bf/8f/036f36c35e0e3c63a4685e306bce6b00b6349fec5b0947586e - Running setup.py bdist_wheel for MarkupSafe ... done - Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57 - Running setup.py bdist_wheel for python-editor ... done - Stored in directory: /root/.cache/pip/wheels/84/d6/b8/082dc3b5cd7763f17f5500a193b6b248102217cbaa3f0a24ca - Running setup.py bdist_wheel for SQLAlchemy ... done - Stored in directory: /root/.cache/pip/wheels/f0/50/ca/3cb6e78527eb05e180d19632343ee14d2e5c164da2e61fbd2d - Running setup.py bdist_wheel for uWSGI ... done - Stored in directory: /root/.cache/pip/wheels/26/d0/48/e7b0eed63b5d191e89d94e72196aafae93b2b6505a9feafdd9 - Running setup.py bdist_wheel for WTForms ... done - Stored in directory: /root/.cache/pip/wheels/36/35/f3/7452cd24daeeaa5ec5b2ea13755316abc94e4e7702de29ba94 - Successfully built alembic Flask-Migrate itsdangerous Mako MarkupSafe python-editor SQLAlchemy uWSGI WTForms - Installing collected packages: SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, click, Werkzeug, Jinja2, itsdangerous, Flask, Flask-SQLAlchemy, Flask-Migrate, WTForms, Flask-WTF, psycopg2, uWSGI - Running setup.py install for psycopg2 ... done - Successfully installed Flask-0.12.2 Flask-Migrate-2.1.0 Flask-SQLAlchemy-2.2 Flask-WTF-0.14.2 Jinja2-2.9.6 Mako-1.0.7 MarkupSafe-1.0 SQLAlchemy-1.1.13 WTForms-2.1 Werkzeug-0.12.2 alembic-0.9.5 click-6.7 itsdangerous-0.24 psycopg2-2.7.3 python-dateutil-2.6.1 python-editor-1.0.3 six-1.10.0 uWSGI-2.0.15 - mkdir: created directory '/var/log/uwsgi' - -Once all the server components have been installed you are prompted to enter -the :guilabel:`PostgreSQL` database password to change it as illustrated below: - -.. code-block:: console - - Enter password for 'postgres' user: - New password: - Retype new password: - passwd: password updated successfully - -Enter `postgres` for the current value of the password and then enter a new -password, retype it to verify the new password and the :guilabel:`PostgreSQL` -database password will be updated. - -The script finalizes installation and finishes. - -.. code-block:: console - - Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /usr/lib/systemd/system/postgresql.service. - Cloning into 'telemetrics-backend'... - remote: Counting objects: 344, done. - remote: Compressing objects: 100% (53/53), done. - remote: Total 344 (delta 30), reused 50 (delta 20), pack-reused 268 - Receiving objects: 100% (344/344), 130.20 KiB | 1.40 MiB/s, done. - Resolving deltas: 100% (177/177), done. - '/tmp/telemetrics-backend/scripts/collector_uwsgi.ini' -> '/tmp/telemetrics-backend/collector/collector_uwsgi.ini' - '/tmp/telemetrics-backend/scripts/telemetryui_uwsgi.ini' -> '/tmp/telemetrics-backend/telemetryui/telemetryui_uwsgi.ini' - mkdir: created directory '/var/www/telemetry/collector/uwsgi-spool' - mkdir: created directory '/var/www/telemetry/telemetryui/uwsgi-spool' - '/tmp/telemetrics-backend/scripts/uwsgi.service' -> '/etc/systemd/system/uwsgi.service' - mkdir: created directory '/etc/nginx' - mkdir: created directory '/etc/nginx/conf.d' - '/usr/share/nginx/conf/nginx.conf.example' -> '/etc/nginx/nginx.conf' - Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service. - mkdir: created directory '/etc/uwsgi' - mkdir: created directory '/etc/uwsgi/vassals' - Created symlink /etc/systemd/system/multi-user.target.wants/uwsgi.service → /etc/systemd/system/uwsgi.service. - ALTER ROLE - sed: can't read /tmp/telemetrics-backend/collector/config.py: No such file or directory - cp: cannot stat '/tmp/telemetrics-backend/collector/config.py': No such file or directory - sed: can't read /tmp/telemetrics-backend/telemetryui/config.py: No such file or directory - cp: cannot stat '/tmp/telemetrics-backend/telemetryui/config.py': No such file or directory - Already using interpreter /usr/bin/python3 - Using base prefix '/usr' - New python executable in /var/www/telemetry/venv/bin/python3 - Not overwriting existing python script /var/www/telemetry/venv/bin/python (you must use /var/www/telemetry/venv/bin/python3) - Installing setuptools, pip, wheel...done. - INFO [alembic.runtime.migration] Context impl PostgresqlImpl. - INFO [alembic.runtime.migration] Will assume transactional DDL. - INFO [alembic.runtime.migration] Running upgrade -> 3230c615d6e0, empty message - INFO [alembic.runtime.migration] Running upgrade 3230c615d6e0 -> 466cf2f35d67, empty message - - Install complete (installation folder: /var/www/telemetry) - -Once the installation is complete you can use your web browser and view the -new server by opening the web browser on the system you installed the backend -server onto and type in ``localhost`` in the address bar. You should see a -web page similar to the one shown in figure 1: - -.. figure:: figures/telemetry-backend-1.png - :alt: Telemetry UI - - Figure 1: :guilabel:`Telemetry UI` - - -Redirect telemetry records -************************** - -Telemetry records from your system are sent to the server location defined in -the :file:`/usr/share/defaults/telemetrics/telemetrics.conf` configuration -file. You can customize this by copying this file to -:file:`/etc/telemetrics/telemetrics.conf` and changing the ``server=`` -setting to your new server location. - -#. Create the :file:`/etc/telemetrics` directory and make it your current - working directory. - - .. code-block:: console - - sudo mkdir -p /etc/telemetrics - cd /etc/telemetrics - - -#. Copy the default :file:`telemetrics.conf` file to the new - :file:`/etc/telemetrics` directory. - - .. code-block:: console - - sudo cp /usr/share/defaults/telemetrics/telemetrics.conf . - -#. Edit the new :file:`/etc/telemetrics/telemetrics.conf` file with your - editor using the :command:`sudo` directive and change the - :guilabel:`server=` setting to ``http://localhost/v2/collector`` and save - this change in the new file. - - .. code-block:: console - - server=http://localhost/v2/collector - - You can also use the fully qualified domain name for your server instead of - :guilabel:`localhost`. - -#. Restart the telemetry daemons to reload the configuration file. - - .. code-block:: console - - telemctl restart - -Test the new telemetry backend server -************************************* - -|CL| includes a telemetry test probe called :command:`hprobe` that will send a -``hello`` record to the telemetry backend server. To test that the telemetry -records are now going to your new destination, run the :command:`hprobe` -command to send a ``hello`` record to the server as follows: - -.. code-block:: console - - hprobe - -The record should show up on your new server console as shown in figure 2: - -.. figure:: figures/telemetry-backend-2.png - :alt: Telemetry UI - - Figure 2: :guilabel:`Telemetry UI` - -Congratulations! You've just set up and enabled a new telemetrics backend -server, redirected the records from your local machine to this new server and -tested it using the :command:`hprobe` command to send a ``hello`` record to -it. - -Related topics -************** - -* `Telemetry feature description`_ -* :ref:`Telemetry architecture` -* :ref:`telemetry-enable` -* https://github.com/clearlinux/telemetrics-client -* https://github.com/clearlinux/telemetrics-backend - -.. _`telemetrics backend`: - https://github.com/clearlinux/telemetrics-backend - -.. _`Intel privacy policies`: - https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html - -.. _`Telemetry feature description`: - https://clearlinux.org/features/telemetry diff --git a/source/clear-linux/tutorials/telemetry-e2e.rst b/source/clear-linux/tutorials/telemetry-e2e.rst deleted file mode 100644 index 7cf8e38f..00000000 --- a/source/clear-linux/tutorials/telemetry-e2e.rst +++ /dev/null @@ -1,617 +0,0 @@ -.. _telemetry-e2e: - -Develop with telemetry -###################### - -This tutorial shows you how to set up a telemetry backend server to -manage your records and how to instrument your application with the telemetry -API. - -|CL-ATTR| includes a telemetry and analytics solution (also known as -telemetrics) as part of the OS, which records events of interest and reports -them back to the development team using the telemetrics client daemons. - -The |CL| telemetry client can be enabled or disabled and records can be -redirected to a desired location. More detailed information about using and -configuring the telemetrics client is found in -the :ref:`telemetrics` guide. - - -.. contents:: :local: - :depth: 1 - - -Prerequisites -============= - - -For this tutorial, you can use an existing |CL| system, or you can start with a clean installation of |CL| on a new system. - -New Installation -**************** - -To setup a new system for your telemetry backend server, follow the :ref:`bare-metal-install` getting started guide and: - -#. Choose to install |CL|. -#. Join the :guilabel:`Stability Enhancement Program` during the installation process to enable the telemetrics client components. -#. Select the manual installation method with the following settings: - * Set the hostname to :guilabel:`clr-telem-server` - * Create an administrative user named :guilabel:`clear` and add this user to sudoers :ref:`enable-user-space` - * Choose the :file:`dev-utils`, :file:`network-basic`, and :file:`openssh-server` bundles from the bundle list - - -.. note:: - - Bundles can also be added to your system after this install process completed. The bundles listed here are a minimal set needed to complete the setup of the telemetry backend server and applications. - -Existing System -*************** - -If you are using an existing |CL| system, make sure you have installed the telemetry and dev-utils bundles. Use the :command:`swupd` utility with the `bundle-list` option and check for "telemetrics" in the list: - -.. code-block:: bash - - sudo swupd bundle-list - -If you need to install the bundles, use :command:`swupd` to do so. - -.. code-block:: bash - - sudo swupd bundle-add telemetrics dev-utils - -More information about enabling and configuring the telemetry client can be found at :ref:`telemetry-enable`. - -You will need to run some of the commands in this tutorial with root privileges. You can create a new user or add your user to the sudoers list :ref:`enable-user-space`. - -Setting up the telemetry backend server -======================================= -We'll be using the :file:`deploy.sh` file from the `clearlinux/telemetrics-backend`_ Git repository to install required dependencies for the web server applications. The script also configures nginx and uwsgi, deploys snapshots of the applications, and starts all required services. - -Clone the clearlinux/telemetrics-backend Git repository -******************************************************* - -With all prerequisite software bundles installed, log in with your administrative user, and from your :file:`$HOME` directory, run :command:`git` to clone the :guilabel:`telemetrics-backend` repository into the :file:`$HOME/telemetrics-backend` directory: - -.. code-block:: bash - - git clone https://github.com/clearlinux/telemetrics-backend - -.. note:: - - You may need to set up the :envvar:`https_proxy` environment variable if you have issues reaching github.com. - -Run the deploy.sh script to install the backend server -****************************************************** - -Change your current working directory to :file:`telemetrics-backend/scripts`. - -.. code-block:: bash - - cd telemetrics-backend/scripts - -Run the :command:`./deploy.sh -h` to see the list of options for the :command:`deploy.sh` script: - -.. code-block:: console - - ./deploy.sh -h - Deploy snapshot of the telemetrics-backend - - -a Perform specified action (deploy, install, migrate, resetdb, - restart, uninstall; default: deploy) - -d Distro to deploy to (ubuntu, centos or clr; default: ubuntu) - -h Print these options - -H Set domain for deployment (only accepted value is "localhost" for - now) - -r Set repo location to deploy from - (default: https://github.com/clearlinux/telemetrics-backend) - -s Set source location (default: "master" branch from git repo) - -t Set source type (tarball, or git; default: git) - -u Perform complete uninstallation - -The :command:`deploy.sh` is a bash shell script that allows you to perform the following actions: - -* *deploy* - install a complete instance of the telemetrics backend server and all required components. This is the default action if no *-a* argument is given on the command line. -* *install* - installs and enables all required components for the telemetrics backend server. -* *migrate* - migrate database to new schema. -* *resetdb* - reset the database. -* *restart* - restart the nginx and uWSGI services. -* *uninstall* - uninstall all packages. - -.. note:: - - The *uninstall* option does not perform any actions if the distro is set to |CL| and will only uninstall packages if the distro is Ubuntu - -Next, we will install the telemetrics backend server with the following options: - -* *-a install* to perform an install -* *-d clr* to install to a |CL| distro -* *-H localhost* to set the domain to localhost - -We do not need to set the following options since the values are set to the correct values we want by default: - -* *-r https://github.com/clearlinux/telemetrics-backend* sets the repo location for :command:`git` to clone from. -* *-s master* to set the location, or branch. -* *-t git* to set the source type to git. - -.. caution:: - - The :file:`deploy.sh` shell script has minimal error checking and makes several changes to your system. Be sure that the options you define on the cmdline are correct before proceeding. - -To begin the installation with the options defined: - -Run the shell script from the :file:`$HOME/telemetrics-backend/scripts` directory: - -.. code-block:: bash - - ./deploy.sh -H localhost -a install -d clr - -The script will start and list all the defined options and prompt you for the :guilabel:`PostgreSQL` database password as shown below: - -.. code-block:: console - - Options: - host: localhost - distro: clr - action: install - repo: https://github.com/clearlinux/telemetrics-backend - source: master - type: git - DB password: (default: postgres): - -For the :guilabel:`DB password:`, press the :kbd:`Enter` key to accept the default password `postgres`. - -The :command:`swupd` begins installing the required software bundles to set up the telemetrics backend server. The output will look similar to the following: - -.. code-block:: console - - swupd-client bundle adder 3.12.7 - Copyright (C) 2012-2017 Intel Corporation - - Downloading packs... - - Extracting application-server pack for version 18740 - ...5% - Extracting database-basic-dev pack for version 18670 - ...10% - Extracting database-basic pack for version 18670 - ...15% - ... - - Extracting c-basic pack for version 18800 - ...89% - Extracting os-core-dev pack for version 18800 - ...94% - Extracting web-server-basic pack for version 18680 - ...100% - Installing bundle(s) files... - ...100% - Calling post-update helper scripts. - Possible filedescriptor leak : 8 (socket:[30833]) - Bundle(s) installation done. - -.. note:: - - This script uses :command:`sudo` to run commands and you may be prompted to enter your user password at any time while the script is executing. If this occurs, enter your user password to execute the :command:`sudo` command. - - .. code-block:: console - - Password: - - You may also see an informational message about setting the :envvar:` - https_proxy` environment variable if this variable isn't set. - -Once the :command:`swupd` command is complete, the script begins processing -the requirements to install and implement the telemetrics server. Finally, -the script enables the server and provides output that finishes with -something similar to: - -.. code-block:: console - - . - . - Successfully built alembic Flask-Migrate itsdangerous Mako MarkupSafe python-editor SQLAlchemy uWSGI WTForms - Installing collected packages: SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, click, Werkzeug, Jinja2, itsdangerous, Flask, Flask-SQLAlchemy, Flask-Migrate, WTForms, Flask-WTF, psycopg2, uWSGI - Running setup.py install for psycopg2 ... done - Successfully installed Flask-0.12.2 Flask-Migrate-2.1.0 Flask-SQLAlchemy-2.2 Flask-WTF-0.14.2 Jinja2-2.9.6 Mako-1.0.7 MarkupSafe-1.0 SQLAlchemy-1.1.13 WTForms-2.1 Werkzeug-0.12.2 alembic-0.9.5 click-6.7 itsdangerous-0.24 psycopg2-2.7.3 python-dateutil-2.6.1 python-editor-1.0.3 six-1.10.0 uWSGI-2.0.15 - -Once all the server components have been installed you are prompted to enter the :guilabel:`PostgreSQL` database password to change it as illustrated below: - -.. code-block:: console - - Enter password for 'postgres' user: - New password: - Retype new password: - passwd: password updated successfully - -Enter `postgres` for the current value of the password and then enter a new -password, retype it to verify the new password and the :guilabel:`PostgreSQL` -database password will be updated. - -The script finalizes installation and finishes. - -.. code-block:: console - - Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /usr/lib/systemd/system/postgresql.service. - Cloning into 'telemetrics-backend'... - remote: Counting objects: 344, done. - remote: Compressing objects: 100% (53/53), done. - remote: Total 344 (delta 30), reused 50 (delta 20), pack-reused 268 - Receiving objects: 100% (344/344), 130.20 KiB | 1.40 MiB/s, done. - Resolving deltas: 100% (177/177), done. - ... - Already using interpreter /usr/bin/python3 - Using base prefix '/usr' - New python executable in /var/www/telemetry/venv/bin/python3 - Not overwriting existing python script /var/www/telemetry/venv/bin/python (you must use /var/www/telemetry/venv/bin/python3) - Installing setuptools, pip, wheel...done. - INFO [alembic.runtime.migration] Context impl PostgresqlImpl. - INFO [alembic.runtime.migration] Will assume transactional DDL. - INFO [alembic.runtime.migration] Running upgrade -> 3230c615d6e0, empty message - INFO [alembic.runtime.migration] Running upgrade 3230c615d6e0 -> 466cf2f35d67, empty message - - Install complete (installation folder: /var/www/telemetry) - -Once the installation is complete you can use your web browser and view the new server by opening the web browser on your system and type in -``localhost`` in the address bar. - -You should see a web page similar to the one shown in figure 1: - - .. figure:: telemetry-backend/figures/telemetry-backend-1.png - :alt: Telemetry UI - :scale: 50% - - Figure 1: :guilabel:`Telemetry UI` - -Redirect telemetry records -************************** - -Telemetry records generated by the telemetrics clients are sent to the -server location defined in the :file:`/usr/share/defaults/telemetrics/ -telemetrics.conf` configuration file. You can customize this setting by -copying this file to :file:`/etc/telemetrics/telemetrics.conf` and changing -the ``server=`` setting to your new server location. - -#. Create the :file:`/etc/telemetrics` directory and make it your current - working directory. - - .. code-block:: bash - - sudo mkdir -p /etc/telemetrics - cd /etc/telemetrics - -#. Copy the default :file:`telemetrics.conf` file to the new - :file:`/etc/telemetrics` directory. - - .. code-block:: bash - - sudo cp /usr/share/defaults/telemetrics/telemetrics.conf - -#. Edit the new :file:`/etc/telemetrics/telemetrics.conf` file with your - editor using the :command:`sudo` directive and change the - :guilabel:`server=` setting to ``http://localhost/v2/collector`` and save - this change in the new file. - - .. code-block:: console - - server=http://localhost/v2/collector - - You can also use the fully qualified domain name for your server instead of :guilabel:`localhost`. - -#. Restart the telemetry daemons to reload the configuration file. - - .. code-block:: bash - - telemctl restart - -Test the new telemetry backend server -************************************* - -|CL| includes a telemetry test probe called :command:`hprobe` that will send -a ``hello`` record to the telemetry backend server. To test that the -telemetry records are now going to your new destination, run the :command:` -hprobe` command to send a ``hello`` record to the server as follows: - -.. code-block:: bash - - hprobe - -The record should show up on your new server console as shown in figure 2: - - .. figure:: telemetry-backend/figures/telemetry-backend-2.png - :alt: Telemetry UI - :scale: 50% - - Figure 2: :guilabel:`Telemetry UI` - -You have now set up the |CL| telemetry backend server, and redirected records from your client to your server. - -Creating custom telemetry events -================================ -For the following steps, we'll be sending records to the backend server we've just set up. If you prefer to keep records locally and not send them to a server, follow the :ref:`telemetrics` guide and enable :record_retention_enabled: in your :file:`etc/telemetrics/telemetrics.conf` to keep the records locally. - -There are two ways to create custom telemetry events: using :command:`telem-record-gen` and using the telemetry API in your applications. - -Using telem-record-gen -********************** - -Enabling telemetry during installation gives us everything we need to create custom telemetry events, even from C programs, because the telemetry bundle provides a simple pipe-based :abbr:`CLI (Commandline Interface)` program named :file:`telem-record-gen` that can be called trivially: - - -.. code-block:: bash - - ~ $ telem-record-gen --help - -.. code-block:: console - - Usage: - telem-record-gen [OPTIONS] - create and send a custom telemetry record - - Help Options: - -h, --help Show help options - - Application Options: - -f, --config-file Path to configuration file (not implemented yet) - -V, --version Print the program version - -s, --severity Severity level (1-4) - (default 1) - -c, --class Classification level_1/level_2/level_3 - -p, --payload Record body (max size = 8k) - -P, --payload-file File to read payload from - -R, --record-version Version number for format of payload (default 1) - -e, --event-id Event id to use in the record - - -.. note:: - - The C library (:file:`libtelemetry.so - man 3 telemetry`) uses the same API parameters and will yield the same effect as :command:`telem-record-gen`. - -Let's try generating a simple heartbeat event with -:command:`telem-record-gen`, similar to the hprobe heartbeat probe that |CL| -includes by default. - -.. code-block:: bash - - ~ $ telem-record-gen -c org.clearlinux/hello/world -p "hello there" - -We won't see anything happen on the console, but we can track existing and -previous telemetry events with :command:`telemctl`: - -.. code-block:: bash - - ~$ sudo telemctl journal -V -c org.clearlinux/hello/world -i - -.. code-block:: console - - org.clearlinux/hello/world Tue 2018-11-06 23:00:48 UTC 72e55923fd21c75142c24dcfe0ae0a79 143f2580dcf80267f8f1dfe448f3c975 75f547ff-e55b-44b1-9333-1106098bd448 - hello there - -Using the telemetry API in your C application -********************************************* - -.. note:: - - More details about the :ref:`the telemetry API ` are - available in the telemetry guide. - -Confirm that the telemetrics header file is located on the system at -:file:`usr/include/telemetry.h` The `latest version`_ of the file can also -be found on github for reference, but installing the `telemetry` bundle will -install the header file that matches your |CL| version. - -You will need to include the following headers in your code to use the API: - -:: - - #define _GNU_SOURCE - #include - #include - #include - #include - - -Use the following code to create the variables we need to hold the data for the record we will be creating: - -:: - - uint32_t severity = 1; - uint32_t payload_version = 1; - char classification[30] = "org.clearlinux/hello/world"; - struct telem_ref *tm_handle = NULL; - char *payload; - int ret = 0; - - - -Severity: - | Type: uint32_t - | Value: Severity field value. Accepted values are in the range 1-4, with 1 being the lowest severity, and 4 being the highest severity. Values provided outside of this range are clamped to 1 or 4. [low, med, high, crit] - -Payload_version: - | Type: uint32_t - | Value: Payload format version. The only supported value right now is 1, which indicates that the payload is a freely-formatted (unstructured) string. Values greater than 1 are reserved for future use. - -Classification: - | Type: char array - | Value: It should have the form, DOMAIN/PROBENAME/REST: DOMAIN is the reverse domain to use as a namespace for the probe (e.g. org.clearlinux); PROBENAME is the name of the probe; and REST is an arbitrary value that the probe should use to classify the record. The maximum length for the classification string is 122 bytes. Each sub-category may be no longer than 40 bytes long. Two / delimiters are required. - -Tm_handle: - | Type: Telem_ref struct pointer - | Value: Struct pointer declared by the caller, The struct is initialized if the function returns success. - -Payload: - | Type: char pointer - | Value: The payload to set - - - -For this example, we'll set the payload to “hello” by using ``asprintf()`` - -:: - - if (asprintf(&payload, "hello\n") < 0) { - exit(EXIT_FAILURE); - } - - - -The functions ``asprintf()`` and ``vasprintf()`` are analogs of ``sprintf(3)`` and ``vsprintf(3)``, except that they allocate a string large enough to hold the output including the terminating null byte ('\0'), and return a pointer to it via the first argument. This pointer should be passed to ``free(3)`` to release the allocated storage when it is no longer needed. - - -Create the new telemetry record -******************************* - -The function ``tm_create_record()`` initializes a telemetry record and sets the severity and classification of that record, as well as the payload version number. The memory needed to store the telemetry record is allocated and should be freed with ``tm_free_record()`` when no longer needed. - -:: - - if ((ret = tm_create_record(&tm_handle, severity, classification, payload_version)) < 0) { - printf("Failed to create record: %s\n", strerror(-ret)); - ret = 1; - goto fail; - } - - -Set the payload field of a telemetrics record -********************************************* - -The function ``tm_set_payload()`` attaches the provided telemetry record data to the telemetry record. The current maximum payload size is 8192b. - -:: - - if ((ret = tm_set_payload(tm_handle, payload)) < 0) { - printf("Failed to set record payload: %s\n", strerror(-ret)); - ret = 1; - goto fail; - } - free(payload); - -The ``free()`` function frees the memory space pointed to by ptr, which must have been returned by a previous call to ``malloc()``, ``calloc()``, or ``realloc()``. Otherwise, or if ``free(ptr)`` has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed. - -Send a record to the telemetrics daemon -*************************************** - -The function ``tm_send_record()`` delivers the record to the local -``telemprobd(1)`` service. Since the telemetry record was allocated by the -program it should be freed with ``tm_free_record()`` when it is no longer -needed. - -:: - - if ((ret = tm_send_record(tm_handle)) < 0) { - printf("Failed to send record to daemon: %s\n", strerror(-ret)); - ret = 1; - goto fail; - } else { - printf("Successfully sent record to daemon.\n"); - ret = 0; - } - fail: - tm_free_record(tm_handle); - tm_handle = NULL; - - return ret; - - -Full sample application with compiling flags -============================================ - -Create a new file test.c add the following code. - -:: - - #define _GNU_SOURCE - #include - #include - #include - #include - - int main(int argc, char **argv) - { - uint32_t severity = 1; - uint32_t payload_version = 1; - char classification[30] = "org.clearlinux/hello/world"; - struct telem_ref *tm_handle = NULL; - char *payload; - - int ret = 0; - - if (asprintf(&payload, "hello\n") < 0) { - exit(EXIT_FAILURE); - } - - if ((ret = tm_create_record(&tm_handle, severity, classification, - payload_version)) < 0) { - printf("Failed to create record: %s\n", strerror(-ret)); - ret = 1; - goto fail; - } - - if ((ret = tm_set_payload(tm_handle, payload)) < 0) { - printf("Failed to set record payload: %s\n", strerror(-ret)); - ret = 1; - goto fail; - } - - free(payload); - - if ((ret = tm_send_record(tm_handle)) < 0) { - printf("Failed to send record to daemon: %s\n", strerror(-ret)); - ret = 1; - goto fail; - } else { - printf("Successfully sent record to daemon.\n"); - ret = 0; - } - fail: - tm_free_record(tm_handle); - tm_handle = NULL; - - return ret; - } - - - -Compile with the gcc compiler, using this command: - -.. code-block:: bash - - gcc test.c -ltelemetry -o test_telem - - -Test to ensure the program is working: - -.. code-block:: bash - - ./test_telem - Successfully sent record to daemon. - -Verify record was received -***************************** - -To verify that the heartbeat message was received by the telemetry backend -server you can check the telemetry client journal, and specify the -classification as org.clearlinux/hello/world - -.. code-block:: bash - - sudo telemctl journal -V -c org.clearlinux/hello/world -i - -.. code-block:: console - - Classification Time stamp Record ID Event ID Boot ID - org.clearlinux/hello/world Tue 2018-11-06 22:58:25 UTC b11db07c58c90d8f496ff963df6c43de 24699c2d60c12d154692875b599ca957 75f547ff-e55b-44b1-9333-1106098bd448 - hello - Total records: 1 - - - -A full example of the `heartbeat probe`_ in C is documented in the source code. For more information about telemetrics in |CL| refer to the -:ref:`telemetrics` guide. - -You can also look for the record on the telemetry backend server. - -.. _latest version: https://github.com/clearlinux/telemetrics-client/tree/master/src - -.. _heartbeat probe: https://github.com/clearlinux/telemetrics-client/tree/master/src/probes/hello.c - - -.. _clearlinux/telemetrics-backend: https://github.com/clearlinux/telemetrics-backend diff --git a/source/clear-linux/tutorials/tutorials.rst b/source/clear-linux/tutorials/tutorials.rst index a7069048..1e516588 100644 --- a/source/clear-linux/tutorials/tutorials.rst +++ b/source/clear-linux/tutorials/tutorials.rst @@ -14,6 +14,7 @@ Explore our tutorials to discover what you can do with |CL|! .. toctree:: :maxdepth: 1 + :glob: wordpress/wordpress flatpak/flatpak @@ -24,12 +25,12 @@ Explore our tutorials to discover what you can do with |CL|! hadoop fmv aws-web/aws-web - telemetry-e2e - telemetry-backend/telemetry-backend smb/smb spark kata kata_migration - kubernetes + kubernetes/kubernetes* greengrass dlrs + yubikey-u2f + nvidia diff --git a/source/clear-linux/tutorials/yubikey-u2f.rst b/source/clear-linux/tutorials/yubikey-u2f.rst new file mode 100644 index 00000000..e75884e2 --- /dev/null +++ b/source/clear-linux/tutorials/yubikey-u2f.rst @@ -0,0 +1,126 @@ +.. _yubikey-u2f: + +Enable YubiKey U2F Support +########################## + +YubiKey\* is a USB security token manufactured by `Yubico`_. Depending on the +model, a YubiKey can support different authentication protocols including: +One-Time Password (OTP), Smart card, FIDO2, and Universal 2nd Factor (U2F). + +These instructions will go over configuring a YubiKey for U2F authentication +through a web browser on a |CL-ATTR| system. + +A list of `websites +accepting U2F authentication with the YubiKey`_ is available on the Yubico +website. See the Yubico website to learn more about the Yubikey: +https://www.yubico.com/getstarted/meet-the-yubikey/ + + + +.. contents:: :local: + :depth: 1 + + + +Prerequisites +************* + +This tutorial assumes you have: + +#. |CL| installed and running. + +#. Mozilla Firefox installed on |CL|. + +#. A YubiKey. + + + +Enable Linux udev rules for YubiKey +*********************************** + +:command:`udev` is the Linux device manager which handles events when USB +devices are added and removed. Custom rules needs to be created to properly +identify the YubiKey and provide applications access. + +These instructions are derived from: `Yubico support article Using Your U2F +YubiKey with Linux +`_ + + +#. Create the udev rules folder under :file:`/etc` + + .. code:: bash + + sudo mkdir -p /etc/udev/rules.d/ + + +#. Download the u2f rules from the Yubico GitHub: + + .. code:: bash + + curl -O https://raw.githubusercontent.com/Yubico/libu2f-host/master/70-u2f.rules + + +#. Move the downloaded :file:`70-u2f.rules` file into the :file:`/etc/udev` + folder + + .. code:: bash + + sudo mv 70-u2f.rules /etc/udev/rules.d/ + + +#. The udev rules should automatically be reloaded. However, they can be + manually reloaded or reboot the system: + + .. code:: bash + + sudo udevadm control --reload-rules && udevadm trigger + + +#. Plugin and validate the YubiKey appears as a USB device: + + .. code:: bash + + lsusb + + + +Enable U2F in Mozilla Firefox +***************************** + +Firefox comes with U2F web authentication support disabled by default. U2F +needs to be enabled in the advanced settings. + +These instructions are derived from: `Yubico support article Enabling U2F +support in Mozilla Firefox +`_ + + +#. Launch Mozilla Firefox + +#. In the URL bar, type :command:`about:config` to access the advanced + settings. + + .. code:: bash + + about:config + +#. Click the *I accept the risk!* button to continue to the advanced settings + +#. Search for the :command:`security.webauth.u2f` and double-click it + so *Value* becomes **true**. + + +Your YubiKey is now usable on |CL| with Mozilla Firefox with websites that +support U2F authentication. + + +Related topics +************** + +- |CL| :ref:`security` + + +.. _`Yubico`: https://www.yubico.com/ + +.. _`websites accepting U2F authentication with the YubiKey`: https://www.yubico.com/works-with-yubikey/catalog/#protocol=universal-2nd-factor-(u2f)&usecase=all&key=all diff --git a/source/scripts/_python/bundle_lister.py b/source/scripts/_python/bundle_lister.py index 8440ab61..a9360e19 100644 --- a/source/scripts/_python/bundle_lister.py +++ b/source/scripts/_python/bundle_lister.py @@ -6,7 +6,7 @@ import jinja2 from jinja2 import Environment, FileSystemLoader, Template import git from operator import itemgetter -from datetime import datetime +from datetime import datetime GITHUB_BASE = "https://github.com/clearlinux/clr-bundles/tree/master/bundles/" PUNDLES = "https://github.com/clearlinux/clr-bundles/blob/master/packages" @@ -15,16 +15,14 @@ PATTERN1 = re.compile(r"#\s?\[TITLE]:\w?(.*)") PATTERN2 = re.compile(r"#\s?\[DESCRIPTION]:\w?(.*)") PATTERN3 = re.compile(r"\(([^()]*|include)\)", re.MULTILINE) PATTERN4 = re.compile(r"^((?:(?!#)\w+[^-\s][-])\w+|\w+[^\s-])", re.MULTILINE) -# ALT PATTERN4 = re.compile(r"^((?:(?!#)(\w+[^-\s])[-]\w+.)[^\s]{1,}[^\s]|\w+[^\s-])", re.MULTILINE) - -PATTERN5 = re.compile(r"^(?!=a)\w.+\s[#]\s(\w+.*)?", re.MULTILINE) -# Previous version: PATTERN5 = re.compile(r"^[^#].*(?<=\s\-\s)(\w+.*)?", re.MULTILINE) +PATTERN5 = re.compile(r"^\w.+\s[#]\s(\w?.*)?", re.MULTILINE) def extractor(lines): bundle_title = "title" data_desc = "description" url = "url" include_list = [] + include_unique = [] for i in lines: title = PATTERN1.match(i) @@ -37,26 +35,26 @@ def extractor(lines): data_desc = desc.groups(0)[0].strip() if url: url = os.path.join(GITHUB_BASE, bundle_title) - if includes: include_text = includes[0].strip("()") include_list.append(include_text) - return {"title": bundle_title, "data_desc": data_desc, "include_list": include_list, "url": url} + include_unique = set(include_list) + return {"title": bundle_title, "data_desc": data_desc, "include_list": include_unique, "url": url} def pundler(): with io.open("./cloned_repo/clr-bundles/packages") as file_obj: lines = file_obj.readlines() pundle_title = "pundle_title" pundle_desc = "pundle_desc" - purl = "purl" + purl = "purl" pundle_list = [] pun_desc = [] pundle_master = [] - + for i in lines: pundle = PATTERN4.findall(i) pundle_plus = PATTERN5.findall(i) - + if pundle: pundle_title = pundle[0] pundle_list.append(pundle_title) @@ -65,7 +63,7 @@ def pundler(): pundle_desc = pundle_plus[0].strip("[]") pun_desc.append(pundle_desc) - for pun, desc in zip(pundle_list, pun_desc): + for pun, desc in zip(pundle_list, pun_desc): pundle_master.append({"title": pun, "pun_desc": desc, "purl": PUNDLES}) return pundle_master @@ -82,7 +80,7 @@ def bundler(): data.append(extractor(lines)) pundle_master = pundler() - data = data + pundle_master + data = data + pundle_master filtered = list(filter(lambda x: x.get('title'), data)) sortedData = sorted(filtered, key=lambda x:x['title'].lower()) #ALT sortedData2 = sorted(sortedData, key=itemgetter('title')) @@ -93,6 +91,6 @@ def bundler(): output = template.render(data=sortedData, now=datetime.utcnow()) with io.open('bundles.html.txt', 'w') as file: - file.write(output) - + file.write(output) + bundler() diff --git a/source/scripts/_python/template.html b/source/scripts/_python/template.html index 416d4e73..7e8d581d 100644 --- a/source/scripts/_python/template.html +++ b/source/scripts/_python/template.html @@ -25,8 +25,8 @@ + - {% for d in data %} {% if d.url %} diff --git a/source/scripts/_python/utilities/code-blocks.py b/source/scripts/_python/utilities/code-blocks.py new file mode 100644 index 00000000..2c805cc7 --- /dev/null +++ b/source/scripts/_python/utilities/code-blocks.py @@ -0,0 +1,40 @@ +''' +Usage: python code-blocks.py path/to/file.rst + +Default output is to stdout + +Parses all code found in blocks listed in blockTypes out of rst file for instruction testing. +''' +import sys + +blockTypes = ["bash","console"] + +def code_blocks(filename): + with open(filename, 'r') as fd: + c_indent = '' + in_section = [] + for line in fd: + blockFound = False + indent = line[:len(line) - len(line.lstrip())] + for blockType in blockTypes: + if 'code-block:: ' + blockType in line: + blockFound = True + if blockFound: + in_section = [line.strip()] + c_indent = '' + blockFound = False + elif in_section: + if not c_indent and line.strip(): + c_indent = indent + if not (len(indent) >= len(c_indent)) and line.strip(): + yield in_section[2:] + in_section = [] + else: + in_section.append(line[len(c_indent):].rstrip()) + +def main(): + for code_block in code_blocks(sys.argv[1]): + print('\n'.join(code_block) + '\n') + +if __name__ == '__main__': + main() diff --git a/source/substitutions.txt b/source/substitutions.txt index 24330b4c..6790a2c3 100644 --- a/source/substitutions.txt +++ b/source/substitutions.txt @@ -7,4 +7,5 @@ .. |CC| replace:: Clear Containers .. |CAUTION-BACKUP-USB| replace:: - Burning an image formats the USB drive, thus destroying all existing content. Backup your data before proceeding. + Burning an image formats the USB drive, and will destroy all pre-existing + content. Back up your data before proceeding.