diff --git a/source/clear-linux/concepts/restart.rst b/source/clear-linux/concepts/restart.rst new file mode 100644 index 00000000..2b3e0fb7 --- /dev/null +++ b/source/clear-linux/concepts/restart.rst @@ -0,0 +1,184 @@ +.. _cl-restart: + +Restart system services after an OS update +########################################## + +The software life cycle describes how software is created, developed, and +deployed, and includes how to replace or update software. A good OS +provides tools for the entire software life cycle. These tools must include +ways to remove software components properly when replaced with something else. + +Most of the work on software update code in |CL| was focused on adding new +software to the system. We recommended that users reboot their system once in +a while, but we did not provide any tools to restart services easily, until +now. + +User challenges +*************** + +It is difficult to determine which services to restart. You can either +evaluate each system and reboot manually, or figure out which services to +restart based on documentation like the |CL| release notes. Since neither +option solves the issue completely, the |CL| team created a solution. + +Over the years, several OSes approached the problem and created partial +solutions such as the following: + +* Automatically restart services during an upgrade. +* Evaluate services using these steps: + + * Mark updates requiring a reboot, such as kernel updates. + * Inform the user of those updates. + * Ask the user to restart the OS. + +Both solutions are acceptable for many OSes. However, |CL| updates software +automatically and users do not see notices from the updater unless they review +the journal. |CL| requires a completely different solution, with the following +requirements: + +* Eliminate the guesswork about what to restart and under what circumstances. +* Cannot restart everything. Many service daemons do not support an automatic + background restart. +* Fit into the |CL| architectural perspective: be small, quick, and lean. + +clr-service-restart functionality +********************************* + +Typical reasons to restart a service daemon include: + +* A new version replaces the executable file itself. +* A new version replaces a library component used by a service daemon. + +Our method restarts daemons when it is really needed, especially +in the case of security updates. The tool restarts daemons by reading +various files in the :file:`procfs` filesystem provided by the kernel. + +The second part of the problem is to determine whether or not running +processes are part of a system service. The tool focuses on system services +because most system services are background tasks with no direct user +interaction. Fortunately, :command:`systemd` provides a simple way to: + +* Determine which active tasks are within the system domain. +* Determine which tasks map to which service. + +We combined both solutions into a low-overhead tool that shows which system +daemons require a restart, as shown below: + +Figure 1: Invoke :command:`clr-service-restart`. + +.. code-block:: bash + + sudo clr-service-restart -a -n + +.. code-block:: console + + upower.service: needs a restart (a library dependency was updated) + /usr/bin/systemctl --no-ask-password try-restart upower.service + NetworkManager.service: needs a restart (a library dependency was + updated) + /usr/bin/systemctl --no-ask-password try-restart NetworkManager.service + .... + +:command:`clr-service-restart` implements a whitelist to identify which +daemons can be restarted. The system administrator can customize the default +|CL| OS whitelist using :option:`allow` or :option:`disallow` options for +restarting system services. When a software update occurs, +:command:`clr-service-restart` consults the whitelist to see if a service +daemon is allowed to be restarted or not. See the options section for +details. + + +Options for clr-service-restart +******************************* + +The :option:`allow` option identifies a daemon to restart after an OS software +update. The :command:`clr-service-restart` daemon creates a symlink in +:file:`/etc/clr-service-restart` as a record. The example below tells +:command:`clr-service-restart` to restart the :option:`tallow` daemon after an +OS software update. + +.. code-block:: bash + + sudo clr-service-restart allow tallow.service + +The :option:`disallow` option tells :command:`clr-service-restart` not to +restart the specified daemon even if the OS defaults permit the daemon to be +restarted. The :command:`clr-service-restart` daemon creates a symlink in +:file:`/etc/clr-service-restart` that points to :file:`/dev/null` as a record. +The example below tells :command:`clr-service-restart` not to restart the +:option:`rngd` daemon after an OS software update. + +.. code-block:: bash + + sudo clr-service-restart disallow rngd + +The :option:`default` option makes :command:`clr-service-restart` revert back +to the OS defaults and delete any symlink in :file:`/etc/clr-service-restart`. +The example below tells :command:`clr-service-restart` to restart +:option:`rngd` automatically again, because :option:`rngd` is whitelisted for +automatic service restarts by default in |CL|. + +.. code-block:: bash + + sudo clr-service-restart default rngd + +Monitor options for clr-service-restart +======================================= + +:command:`clr-service-restart` works in the background and is invoked with +:command:`swupd` automatically. Review the journal output to verify that +services are restarted after an OS software update. + +To monitor :command:`clr-service-restart`, use one or both options described +below. + +:option:`-n` + +This option makes :command:`clr-service-restart` perform no restarts. Instead +it displays the services that could potentially be restarted. When used, +:command:`clr-service-restart` outputs a list of messages showing: + +* Which service needs a restart. +* What unit it is. +* Why it needs a restart. +* Which command is required to restart the unit. + +:option:`-a` + +This option makes :command:`clr-service-restart` consider all system services, +not only the ones that are whitelisted. Because the default whitelist in |CL| +is relatively short, you can use this option to restart all impacted services +when you log in on the system. + +If you pass both options (:option:`-a` and :option:`-n`), +:command:`clr-service-restart` displays a complete list of system services +that require a restart. Use both options to verify that all desired daemons +are restarted. + + +Telemetry +********* + +:command:`clr-service-restart` may cause problems such as a short service +outage when a daemon is being restarted, or if a daemon fails to properly +restart. To minimize issues, :command:`clr-service-restart` creates a +telemetry record and sends it to the optional |CL| telemetry service if both +conditions below are met: + +* If a unit fails to automatically restart after an OS update. +* If that unit resides in the system location :file:`/usr/lib/systemd/system`. + +If you do not install the |CL| telemetrics bundle, the data is discarded. If +you install the telemetrics bundle and you opt to send telemetry, then the +system unit name is sent to the |CL| telemetry service. We evaluate the report +and update the whitelist to remove services that are not safe to restart. + +Conclusion +********** + +The |CL| team enjoys coming up with simple and efficient solutions to make +your work easier. We made a github project of :command:`clr-service-restart` +and we invite you to look at the code, share your thoughts, and work with us +on improving the project. You can find the project at: + +https://github.com/clearlinux/clr-service-restart diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-01.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-01.png new file mode 100644 index 00000000..69b6a24e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-01.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-02.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-02.png new file mode 100644 index 00000000..d24c0612 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-02.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-03.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-03.png new file mode 100644 index 00000000..2fbe4cea Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-03.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-04.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-04.png new file mode 100644 index 00000000..6a01685f Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-04.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-05.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-05.png new file mode 100644 index 00000000..22f40dfc Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-05.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-06.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-06.png new file mode 100644 index 00000000..eeb4b36e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-06.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-07.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-07.png new file mode 100644 index 00000000..4f46b24e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-07.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-08.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-08.png new file mode 100644 index 00000000..d86378bb Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-08.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-09.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-09.png new file mode 100644 index 00000000..416fb206 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-09.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-10.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-10.png new file mode 100644 index 00000000..3efc71e4 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-10.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-11.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-11.png new file mode 100644 index 00000000..8cd9d0a9 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-11.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-12.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-12.png new file mode 100644 index 00000000..36f43d3e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-12.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-13.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-13.png new file mode 100644 index 00000000..30f77a88 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-13.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-14.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-14.png new file mode 100644 index 00000000..e115111e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-14.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-15.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-15.png new file mode 100644 index 00000000..7d2111fa Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player-preconf/vmw-player-preconf-15.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-01.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-01.png new file mode 100644 index 00000000..69b6a24e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-01.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-02.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-02.png new file mode 100644 index 00000000..3f1833c7 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-02.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-03.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-03.png new file mode 100644 index 00000000..2fbe4cea Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-03.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-04.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-04.png new file mode 100644 index 00000000..6a01685f Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-04.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-05.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-05.png new file mode 100644 index 00000000..a0bd62a2 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-05.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-06.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-06.png new file mode 100644 index 00000000..eeb4b36e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-06.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-07.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-07.png new file mode 100644 index 00000000..0a86cdbb Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-07.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-08.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-08.png new file mode 100644 index 00000000..4f46b24e Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-08.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-09.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-09.png new file mode 100644 index 00000000..7d2111fa Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-09.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-10.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-10.png new file mode 100644 index 00000000..7ef566b2 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-10.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-11.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-11.png new file mode 100644 index 00000000..2387eb46 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-11.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-12.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-12.png new file mode 100644 index 00000000..7ea29cc7 Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-12.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-13.png b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-13.png new file mode 100644 index 00000000..7d2111fa Binary files /dev/null and b/source/clear-linux/get-started/virtual-machine-install/figures/vmw-player/vmw-player-13.png differ diff --git a/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst b/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst index a97fe829..760f8500 100644 --- a/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst +++ b/source/clear-linux/get-started/virtual-machine-install/virtual-machine-install.rst @@ -32,6 +32,7 @@ appropriate set of step-by-step instructions to proceed. virtualbox vmware-esxi-install-cl vmware-esxi-preconfigured-cl-image - vmware-player + vmw-player + vmw-player-preconf hyper-v diff --git a/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst b/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst new file mode 100644 index 00000000..4cde924b --- /dev/null +++ b/source/clear-linux/get-started/virtual-machine-install/vmw-player-preconf.rst @@ -0,0 +1,314 @@ +.. _vmw-player-preconf: + +Run pre-configured Clear Linux image as a VMware\* Workstation Player guest OS +############################################################################## + +`VMware Workstation 14 Player`_ is a type 2 hypervisor. It runs on top of +another operating system such as Windows\* or Linux\*. With VMware ESXi, you +can create, configure, manage, and run |CLOSIA| :abbr:`VMs (Virtual Machines)` +on your local system. + +This section shows how to deploy a pre-configured |CL| VMware image on +VMware Workstation 14 Player. + +In this tutorial, we perform the following steps: + +#. Install the VMware Workstation Player hypervisor +#. Download the latest |CL| pre-configured image +#. VMware image Verify the integrity of the |CL| image +#. Uncompress the |CL| image +#. Create and configure a new VM +#. Attach the pre-configured VMware |CL| image +#. Enable EFI boot support +#. Power on the VM + +.. note:: + + The screenshots on this document show the Windows\* version of the + VMware Workstation 14 Player. The menus and prompts are similar to those + in the Linux version save some minor wording differences. + +Install the VMware Workstation Player hypervisor +************************************************ + +#. Enable :abbr:`Intel® VT (Intel® Virtualization Technology)` and + :abbr:`Intel® VT-d (Intel® Virtualization Technology for Directed I/O)` in + your system's BIOS. + +#. `VMware Workstation 14 Player`_ is available for Windows and Linux. + Download your preferred version. + +#. Depending on which OS you're running, install it by following one of these + instructions: + + * On supported Linux distros: + + #. Enable a GUI desktop. + + #. Start a terminal emulator. + + #. Start the installer by issuing the command below and follow the guided + steps. + + .. code-block:: console + + sudo sh ./VMware-Player-[version number].x86_64.bundle + + * On Windows: + + #. Start the installer. + #. Follow the setup wizard. + +For additional help, see the `VMware Workstation Player guide`_. + +Clear Linux image types +*********************** + +.. include:: ../../guides/maintenance/image-types.rst + :Start-after: image-types-content: + +Download the latest Clear Linux VMware image +******************************************** + +Get the latest |CL| VMware image from the `image`_ repository. +Look for :file:`clear-[version number]-vmware.vmdk.xz`. + +.. include:: ../../guides/maintenance/download-verify-uncompress-windows.rst + :Start-after: verify-windows: + +We also provide instructions for other operating systems: + +* :ref:`download-verify-uncompress-linux` +* :ref:`download-verify-uncompress-mac` + +Create and configure a new VM +***************************** + +#. Start the `VMware Workstation Player` app. +#. On the home screen, click :guilabel:`Create a New Virtual Machine`. See + figure 1. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-01.png + :scale: 100% + :alt: VMware Workstation 14 Player - Create a new virtual machine + + Figure 1: VMware Workstation 14 Player - Create a new virtual machine + +#. On the :guilabel:`Welcome to the New Virtual Machine Wizard` screen, select + the :guilabel:`I will install the operating system later` option. + See figure 2. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-02.png + :scale: 100% + :alt: VMware Workstation 14 Player - Select install operating system + + Figure 2: VMware Workstation 14 Player - Select install operating system + later. + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Select a Guest Operating System` screen, set the + :guilabel:`Guest operating system` setting to :guilabel:`Linux`. + See figure 3. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-03.png + :scale: 100% + :alt: VMware Workstation 14 Player - Select guest operating system type + + Figure 3: VMware Workstation 14 Player - Select guest operating system + type + +#. Set :guilabel:`Version` setting to + :guilabel:`Other Linux 3.x or later kernel 64-bit`. + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Name the Virtual Machine` screen, give your new VM a name. + See figure 4. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-04.png + :scale: 100% + :alt: VMware Workstation 14 Player - Name virtual machine + + Figure 4: VMware Workstation 14 Player - Name virtual machine + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Specify Disk Capacity` screen, click the :guilabel:`Next` + button. Keep the default disk settings unchanged. When we attach the pre-configured |CL| + VMware image, we will remove the default virtual disk and replace it with the + pre-configured one. See figure 5. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-05.png + :scale: 100% + :alt: VMware Workstation 14 Player - Set disk capacity + + Figure 5: VMware Workstation 14 Player - Set disk capacity + +#. On the :guilabel:`Ready to Create Virtual Machine` screen, click the + :guilabel:`Customize Hardware...` button. See figure 6. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-06.png + :scale: 100% + :alt: VMware Workstation 14 Player - Customize hardware + + Figure 6: VMware Workstation 14 Player - Customize hardware + +#. Under the :guilabel:`Device` list, select :guilabel:`Processors`. See + figure 7. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-07.png + :scale: 100% + :alt: VMware Workstation 14 Player - Set virtualization engine option + + Figure 7: VMware Workstation 14 Player - Set virtualization engine option + +#. Under the :guilabel:`Virtualization engine` section, + check :guilabel:`Virtualize Intel VT-x/EPT or AMD-V/RVI`. + +#. To disconnect the virtual CD/DVD (IDE) since it is not needed, under the + :guilabel:`Device` list, select :guilabel:`New CD/DVD (IDE)`. See figure 8. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-08.png + :scale: 100% + :alt: VMware Workstation 14 Player - Disconnect CD/DVD (IDE) + + Figure 8: VMware Workstation 14 Player - Disconnect CD/DVD (IDE) + +#. Under the :guilabel:`Device status` section, uncheck + :guilabel:`Connect at power on`. + +#. Click the :guilabel:`Close` button. + +#. Click the :guilabel:`Finish` button. + +Attach the pre-configured Clear Linux VMware image +************************************************** + +#. Move the downloaded and uncompressed pre-configured |CL| VMware image file + :file:`clear-[version number]-basic.vmdk` to the directory where your + newly-created VM resides. + + .. note:: + + Depending on the OS, you can typically find the VMware VM files under: + + * On Linux distros: :file:`/home/username/vmware` + * On Windows: :file:`C:\Users\username\Documents\Virtual Machines` + +#. On the :guilabel:`VMware Workstation Player` home screen, select your + newly-created VM. See figure 9. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-09.png + :scale: 100% + :alt: VMware Workstation 14 Player - Edit virtual machine settings + + Figure 9: VMware Workstation 14 Player - Edit virtual machine settings + +#. Click :guilabel:`Edit virtual machine settings`. + +#. To remove the default hard disk, under the :guilabel:`Device` list, select + :guilabel:`Hard Disk (SCSI)`. See figure 10. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-10.png + :scale: 100% + :alt: VMware Workstation 14 Player - Remove hard drive + + Figure 10: VMware Workstation 14 Player - Remove hard drive + +#. Click the :guilabel:`Remove` button. + +#. To add a new hard disk and attach the pre-configured |CL| VMware image, + click the :guilabel:`Add...` button. See Figure 11. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-11.png + :scale: 100% + :alt: VMware Workstation 14 Player - Add new hard drive + + Figure 11: VMware Workstation 14 Player - Add new hard drive + +#. Under the :guilabel:`Hardware types` section, select :guilabel:`Hard Disk`. + +#. Click the :guilabel:`Next` button. + +#. Select your preferred :guilabel:`Virtual disk type`. See figure 12. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-12.png + :scale: 100% + :alt: VMware Workstation 14 Player - Select virtual disk type + + Figure 12: VMware Workstation 14 Player - Select virtual disk type + +#. Select the :guilabel:`Use an existing virtual disk` option. See figure 13. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-13.png + :scale: 100% + :alt: VMware Workstation 14 Player - Use existing virtual disk + + Figure 13: VMware Workstation 14 Player - Use existing virtual disk + +#. Click the :guilabel:`Browse` button and select the pre-configured |CL| +VMware image file. See figure 14. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-14.png + :scale: 100% + :alt: VMware Workstation 14 Player - Select ready-made VMware |CL| + + Figure 14: VMware Workstation 14 Player - Select ready-made VMware |CL| + image file + +#. Click the :guilabel:`Finish` button. + + .. note:: + + When asked to convert the existing virtual disk to a newer format, + selecting either option works. + +Enable UEFI boot support +************************ + +|CL| needs UEFI support to boot.To enable it, add the +following line to the end of your VM's :file:`.vmx` file: + +.. code-block:: console + + firmware = "efi" + +.. note:: + + Depending on the OS, you can typically find the VMware VM files under: + + * On Linux distros: :file:`/home/username/vmware` + * On Windows: :file:`C:\\Users\\username\\Documents\\Virtual Machines` + +Power on the VM +*************** + +After configuring the settings above, power on your |CL| virtual machine. + +#. On the :guilabel:`VMware Workstation Player` home screen, select your + VM. See figure 15. + + .. figure:: figures/vmw-player-preconf/vmw-player-preconf-15.png + :scale: 100% + :alt: VMware Workstation 14 Player - Power on virtual machine + + Figure 15: VMware Workstation 14 Player - Power on virtual machine + +#. Click :guilabel:`Play virtual machine`. + +For other guides on using the VMWare Player and ESXi, see: + +* :ref:`vmw-player` +* :ref:`vmware-esxi-install-cl` +* :ref:`vmware-esxi-preconfigured-cl-image` + +.. _VMware ESXi: https://www.vmware.com/products/esxi-and-esx.html +.. _VMware Workstation 14 Player: https://www.vmware.com/products/workstation-player.html +.. _VMware Workstation Player guide: https://docs.vmware.com/en/VMware-Workstation-Player/index.html +.. _latest: https://download.clearlinux.org/image/ +.. _image: https://download.clearlinux.org/image + + + + diff --git a/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst b/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst new file mode 100644 index 00000000..5bd5afcd --- /dev/null +++ b/source/clear-linux/get-started/virtual-machine-install/vmw-player.rst @@ -0,0 +1,313 @@ +.. _vmw-player: + +Install Clear Linux as a VMware\* Workstation Player guest OS +############################################################# + +`VMware Workstation 14 Player`_ is a type 2 hypervisor. It runs on top of +another operating system such as Windows or Linux. With VMware ESXi, you can +create, configure, manage, and run |CLOSIA| :abbr:`VMs (Virtual Machines)` on +your local system. + +This section shows how to create a new VM and install |CL| into it with the +VMware Workstation 14 Player hypervisor. Installing |CL| into a new VM +provides you flexibility when configuring the VM. You can configure the VM's +size, number of partitions, installed bundles, etc. + +In this tutorial, we perform the following steps: + +#. Install the VMware Workstation Player hypervisor +#. Download the latest |CL| installer ISO +#. Verify the integrity of the |CL| image +#. Uncompress the |CL| image +#. Create and configure a new VM +#. Attach the |CL| installer ISO to the VM +#. Install |CL| into the new VM +#. Detach the |CL| installer ISO from the VM +#. Power off the VM +#. Enable EFI boot support +#. Power on the VM + +If you prefer to use a pre-configured |CL| VMware image instead, +see our :ref:`vmw-player-preconf` guide. + +VMware offers a type 1 hypervisor called `VMware ESXi`_ designed for the +cloud environment. For information on how to install |CL| as guest OS on +it, see :ref:`vmware-esxi-install-cl`. + +.. note:: + + The screenshots on this document show the Windows\* version of the + VMware Workstation 14 Player. The menus and prompts are similar to those + in the Linux version save some minor wording differences. + +Install the VMware Workstation Player hypervisor +************************************************ + +#. Enable :abbr:`Intel® VT (Intel® Virtualization Technology)` and + :abbr:`Intel® VT-d (Intel® Virtualization Technology for Directed I/O)` in + your system's BIOS. + +#. `VMware Workstation 14 Player`_ is available for Windows and Linux. + Download your preferred version. + +#. Install VMware Workstation 14 Player following the instructions + appropriate for your system's OS: + + * On supported Linux distros: + + #. Enable a GUI desktop. + #. Start a terminal emulator. + #. Start the installer by issuing the command below and follow the + guided steps. + + .. code-block:: console + + $ sudo sh ./VMware-Player-[version number].x86_64.bundle + + * On Windows: + + #. Start the installer. + #. Follow the setup wizard. + +For additional help, see the `VMware Workstation Player guide`_. + +Clear Linux image types +*********************** + +.. include:: ../../guides/maintenance/image-types.rst + :Start-after: image-types-content: + + +Download the latest Clear Linux installer ISO +********************************************* + +Get the latest |CL| installer ISO image from the `image`_ repository. +Look for :file:`clear-[version number]-installer.iso.xz`. + +.. include:: ../../guides/maintenance/download-verify-uncompress-windows.rst + :Start-after: verify-windows: + +We also provide instructions for other operating systems: + +* :ref:`download-verify-uncompress-linux` +* :ref:`download-verify-uncompress-mac` + +Create and configure a new VM +***************************** + +#. Start the `VMware Workstation Player` app. + +#. On the home screen, click :guilabel:`Create a New Virtual Machine`. See + Figure 1. + + .. figure:: figures/vmw-player/vmw-player-01.png + :scale: 100% + :alt: VMware Workstation 14 Player - Create a new virtual machine + + Figure 1: VMware Workstation 14 Player - Create a new virtual + machine + +#. On the :guilabel:`Welcome to the New Virtual Machine Wizard` screen, + select the :guilabel:`Installer disc image file (iso)` option. + See Figure 2. + + .. figure:: figures/vmw-player/vmw-player-02.png + :scale: 100% + :alt: VMware Workstation 14 Player - Select |CL| installer ISO + + Figure 2: VMware Workstation 14 Player - Select |CL| installer ISO + +#. Click the :guilabel:`Browse` button and select the uncompressed |CL| + installer ISO. + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Select a Guest Operating System`, set the + :guilabel:`Guest operating system` setting to :guilabel:`Linux`. See + Figure 3. + + .. figure:: figures/vmw-player/vmw-player-03.png + :scale: 100% + :alt: VMware Workstation 14 Player - Select guest operating system type + + Figure 3: VMware Workstation 14 Player - Select guest operating system + type + +#. Set the :guilabel:`Version` setting to + :guilabel:`Other Linux 3.x or later kernel 64-bit`. + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Name the Virtual Machine` screen, name the new VM. See + Figure 4. + + .. figure:: figures/vmw-player/vmw-player-04.png + :scale: 100% + :alt: VMware Workstation 14 Player - Name virtual machine + + Figure 4: VMware Workstation 14 Player - Name virtual machine + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Specify Disk Capacity` screen, set the VM's maximum disk + size. See Figure 5. + + .. figure:: figures/vmw-player/vmw-player-05.png + :scale: 100% + :alt: VMware Workstation 14 Player - Set disk capacity + + Figure 5: VMware Workstation 14 Player - Set disk capacity + + .. note:: + A minimal |CL| installation can exist on 600MB of drive space. + See :ref:`system-requirements` for more details. + +#. Click the :guilabel:`Next` button. + +#. On the :guilabel:`Ready to Create Virtual Machine` screen, click the + :guilabel:`Customize Hardware...` button. See Figure 6. + + .. figure:: figures/vmw-player/vmw-player-06.png + :scale: 100% + :alt: VMware Workstation 14 Player - Customize hardware + + Figure 6: VMware Workstation 14 Player - Customize hardware + +#. Select :guilabel:`Memory` and set the size to 2GB. See Figure 7. + + .. figure:: figures/vmw-player/vmw-player-07.png + :scale: 100% + :alt: VMware Workstation 14 Player - Set memory size + + Figure 7: VMware Workstation 14 Player - Set memory size + + .. note:: + The |CL| installer ISO needs a minimum of 2GB of RAM. + After completing installation, |CL| can run on as little as + 128MB of RAM. Thus, you can reduce the memory size if needed. + See :ref:`system-requirements` for more details. + +#. Under the :guilabel:`Device` list, select :guilabel:`Processors`. See + Figure 8. + + .. figure:: figures/vmw-player/vmw-player-08.png + :scale: 100% + :alt: VMware Workstation 14 Player - Set virtualization engine option + + Figure 8: VMware Workstation 14 Player - Set virtualization engine + option + +#. Under the :guilabel:`Virtualization engine` section, + check :guilabel:`Virtualize Intel VT-x/EPT or AMD-V/RVI`. + +#. Click the :guilabel:`Close` button. + +#. Click the :guilabel:`Finish` button. + +Install Clear Linux into the new VM +*********************************** + +#. Select the newly-created VM and click the :guilabel:`Play virtual machine` + button. See Figure 9. + + .. figure:: figures/vmw-player/vmw-player-09.png + :scale: 100% + :alt: VMware Workstation 14 Player - Power on virtual machine + + Figure 9: VMware Workstation 14 Player - Power on virtual machine + +#. Follow the :ref:`install-on-target` guide to complete the installation of + |CL|. + +#. After the installation completes, reboot the VM. This reboot restarts the + |CL| installer. + +Detach the |CL| installer ISO from the VM +***************************************** + +#. To enable the mouse pointer so you access VMware Workstation Player's + menus, press :kbd:`` + :kbd:`` on the keyboard. + +#. To disconnect the CD/DVD to stop it from booting the |CL| installer ISO + again, click the :guilabel:`Player` menu. See Figure 10. + + .. figure:: figures/vmw-player/vmw-player-10.png + :scale: 100% + :alt: VMware Workstation 14 Player - Edit CD/DVD settings + + Figure 10: VMware Workstation 14 Player - Edit CD/DVD settings + +#. Go to :menuselection:`Removable Devices-->CD/DVD (IDE)-->Settings`. + +#. On the :guilabel:`Device status` section, uncheck the + :guilabel:`Connected` and the :guilabel:`Connect at power on` settings. + See Figure 11. + + .. figure:: figures/vmw-player/vmw-player-11.png + :scale: 100% + :alt: VMware Workstation 14 Player - Disconnect CD/DVD + + Figure 11: VMware Workstation 14 Player - Disconnect CD/DVD + +#. Click the :guilabel:`OK` button. + +#. To power off the VM, click the :guilabel:`Player` menu. See Figure 12. + + .. figure:: figures/vmw-player/vmw-player-12.png + :scale: 100% + :alt: VMware Workstation 14 Player - Power off virtual machine + + Figure 12: VMware Workstation 14 Player - Power off virtual machine + +#. Go to :guilabel:`Power` and select :guilabel:`Shut Down Guest`. + +Enable UEFI boot support +************************ + +|CL| needs UEFI support to boot. To enable UEFI, add the +following line to the end of your VM's :file:`.vmx` file: + +.. code-block:: bash + + firmware = "efi" + +.. note:: + + Depending on the OS, you can typically find the VMware VM files under: + + * On Linux distros: :file:`/home/username/vmware` + * On Windows: :file:`C:\\Users\\username\\Documents\\Virtual Machines` + +Power on the VM +*************** + +After configuring the settings above, power on your |CL| virtual machine. + +#. On the :guilabel:`VMware Workstation Player` home screen, select your + VM. See Figure 13. + + .. figure:: figures/vmw-player/vmw-player-13.png + :scale: 100% + :alt: VMware Workstation 14 Player - Power on virtual machine + + Figure 13: VMware Workstation 14 Player - Power on virtual machine + +#. Click :guilabel:`Play virtual machine`. + +For other guides on using the VMWare Player and ESXi, see: + +* :ref:`vmw-player-preconf` +* :ref:`vmware-esxi-install-cl` +* :ref:`vmware-esxi-preconfigured-cl-image` + +.. _VMware ESXi: https://www.vmware.com/products/esxi-and-esx.html + +.. _VMware Workstation 14 Player: + https://www.vmware.com/products/workstation-player.html + +.. _VMware Workstation Player guide: + https://docs.vmware.com/en/VMware-Workstation-Player/index.html + +.. _latest: https://download.clearlinux.org/image/ + +.. _image: https://download.clearlinux.org/image diff --git a/source/clear-linux/get-started/virtual-machine-install/vmware-player.rst b/source/clear-linux/get-started/virtual-machine-install/vmware-player.rst deleted file mode 100644 index b2d36417..00000000 --- a/source/clear-linux/get-started/virtual-machine-install/vmware-player.rst +++ /dev/null @@ -1,116 +0,0 @@ -.. _vmware-player: - -Use VMware\* Player -################### - -This section explains how to run Clear Linux OS for Intel® Architecture -within a `VMware Player`_ environment. - -Please ensure you have enabled `Intel® Virtualization Technology -`_ -(Intel® VT) and `Intel® Virtualization Technology for Directed I/O -`_ -(Intel® VT-d) in your BIOS/UEFI firmware configuration. - -Install VMware Player -===================== - -VMware Workstation Player, formerly VMware Player, is a virtualization -software package for x64 computers running Microsoft Windows or Linux. -Download VMware player from the `VMware website`_. - -Player on Linux ---------------- - -For the Linux option, you should have a :file:`VMware-Player-{version}_FILE.bundle` -file. To install it, run: - -:: - - $ sudo bash ./VMware-Player-{version}.x86_64.bundle - -Player on Windows ------------------ - -Follow the instructions from the Setup Assistant. - - -Prepare Image -============= - -#. Download the `latest`_ |CL| **live** version (clear-XXXX-live.img.xz) - -#. Decompress the downloaded image. Uncompressed image size is ~ **5GB**. - - + On Linux :: - - $ xz -d clear-XXXX-live.img.xz - - + On Windows you can use `7zip`_. - - - Right-click the file to *extract in the same directory*. - - .. image:: ./figures/7zipwin.png - :alt: 7zip extract here command - -#. Convert the installer to :abbr:`VMDK (Virtual Machine Disk)` format. - - * On Linux, you can use ``qemu-img convert``:: - - $ qemu-img convert -O vmdk clear-VERSION-live.img clear.vmdk - - * On Windows, you can convert the live image to VMDK format - (from RAW format to VMDK) with a tool like *VBoxManage* from - `VirtualBox`_. You can refer on - :ref:`how to create a VM on VirtualBox ` as example. - - -Run using VMware Player -======================= - - -Create a new virtual machine by following the next steps. - -#. Launch **VMWare Workstation Player**. - -#. On main window click on “Create a new Virtual Machine”. - - * Select “**I will install the operating system later**”, and click on - “Next”. - * Select “**Linux**” as “Guest operating system” and version **Other Linux - 3.x kernel 64-bit**. - * Type a name for the new virtual machine. - * Perform the *remaining steps* using the default options. - -#. Change boot type to EFI. You must change the VMware virtual machine - *configuration* to **Support EFI firmware**; you can do this by editing - the configuration ``.vmx`` file located in the virtual machine folder and - adding the following line:: - - firmware = "efi" - -#. Attach the prepared image as SATA disk. And when you have a new virtual - machine, edit its configuration as follows: - - * Click on “Edit virtual machine settings”. - * Remove any default attached hard disk. - * Click on “Add” option below devices list tab and choose Hard disk. - - * Choose **SATA** as the virtual disk type. - * Use the existing Clear Linux OS for Intel Architecture virtual disk - - The live disk must be set as ``SATA 0:1 Hard Disk (SATA)``; you can - verify this under the “Advanced" section of the disk settings. - -Start the virtual machine -========================= - -After configuring the settings above, start the virtual machine. - - -.. _VMware website: https://www.vmware.com/products/player/playerpro-evaluation.html -.. _VMware Player: http://www.vmware.com/products/player/ -.. _latest: https://download.clearlinux.org/image/ -.. _7zip: http://www.7-zip.org/ -.. _VirtualBox: https://www.virtualbox.org/ - diff --git a/source/clear-linux/guides/maintenance/download-verify-uncompress-linux.rst b/source/clear-linux/guides/maintenance/download-verify-uncompress-linux.rst index 8cd944c7..0a123cb9 100644 --- a/source/clear-linux/guides/maintenance/download-verify-uncompress-linux.rst +++ b/source/clear-linux/guides/maintenance/download-verify-uncompress-linux.rst @@ -33,11 +33,12 @@ SHA512 checksum file, which is designated with the suffix `-SHA512SUMS`. .. code-block:: console - $ sha512sum ./clear-[version number]-[image type].[compression type] | diff ./clear-[version number]-[image type].[compression type]-SHA512SUMS - + $ sha512sum -c ./clear-[version number]-[image type].[compression type]-SHA512SUMS If the checksum of the downloaded image is different than the original -checksum, the differences will displayed. Otherwise, an empty output indicates -a match and your downloaded image is good. +checksum, a warning will be displayed with a message indicating the computed +checksum does **not** match. Otherwise, the name of the image will be printed on +the screen followed by `OK`. Uncompress the Clear Linux image ******************************** diff --git a/source/clear-linux/guides/maintenance/enable-user-space.rst b/source/clear-linux/guides/maintenance/enable-user-space.rst index e3a706bc..8f8e23d1 100644 --- a/source/clear-linux/guides/maintenance/enable-user-space.rst +++ b/source/clear-linux/guides/maintenance/enable-user-space.rst @@ -97,6 +97,10 @@ To be able to execute all applications with root privileges, we must add the Install a GUI to test sudo -------------------------- +.. note:: +If you are following this sequence after just setting up the pre-configured VMware virtual machine from +the repo, you must :ref:`increase virtual disk size` or the following step +will fail. To test the :command:`sudo` command and ensure it is set up correctly, install the Gnome Desktop Manager (gdm) and start it. diff --git a/source/clear-linux/guides/maintenance/maintenance.rst b/source/clear-linux/guides/maintenance/maintenance.rst index e2e48bd6..2ca86462 100644 --- a/source/clear-linux/guides/maintenance/maintenance.rst +++ b/source/clear-linux/guides/maintenance/maintenance.rst @@ -13,6 +13,7 @@ maintaining |CLOSIA| after :ref:`installation ` is completed. update bulk-provision mixer + swupdaddpkg validate-signatures telemetry-enable time diff --git a/source/clear-linux/guides/maintenance/mixer.rst b/source/clear-linux/guides/maintenance/mixer.rst index be802d11..90e54dec 100644 --- a/source/clear-linux/guides/maintenance/mixer.rst +++ b/source/clear-linux/guides/maintenance/mixer.rst @@ -3,278 +3,784 @@ Use mixer tool ############## -*Mixing* refers to composing an operating system for specific use cases. While -the default Clear Linux\* OS for Intel® Architecture provides options to -install bundles for various server capabilities, some developers may wish to 1) -augment the operating system itself with functionality from their own packages -or 2) modify the structure of current bundles to cater to their particular -needs. +*Mixing* refers to composing an operating system for specific use cases. +While the default |CLOSIA| provides options to install bundles for various +server capabilities, some developers may wish to either augment the +operating system itself with functionality from their own packages or modify +the structure of current bundles to cater to their particular needs. Prerequisites -============= +************* -To start working with the Mixer tools, you'll need a recent image of Clear -Linux OS for Intel Architecture with the following bundle installed. If you -don't have it already, you can add it with the :command:`swupd bundle-add` -command like so:: +To start working with the mixer tools, you need a recent image of |CL| with +the `mixer` bundle installed. If the bundle is not yet installed, you can +add it with the :command:`swupd bundle-add` command as follows: - # swupd bundle-add mixer + .. code-block:: bash -Current Workflow -================ + sudo swupd bundle-add mixer -Mixing ------- +Current mixing workflow +*********************** -#. **Create a workspace**. Create an empty directory in your Clear image to - use as a "workspace" for mixing. For these steps, we assume your workspace - location is :file:`/home/clr/mix`. +There are two different workflows to create your own mix. -#. **Configure builder.conf**. Copy the template conf file +The following workflow applies if the mix includes your own +:abbr:`RPMs (RPM Package Manager files)`: + +#. `Create a workspace`_ +#. `Generate the starting point for your mix`_ +#. `Edit builder.conf`_ +#. `Create or locate RPMs for the mix`_ +#. `Import RPMs into workspace`_ +#. `Create a local RPM repo`_ +#. `List, edit, create, add, remove, or validate bundles`_ +#. `Build the bundle chroots`_ +#. `Create an update`_ +#. `Create an image`_ + +Alternatively, the following workflow applies if the mix only uses |CL| +content. + +#. `Create a workspace`_ +#. `Generate the starting point for your mix`_ +#. `Edit builder.conf`_ +#. `List, edit, create, add, remove, or validate bundles`_ +#. `Build the bundle chroots`_ +#. `Create an update`_ +#. `Create an image`_ + +The following sections contain detailed information on every step of +these workflows. + +Create a workspace +****************** + +Use the following command to create an empty directory in your |CL| image to +use as a **workspace** for mixing: + +.. code-block:: bash + + mkdir /home/clr/mix + +This guide assumes your workspace location is :file:`/home/clr/mix`. + +Generate the starting point for your mix +**************************************** + +In your workspace, initialize mixer with the following command: + +.. code-block:: bash + + mixer init + +This command initializes your workspace so you can make a mix at version 10 +based on the latest released upstream |CL| version. If a :file:`builder.conf` +file is not already present in your workspace, mixer creates a default +configuration file. Additionally, mixer creates several version and tracking +files, and two bundle directories: :file:`local-bundles` and +:file:`upstream-bundles`. + +If you wish to start with a different version of upstream |CL| or a +different initial mix version, you can specify these options as flags. +For example: + +.. code-block:: bash + + mixer init --clear-version 21060 --mix-version 100 + + +Additionally, to build a mix with your own custom RPMs, use the optional +:option:`--local-rpms` flag, for example: + +.. code-block:: bash + + mixer init --local-rpms + +This command creates the :file:`local-yum` and :file:`local-rpms` +directories in your mix workspace and adds their paths to the generated +:file:`builder.conf`. If the configuration file already exists, you must add +the paths manually. For more information on using these directories or +setting them up manually, see `Create or locate RPMs for the mix`_. + +If all upstream |CL| bundles will be part of the mix, you can easily add them all +during initialization with the optional :option:`--all-upstream` flag. For +example: + +.. code-block:: bash + + mixer init --all-upstream + +Finally, you may want to track the contents of your mixer workspace with a +git repository. This is a great way to track changes to your mix's content +or to revert to earlier versions if something goes wrong. Mixer can set this +up automatically with the optional :option:`--git` flag, for example: + +.. code-block:: bash + + mixer init --git + +.. note:: + You can use any or all of the above optional flags at the same time, for example: + + .. code-block:: bash + + mixer init --clear-version 21060 --mix-version 100 --local-rpms --all-upstream --git + +Edit builder.conf +***************** + +To configure the mixer tool, edit the :file:`builder.conf` as needed. + +The file :file:`builder.conf` is read automatically from the current +workspace directory. Use the :option:`--config` flag during initialization +to specify an alternate path to the file as needed. + +The :file:`builder.conf` file has different sections, for example: + +* The `[Builder]` section provides the mixer tools with the required + configuration options. This section defines the path where the generated + bundles and update metadata are published. + +* The `[swupd]` section contains specific update parameters. The + :abbr:`swupd-server (software update server)` creates an update using + said specific update parameters. + +Edit the configuration file according to your needs with the command: + +.. code-block:: bash + + vim /home/clr/mix/builder.conf + +Your version of the :file:`builder.conf` file should resemble the +following example: + +.. code-block:: console + + [Mixer] + LOCAL_BUNDLE_DIR=/home/clr/mix/local-bundles + + [Builder] + SERVER_STATE_DIR=/home/clr/mix/update + BUNDLE_DIR=/home/clr/mix/mix-bundles + YUM_CONF=/home/clr/mix/.yum-mix.conf + CERT=/home/clr/mix/Swupd_Root.pem + VERSIONS_PATH=/home/clr/mix + + [swupd] + BUNDLE=os-core-update + CONTENTURL= + VERSIONURL= + FORMAT=1 + + [Server] + debuginfo_banned=true + debuginfo_lib=/usr/lib/debug/ + debuginfo_src=/usr/src/debug/ + +The following variables require further explanation: + +* The `LOCAL_BUNDLE_DIR` variable sets the path where mixer stores the local + bundle definition files. These bundle definition files include any new, + original bundles you create, along with any edited versions of upstream + |CL| bundles. + +* The `SERVER_STATE_DIR` variable sets the path for the output of the mix + content. Mixer automatically creates the path for you, but the path can be + set to any location. In this example, we use the workspace directory. + +* The `BUNDLE_DIR` variable sets the path where mixer temporarily stores the + bundle definition files while building chroots. Only the legacy + chroot-builder uses this path. By default, mixer does not generate this + directory until the directory is needed. In our example, the path is set to + :file:`/home/clr/mix/mix-bundles`. The new chroot-builder does not generate + the folder at all. + +* The `YUM_CONF` variable sets the path where mixer automatically generates + the :file:`.yum-mix.conf` yum configuration file. The yum configuration file + points the chroot-builder to the path where the RPMs are stored. + +* The `CERT` variable sets the path where mixer stores the + :file:`Swupd_Root.pem` certificate file. The chroot-builder needs the + certificate file to sign the root :file:`Manifest.MoM` file to provide + security for content verification. The value of the `CERT` variable can + point to a different certificate. The chroot-builder inserts the + certificate specified in this value into the + :file:`/os-core-update/usr/share/clear/update-ca/` path. The software update + client uses this certificate to verify the :file:`Manifest.MoM` file's + signature. For now, we **strongly** recommend that you do not modify + this line, as the certificate that :abbr:`swupd (Software Updater)` + expects needs to have a very specific configuration to sign and verify + properly. Mixer automatically generates the certificate, if you do not + provide the path to an existing one, and signs the :file:`Manifest.MoM` + file to provide security for the updated content you create. + +* The `CONTENTURL` and `VERSIONURL` variables set the domain or IP address + where swupd looks for your update content and the corresponding version. You + must set these variables to the domain or IP-address of the server hosting the + update content. You can use any web server to host your update content. To learn + how to install and configure web server using |CL|, visit + :ref:`web-server-install`. For our example, the web update content within + the `SERVER_STATE_DIR` directory is located here: + :file:`/home/clr/mix/update/www`. If the web server is on the same machine as + this directory, you can create a symlink to the directory in your web + server's document root to easily host the content. These URLs are + embedded in images created for your mix. The `swupd-client` looks at + these URLs to determine if a new version is available and the location + from where to download the updated content. These links are equivalent + to the |CL| `update page`_ but for the mix. + +* The `FORMAT` variable relates to format bumps. To learn more about the + `FORMAT` option, refer to :ref:`mixer-format` and the `format bumps wiki`_. + For now, leave the `FORMAT` value unchanged. + +* The `VERSIONS_PATH` variable sets the path for the mix version and upstream + |CL| version's two state files: :file:`mixversion` and + :file:`upstreamversion`. Mixer creates both files for you when you set up + the workspace. + +.. note:: If you are working only with |CL| bundles, then + skip to `List, edit, create, add, remove, or validate bundles`_. + +Create or locate RPMs for the mix +********************************* + +If you create RPMs from scratch, you can use `autospec`, `mock`, `rpmbuild`, +or similar tools to build them. If the RPMs are not built on |CL|, ensure +your configuration and toolchain builds them correctly for |CL|, or else +there is no guarantee they will be compatible. For more information on +building the RPMs properly, refer to our `build RPMs instructions`_. + +Import RPMs into workspace +************************** + +#. Create a :file:`local-rpms` directory in your workspace, for example, + :file:`/home/clr/mix/local-rpms`. + +#. Copy the RPMs into the directory you created. + +#. Add the following line to your :file:`builder.conf` file: .. code-block:: console - # cp /usr/share/defaults/bundle-chroot-builder/builder.conf /home/clr/mix/ + LOCAL_RPM_DIR=/home/clr/mix/local-rpms - The file ``builder.conf`` will be read automatically from the current - workspace directory, but the ``-config`` option exists to specify where the - file is if you want to store it elsewhere. To use one in your current - workspace, copy the template to /home/clr/mix. The :file:`.yum-mix.conf` - file will be auto-generated for you, as will the :file:`Swupd_Root.pem`. A - yum configuration is needed for the chroot-builder to know where the RPMs - are hosted, and the certificate file is needed to sign the root Manifest to - provide security for content verification. +Mixer uses this directory to find the RPMs to build a local RPM repo for +yum to use. - Note there are different sections to the builder.conf. The ``[Builder]`` - section provides the mixer tools with required configuration options, - defining where generated bundles and update metadata should get published. - The ``[swupd]`` section is used by swupd-server to create an update with - specific update parameters. +Create a local RPM repo +*********************** - Edit the template configuration file according to your needs. For this - example, your ``builder.conf`` should look like this, with both URL - variables set to the domain or IP of the update server:: - - # vim /etc/bundle-chroot-builder/builder.conf: - - [Builder] - SERVER_STATE_DIR=/home/clr/mix/update - BUNDLE_DIR=/home/clr/mix/mix-bundles - YUM_CONF=/home/clr/mix/.yum-mix.conf - CERT=/home/clr/mix/Swupd_Root.pem - VERSIONS_PATH=/home/clr/mix/ - - [swupd] - BUNDLE=os-core-update - CONTENTURL= - VERSIONURL= - FORMAT=1 - - The SERVER_STATE_DIR is where the mix content will be outputted to, and it - is automatically created for you by the mixer. This can be set to any - location, but for this example let's use the workspace directory. The same - applies for BUNDLE_DIR; it will be generated for you in the location - specified in the builder.conf, in this case ``/home/clr/mix/mix-bundles``. - This is where the bundle definitions are stored for your mix, and it's where - the chroot-builder looks to know what bundles must be installed. - - You may change the ``CERT=/path/to/cert`` line, which tells the chroot - builder to insert the certificate specified for the mix in ``/os-core- - update/usr/share/clear/update-ca/``. This is the certificate used by the - software update client to verify the Manifest.MoM signature. For now, it is - `HIGHLY` recommended that you do not modify this line, as the certificate - swupd expects needs a very specific configuration to sign and verify - properly. The certificate will be automatically generated for you, and the - Manifest.MoM will be signed automatically as well, providing security for - the update content you create. - - The CONTENTURL and VERSIONURL may be an IP address, or a domain name, which - hosts the /home/clr/mix/update/www (SERVER_STATE_DIR) directory. Creating a - symlink to the directory in your server webdir is an easy way to host the - content. A client running the mix will look to that URL to figure out if - there is a new version available and the location from which to download the - update content. - - To learn more about the FORMAT option, please refer to the bottom of this - document "Format Version" and https://github.com/clearlinux/swupd-server/wiki/Format-Bumps. - For now leave the FORMAT value alone and do not increment it. - - The mix version and Clear version will come from two state files: - :file:`.mixversion` and :file:`.clearversion`, both of which will be created - for you when you set-up the workspace and added to the VERSIONS_PATH - defined. - -#. **Generate the starting point for your Mix**. In your workspace, run:: - - # sudo mixer init-mix -clearver 13180 -mixver 10 - - *If you wish to just build a mix that includes all Clear bundles with no modifications, run*:: - - # sudo mixer init-mix -all -clearver 13180 -mixver 10 - -#. **Create/locate RPMs for mix.**. (Steps 4-6 are necessary only if you - want to add your own RPMs to the Mix. If you are working only with Clear - bundles, then skip to Step 7.) - - If you are creating RPMs from scratch, you may use ``autospec``, - ``mock``, ``rpmbuild``, etc. to build them. If they are not - built on Clear, make sure your configuration and toolchain builds them correctly for Clear, or there is no guarantee - they will be compatible. - -#. **Import RPMs into workspace**. The way to do this is to create an - ``rpms`` directory in your workspace (for example ``/home/clr/mix/rpms``), - and to copy the RPMs you want into that directory. The mixer script will - look here for RPMs in order to build a local RPM repo for yum to use. - -#. **Create a local RPM repo**. Create an empty directory in your workspace - named ``local`` and add the paths in your builder.conf:: - - RPMDIR=/home/clr/mix/rpms - REPODIR=/home/clr/mix/local - - These variables are automatically read; you simply need to run:: - - # sudo mixer add-rpms - - After the script exits, you should see your RPMs and a repodata directory in - ``/home/clr/mix/local``. If the RPMs are not all in the local directory, check - to make sure that they are indeed valid RPM files and not corrupt. - -#. **Update/Add bundle definitions**. The mixer uses a local clone of the - ``clr-bundles`` repo to define bundles for the mix. - - To define your bundles: - - #. Navigate to the ``mix-bundles/`` directory. - #. Make any needed modifications to the bundle set. - #. Commit the result:: - - $ git add . - $ git commit -s -m 'Update bundles for mix #' - - You can easily copy bundles over from the - ``clr-bundles/clr-bundles-VER/bundles/`` directory in the case that you - want to simply use existing bundle sets. Note that ``mix-bundles`` should - not have any folders inside of it, only bundle definitions. - - Do *not* modify things in the clr-bundles dir, this is simply a mirror for - you to use or refer to the Clear Linux OS bundle definitions. - - Why do this? With Git history, mixes are easy to revert to or refer - to in the future if something were to go wrong with a new mix. If - you're just testing this out, or if you really do not want to mess with Git, - you can ignore committing for now. - - To add your own bundle, create a bundle definition file in ``mix-bundles/`` - and refer to :file:`mix-bundles/os-core-update` for formatting, but be sure - that the name does not conflict with another bundle. Add your package - name(s) in that bundle definition file to tell it what package(s) must be - installed as part of that bundle. - -#. **Build the bundle chroots** To build all of the ``chroots`` - that are based on the bundles you defined, in your workspace run:: - - # sudo mixer build-chroots - - If you have many bundles defined for your mix, this step may take some time. - -#. **Create update**. In the workspace, run:: - - # sudo mixer build-update - - When the build completes, you'll find your mix update content under - ``/home/clr/mix/update/www/VER``. In this example, it will be located in - ``/home/clr/mix/update/www/``, where is the mix - version you defined, or 10 by default. - - All content to make a fully usable mix will be created by this step, but - note that only zero packs are automatically generated. To create optional - delta packs, run the pack-maker as follows:: - - # sudo mixer-pack-maker.sh --to --from -S /home/clr/mix/update - - The pack-maker will generate all delta packs for changed bundles from - PAST_VERSION to MIX_VERSION. If your STATE_DIR is in a different location be - sure to specify where with the -S option. For the first build, no delta - packs can be created because the "update" is from version 0, which impicitly - has no content, thus no deltas can be generated. For subsequent builds, - mixer-pack-maker.sh can be run to generate delta content between them (i.e - 10 to 20). - -#. **Creating an image** To create a bootable image from your update content, - you will need the configuration file for ister to create images:: - - # curl -O https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json - - Edit this to include all the bundles you want pre-installed into your - image. For a minimal, base image this would be:: - - "Bundles": ["os-core", "os-core-update", "kernel-native"] - - And lastly, set the "Version:" to say which mix version content the image - should be built from, i.e. 10 for your first build. To build the image, - run:: - - # sudo mixer build-image -format 1 - - The output from this should be an image that is bootable as a VM or - installable to baremetal. - - .. note:: - You need to pass in -format if the format you are - building is different than the format of Clear Linux OS you are currently - building on. Format version can be found via +#. Create an empty directory in your workspace named :file:`local-yum`. +#. Add the path to your :file:`builder.conf` file: .. code-block:: console - # cat /usr/share/defaults/swupd/format + LOCAL_REPO_DIR=/home/clr/mix/local-yum -Creating your next Mix version ------------------------------- +#. With these values configured, generate the yum repo with the following + command: -#. **Initialize next Mix version info**. To update the versions and prep for - your next mix: + .. code-block:: bash - Update the .mixversion file to the next version number you want to build. - From this point you can iterate through, starting again at step 4 and doing - modifications as needed. For example: + sudo mixer add-rpms - - Add/Remove/Modify Bundles - - sudo mixer build-chroots - - sudo mixer build-update - - (Optionally) sudo mixer-pack-maker.sh --to --from -S /home/clr/mix/update +After the tool exits, you should see the RPMs and a repository data +directory in :file:`/home/clr/mix/local-yum`. If the RPMs are not all in this +:file:`local-yum` directory, check to ensure that the RPM files are valid +and not corrupt. +List, edit, create, add, remove, or validate bundles +**************************************************** -#. **Update Bundles (Optional)**. Update ``clr-bundles``. In the workspace, - run:: +The bundles in the mix are specified in the mix bundle list. Mixer stores +this list as a flat file called :file:`mixbundles` in the path set by the +`VERSIONS_PATH` variable of the :file:`builder.conf` file. Mixer +automatically generates the :file:`mixbundles` list file during +initialization. Mixer reads and writes the bundle list file when you change +the bundles of the mix. - # sudo mixer get-bundles +List the bundles in the mix +=========================== - This step is optional because it is only needed when you want to update the - upstream clr-bundles in your workspace to a new version, which requires - updating the .clearversion file. +To view the bundles already in the mix, enter the following command: -Format Version --------------- +.. code-block:: bash -The "format" used in ``builder.conf`` might be more precisely referred to as an -OS "compatibility epoch". Versions of the OS within a given epoch are fully -compatible with themselves and can update to any version in that epoch. Across -the format boundary *something* has changed in the OS, such that updating from -build M in format X, to build N in format Y will not work. Generally this -occurs when the software updater or manifests changed in a way that is no -longer compatible with the previous update scheme. + mixer bundle list -A format increment is the way we insure pre- and co-requisite changes flow out -with proper ordering. The update client will only ever update to the latest -release in its respective format version (unless overridden by command line -flags), thus we can guarantee all clients will update to the final version in -their given format, which *must* contain all the changes needed to understand -the content built in the following format. Only after reaching the final -release in the old format will a client be able to continue to update to -releases in the new format. +This command shows a list of every bundle in the mix. Bundles can include +other bundles. Those nested bundles can themselves include other +bundles. When listing bundles with this command, mixer automatically +recurses through the includes to show every single bundle in the mix. -For the creation of a custom mix, the format version should start at '1', or -some known number, and increment only when a compatibility breakage is -introduced. Normal updates (updating a software package for example) do not -require a format increment. +If you see an unexpected bundle in the list, that bundle is probably included +in another bundle. Use the :option:`--tree` flag to get a better view of how +a bundle ended up in the mix, for example: + +.. code-block:: bash + + mixer bundle list --tree + +This command shows a visual representation of the inclusion relationships +between the bundles in the mix. + +Bundles fall into two categories: **upstream** and **local**. + +Upstream bundles are those provided by |CL|. + +Mixer automatically downloads and caches upstream bundle definition files. +These definition files are stored in the :file:`upstream-bundles` directory +in the workspace. Do **not** modify the files in this directory. This +directory is simply a mirror for mixer to use. + +The mixer tool automatically caches the bundles for the |CL| version +configured in the :file:`upstreamversion` file. Mixer also cleans up old +versions once they are no longer needed. You can see the available upstream +bundles with the following command: + +.. code-block:: bash + + mixer bundle list upstream + +Local bundles are bundles that you create, or are edited versions of upstream +bundles. + +Local bundle definition files live in the :file:`local-bundles` directory. +The `LOCAL_BUNDLE_DIR` variable sets the path of this directory in your +:file:`builder.conf` configuration file. For this example, the path is +:file:`/home/clr/mix/local-bundles`. You can see the available local bundles +with the following command: + +.. code-block:: bash + + mixer bundle list local + +Both the local and upstream :command:`bundle list` commands accept the +:option:`--tree` flag to show a visual representation of the inclusion relationships +between the bundles in the mix. + +Edit the bundles in the mix +=========================== + +**Mixer always checks local bundles first and the upstream bundles second.** + +Therefore, bundles in the :file:`local-bundles` directory always take +precedence over any upstream bundles that have the same name. + +This precedence enables you to edit upstream bundles. The local, edited +version of the bundle overrides the bundle version found upstream. + +For example, to edit the `bundle1` definition file, we use the following +command: + +.. code-block:: bash + + mixer bundle edit bundle1 + +If `bundle1` is found in your local bundles, mixer edits this bundle +definition file. If instead `bundle1` is only found upstream, mixer copies +the bundle definition file from upstream into your :file:`local-bundles` +directory first. + +In both cases, mixer launches your default editor to edit the file. When the +editor closes, mixer automatically validates the edited bundle file and +reports any errors found. If mixer finds an error, you can edit the file +as-is, revert and edit, or skip and move on to the next bundle. If you skip a +file, mixer saves a backup of the original file with the ``.orig`` suffix. +Because mixer always checks your local bundles first, edited copies of an +upstream bundle always take precedence over their upstream counterpart. You +can edit multiple bundles with the following command: + +.. code-block:: bash + + mixer bundle edit bundle1 bundle2 [bundle3 ...] + +Create bundles for the mix +========================== + +To create a totally **new bundle**, the bundle name you specify cannot exist +upstream. If that is the case, create a :file:`new-bundle` with the following +command: + +.. code-block:: bash + + mixer bundle edit new-bundle + +This command generates a blank template in :file:`local-bundles` with the +:file:`new-bundle` filename. Mixer launches the editor for you to fill out +the bundle and performs validation when you exit the editor. Add your package +or packages to the bundle definition file to define the packages to install +as part of the bundle. + +.. note:: + + The :command:`mixer bundle edit` command accepts multiple bundles at once. + Thus, you can create multiple new bundles in a single command, for example: + + .. code-block:: bash + + mixer bundle edit new-bundle1 new-bundle2 [new-bundle3 ...] + +Add bundles to the mix +====================== + +Add `bundle1` to your mix with the following command: + +.. code-block:: bash + + mixer bundle add bundle1 + +This command adds the specified bundles to the mix bundles list stored in +your :file:`mixbundles` file. For each bundle you add, mixer checks your +local and upstream bundles to ensure the added bundles actually exist. If +mixer cannot find the bundle, it reports back an error. Additionally, when +mixer adds a bundle, it tells you whether the bundle is local or upstream. +Alternatively, you can learn this information with the +:command:`mixer bundle list` command. Refer to `List the bundles in the mix`_. + +To add multiple bundles at once, use the following command: + +.. code-block:: bash + + mixer bundle add bundle1 bundle2 [bundle3 ...] + +Remove bundles from the mix +=========================== + +Remove `bundle1` from your mix with the following command: + +.. code-block:: bash + + mixer bundle remove bundle1 + +This command removes `bundle1` from the mix bundle list stored in your +:file:`mixbundles` file. By default, this command does not remove the bundle +definition file from your local bundles. To completely remove a bundle, +including its local bundle definition file, use the following command with +the :option:`--local` flag: + +.. code-block:: bash + + mixer bundle remove --local bundle1 + +By default, removing a local bundle file with this command removes the bundle +from the mix as well. To only remove the local bundle definition file, use +the following command with the :option:`--mix=false` flag: + +.. code-block:: bash + + mixer bundle remove --local --mix=false bundle1 + +If you remove a local, edited version of an upstream bundle and keep the +bundle in the mix, the mix then references the original upstream version of +the bundle. + +On the other hand, if you remove a bundle that is only found locally but +keep the bundle in the mix bundles list, mixer will not find a valid +bundle definition file and will produce an error. + +Validate the bundles in the mix +=============================== + +Mixer performs basic validation on all bundles when used throughout the +system. + +Mixer checks the validity of the bundle's syntax and name. Optionally, you can +run this validation manually on `bundle1` with the following command: + +.. code-block:: bash + + mixer bundle validate bundle1 + +.. note:: This command can be useful in many circumstances. One example is + when importing already-existing local bundles from other projects. + +If you use the optional :option:`--strict` flag, the command additionally +checks if the rest of the bundle header fields can be parsed, if the bundle +header fields are non-empty, and if the bundle header ``Title`` field and +the bundle filename match. Perform a strict validation of `bundle1` with the +following command: + +.. code-block:: bash + + mixer bundle validate --strict bundle1 + +Validate multiple bundles with the following command: + +.. code-block:: bash + + mixer bundle validate bundle1 bundle2 [bundle3 ...] + +Managing bundles with Git +========================= + +If you initialized your workspace to be tracked as a git repository +with the :command:`mixer init --git` command, it might be useful to apply a +git commit after you modify the mix bundle list or edit a bundle definition +file. + +All the :command:`mixer bundle` commands in the previous sections support an +optional :option:`--git` flag. This flag automatically applies a git commit +when the command completes, for example: + +.. code-block:: bash + + mixer bundle remove --git bundle1 + +Build the bundle chroots +************************ + +To build all the ``chroots`` based on the defined bundles, use the following +command in your workspace: + +.. code-block:: bash + + sudo mixer build chroots + +If the mix has many bundles, this step might take some time. + +By default, mixer uses the legacy chroot-builder. In this mode, mixer +automatically gathers the bundle definition files for the bundles in the mix +into a :file:`mix-bundles` directory. The directory's path is set in the +`BUNDLE_DIR` variable in the :file:`builder.conf`. **Do not edit these +files.** Mixer automatically deletes the contents of the :file:`mix-bundles` +directory before repopulating the directory on-the-fly as mixer builds the +chroots. + +We have added a new chroot-builder to the mixer tool itself. While this is +currently an experimental feature, you should use the new chroot-builder. To +use the new chroot-builder, use the following command with the +:option:`--new-chroots` flag: + +.. code-block:: bash + + sudo mixer build chroots --new-chroots + +We will soon deprecate the legacy chroot-builder. When we do, mixer will use +the new version automatically. + +Create an update +**************** + +Create an update with the following command: + +.. code-block:: bash + + sudo mixer build update + +When the build completes, you can find the mix update content under +:file:`/home/clr/mix/update/www/VER`. In our example, the update content is +found in :file:`/home/clr/mix/update/www/{}`. `` +is the defined mix version, which is 10 by default. + +By default, mixer uses the legacy `swupd-server` to generate the update +content. However, we have built a new implementation into the mixer tool +itself. While this is currently an experimental feature, you should use the +new `swupd-server`. To use the the new `swupd-server`, use the following +command with the :option:`--new-swupd` flag: + +.. code-block:: bash + + sudo mixer build update --new-swupd + +We will soon deprecate the legacy `swupd-server`. When we do, mixer will use +the new version automatically. + +Mixer creates all the content needed to make a fully usable mix with this +step. However, only *zero packs* are automatically generated. Zero packs are +the content needed to go from nothing to the mix version for which you just +built the content. + +You can create optional *delta packs*, which allow the transition from one +mix version to another, with the following command: + +.. code-block:: bash + + sudo mixer-pack-maker.sh --to --from -S /home/clr/mix/update + +The pack-maker generates all delta packs for the bundles changed from +`PAST_VERSION` to `MIX_VERSION`. If your `STATE_DIR` is in a different +location, specify the location with the :option:`-S` flag. Mixer cannot +create delta packs for the first build because the update is from version 0. +Version 0 implicitly has no content. Thus, mixer can generate no deltas. + +For subsequent builds, you can run :file:`mixer-pack-maker.sh` to generate +delta content between them, for example: 10 to 20. + +Create an image +***************** + +Since mixer uses the `ister` tool to create a bootable image from your +updated content, we must first configure the `ister` tool. To configure the +image `ister` creates, we need the `ister` configuration file. Obtain a copy +with the default values from the `ister` package with the following command: + +.. code-block:: bash + + sudo cp /usr/share/defaults/ister/ister.json relase-image-config.json + +For reference, you can inspect the `Clear Linux ister configuration file`_ +used for releases. + +Edit the configuration file to include all bundles you want *preinstalled* in +the image. Users can install the bundles in the mix that are not included in +the configuration file with the following command: + +.. code-block:: bash + + sudo swupd bundle-add bundle1 + +Keeping the list of bundles in the configuration file small allows for a +smaller image size. For the minimal base image, the list is: + +.. code-block:: console + + "Bundles": ["os-core", "os-core-update", "kernel-native"] + +Next, set the `Version` field to the mix version that you want the content +mixer to use to build the image. `ister` allows you to build an image from +any mix version that you have built, not just from the current version. In +our example so far, `Version` is set to 10. + +With the `ister` tool configured, build the image with the following command: + +.. code-block:: bash + + sudo mixer build image --format 1 + +Mixer automatically looks for the :file:`release-image-config.json` file, but +you can freely choose the filename. To use a different name, simply pass the +:option:`--template` flag when creating your image, for example: + +.. code-block:: bash + + sudo mixer build image --format 1 --template path/to/file.config + +By default, `ister` uses the format version of the build machine it runs on. +Therefore, if the format you are building differs from the format of the |CL| +OS you are building on, you must use the :option:`--format ` +flag. Find the current format version of your OS with the following command: + +.. code-block:: bash + + sudo cat /usr/share/defaults/swupd/format + +Update the next mix version information +*************************************** + +Increment the mix version number for the next mix with the following command: + +.. code-block:: bash + + mixer versions update + +This command automatically updates the mix version stored in the +:file:`mixversion` file, incrementing it by 10. To increment by a different +amount, use the :option:`--increment` flag, for example: + +.. code-block:: bash + + mixer versions update --increment 100 + +Alternatively, to set the mix version to a specific value, use the +:option:`--mix-version` flag, for example: + +.. code-block:: bash + + mixer versions update --mix-version 200 + +The :command:`mixer versions update` command does not allow you to set the +mix version to a value less than its current value. The mix version is +expected to always increase, even if the new mix is undoing an earlier +change. + +If you have been tracking your workspace with git, you can restore the mix to +an earlier state. However, be careful when "rewriting history" if you have +published the mix content to users already. + +Use the following command with the the :option:`--upstream-version` flag to +update the upstream version of |CL| used as a base for the mix: + +.. code-block:: bash + + mixer versions update --upstream-version 21070 + +This command also accepts the keyword "latest": + +.. code-block:: bash + + mixer versions update --upstream-version latest + +This command sets the upstream version to the latest released version of +upstream |CL| within the same format version. The +:command:`mixer versions update` command does not allow you to set an +upstream version to a value that crosses an upstream format boundary. +Such values require a "format bump" build, which is currently a +manual process. Refer to :ref:`mixer-format` for more information. + +Optionally, you can learn which mix version or upstream version you are +currently using with the following command: + +.. code-block:: bash + + mixer versions + +At this point, you can continue to iterate through the workflows and make +modifications as needed, for example: + +#. Add, remove, or modify bundles. +#. Build the chroots with: + + .. code-block:: bash + + sudo mixer build chroots + +#. Build and update with: + + .. code-block:: bash + + sudo mixer build update + +#. Optionally, you can create delta packs with: + + .. code-block:: bash + + sudo mixer-pack-maker.sh --to --from -S /home/clr/mix/update + +.. _mixer-format: + +Format version +************** + +The `Format` variable set in the :file:`builder.conf` file can be more +precisely referred to as an OS *compatibility epoch*. Versions of the OS +within a given epoch are fully compatible and can update to any other +version within that epoch. Across the `Format` boundary, the OS has changed +in such a way that updating from build M in format X, to build N in format Y +will not work. Generally, this scenario occurs when the software updater or +software manifests change in a way that they are no longer compatible with +the previous update scheme. + +Using a format increment, we insure pre- and co-requisite changes flow out +with proper ordering. The updated client only ever updates to the latest +release in its respective format version, unless overridden by command line +flags. Thus, we can guarantee that all clients update to the final version +in their given format. The given format *must* contain all the changes +needed to understand the content built in the subsequent format. Only after +reaching the final release in the old format can a client continue to update +to releases in the new format. + +When creating a custom mix, the format version should start at "1" or some +known number. The format version should increment only when a compatibility +breakage is introduced. Normal updates, like updating a software package for +example, do not require a format increment. + +.. _update page: https://cdn.download.clearlinux.org/update/ + +.. _format bumps wiki: https://github.com/clearlinux/swupd-server/wiki/Format-Bumps + +.. _build RPMs instructions: https://github.com/clearlinux/common#build-rpms-for-a-package + +.. _Clear Linux ister configuration file: + https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json diff --git a/source/clear-linux/guides/maintenance/swupdaddpkg.rst b/source/clear-linux/guides/maintenance/swupdaddpkg.rst new file mode 100644 index 00000000..f0607033 --- /dev/null +++ b/source/clear-linux/guides/maintenance/swupdaddpkg.rst @@ -0,0 +1,148 @@ +.. _swupdaddpkg: + +Create and add custom bundles to your upstream Clear Linux system +################################################################# + +|CLOSIA| offers many curated bundles that you can install on your system to +create your desired capabilities. If the available upstream bundles do not +meet your needs, you can create and add your own custom bundles to your +system using one of two methods. Note: Upstream refers to the official +version of |CL|. + +The first method is to use the :ref:`mixer tool` to create your own +|CL| image and add your bundles to it. Mixing your own |CL| image can +give you great control and flexibility; however, you must act as an +:abbr:`OSV (Operating System Vendor)` and maintain your releases and +updates because you have forked from upstream. + +The second method is to use the :command:`swupd-add-pkg` tool, which also +makes use of mixer to create custom bundles that you can add to your +upstream |CL| system. This simpler method provides a “light” forking from +upstream, which means you can continue to get upstream bundles and updates. +If needed, you can easily revert your system back to the upstream version. + +This guide shows you how to accomplish the second method by following these +steps: + +#. Set up the workspace. +#. Copy your custom RPM package to the workspace. +#. Create a bundle with your custom RPM package. +#. Migrate your |CL| system to your custom mix. +#. Add your custom bundle to your system. +#. Optional: Revert your system back to 100% upstream. + +Set up the workspace +******************** + +#. Install the mixer bundle to enable mixer. + + .. code-block:: console + + $ sudo swupd bundle-add mixer + +#. Create the workspace. + + .. code-block:: console + + $ sudo mkdir -p /usr/share/mix/rpms + +Copy your custom RPM package to the workspace +********************************************* + +.. note:: + + You cannot simply use RPMs from other Linux distros on |CL|. You must + build RPMs specifically for |CL| in order for them to work properly. + Follow the instructions on how to build RPMs found at the + `Developer tooling framework for Clear Linux`_. + +Copy your RPM package to the workspace. + +.. code-block:: console + + $ sudo cp [RPM] /usr/share/mix/rpms + +Create a bundle with your custom RPM package +******************************************** + +Use the :command:`swupd-add-pkg` command to create a bundle with the RPM +package. + +.. code-block:: console + + $ sudo swupd-add-pkg [RPM] [bundle-name] + +To add more than one RPM to your previously-created bundle, repeat +the :command:`swupd-add-pkg` command and change the RPM name. + +.. note:: + + * If you add the same RPM package more than once, it will simply build a + new mix each time without appending it again to the bundle definition. + + * The first time you run the :command:`swupd-add-pkg` command, mixer + creates a new OS version by taking your current upstream |CL| version + and multiplying it by 1000. For example, if your upstream version is + 21530, your custom version will be 21530000. For each subsequent call + to swupd-add-pkg, mixer will increment the version by 10. For example, + 21530010, 21530020, etc. + +Migrate your Clear Linux system to your custom mix +************************************************** + +Before you can use your custom bundle, you must migrate your |CL| system +to your custom mix to make the bundle accessible. + +.. code-block:: console + + $ sudo swupd update --migrate + +After you migrate, the version of your |CL| system switches over to your +last custom version number as noted in the previous section. + +You can continue to create new bundles with :command:`swupd-add-pkg` +while you are in your custom version of |CL|. You do not need to migrate +again. However, you must run :command:`swupd update` again to update your +system in order to make those bundles visible. + +Add your custom bundle to your system +************************************* + +#. Get a listing of your newly-created bundle. + + .. code-block:: console + + $ sudo swupd bundle-list -a + + The listing includes all upstream bundles. + +#. Add your bundle. + + .. code-block:: console + + $ sudo swupd bundle-add [bundle-name] + +.. note:: + + You can also update your system to the latest upstream version using + this command: + + .. code-block:: console + + $ sudo swupd update + +Optional: Revert your system back to 100% upstream +************************************************** + +If you want to revert your |CL| system back to the official upstream +version, use this command: + +.. code-block:: console + + $ sudo swupd verify --fix --force --picky -m [upstream-version-number] -C /usr/share/clear/update-ca/Swupd_Root.pem + +After the command completes, all custom RPMs and bundles are unavailable +because :file:`/usr/share/mix` is deleted as part of the reversion process. + +.. _Developer tooling framework for Clear Linux: + https://github.com/clearlinux/common diff --git a/source/clear-linux/guides/network/custom-clear-container.rst b/source/clear-linux/guides/network/custom-clear-container.rst index 3981563f..5615a1d8 100644 --- a/source/clear-linux/guides/network/custom-clear-container.rst +++ b/source/clear-linux/guides/network/custom-clear-container.rst @@ -3,41 +3,40 @@ Build a custom Clear Linux based Docker container image ####################################################### -The official base |CLOSIA| container image is published on Docker\* Hub and -is updated on a regular basis. This section contains the steps to build a -custom image. +The official base |CLOSIA| container image is published on Docker\* Hub and is +updated on a regular basis. This guide contains the steps to build a custom +container image. Prerequisites ************* -* These steps must be performed on a |CL| system because the `swupd` command - is needed to manage bundles in the container. -* The `containers-basic` bundle must be installed on the |CL| system for - Docker to work. -* Basic knowledge of Docker is required. +* You must perform these steps on a |CL| system because the + :abbr:`swupd (software updater)` is used to manage bundles in the + container. +* You must install the :file:`containers-basic` bundle on the |CL| system + or Docker will not work. +* You have a basic understanding of Docker. -Build the base Clear Linux container image -****************************************** +Build the base container image +****************************** #. Log in and get root privileges. - .. code-block:: console + .. code-block:: bash - $ sudo -s + sudo -s #. Verify Docker is installed and running. - .. code-block:: console + .. code-block:: bash - # docker info + docker info - If Docker is installed and running, the expected output will be similar to - this: + If Docker is installed and running, the output is similar to + this example: .. code-block:: console - # docker info - Containers: 0 Running: 0 Paused: 0 @@ -76,135 +75,135 @@ Build the base Clear Linux container image 127.0.0.0/8 Live Restore Enabled: false - If Docker is not installed, perform these steps: + If Docker is not installed, enter the commands: - .. code-block:: console + .. code-block:: bash - # swupd bundle-add containers-basic - # systemctl start docker + swupd bundle-add containers-basic + systemctl start docker -#. Create the directory structure needed for building the |CL| container. +#. Create the directory structure to build the |CL| container. - .. code-block:: console + .. code-block:: bash - # mkdir -p ./custom-clear-linux-container/base/usr/share/clear/bundles - # cd custom-clear-linux-container + mkdir -p ./custom-clear-linux-container/base/usr/share/clear/bundles + cd custom-clear-linux-container .. note:: - * The directories `customer-clear-linux-container` and `base` are for - the purpose of staging and can be named something else, if preferred. - * The remaining directories (`/usr/share/clear/bundles`) are mandatory. + * The directories :file:`custom-clear-linux-container` and + :file:`base` are used for staging. You can rename these directories. -#. Create the reference files of the minimum required |CL| bundles (`os-core` and - `os-core-update`). `swupd` determines which bundles to download and install - by using the reference filenames. + * The directories :file:`/usr/share/clear/bundles` are mandatory and + cannot be renamed. - .. code-block:: console +#. Create the reference files of the minimum required |CL| bundles, + :file:`os-core` and :file:`os-core-update`. The software updater + uses the reference filenames to determine which bundles to download and + install. - # touch ./base/usr/share/clear/bundles/os-core - # touch ./base/usr/share/clear/bundles/os-core-update - .. note:: + .. code-block:: bash - * `os-core` provides the minimal Linux namespace. - * `os-core-update` provides basic suite for running the |CL| - for iA Updater + touch ./base/usr/share/clear/bundles/os-core + touch ./base/usr/share/clear/bundles/os-core-update -#. Optionally, additional bundles can be included with the base image. + .. note:: - #. Identify the desired bundles by going to the |CL| website's - :ref:`available-bundles` page or by executing the - `swupd bundle-list -a` command + * :file:`os-core` provides the minimal Linux namespace. + * :file:`os-core-update` provides the basic suite for running the |CLOSIA| + updater. - #. Create the reference files for the identified bundles. For example, - to include the `editors` and `network-basic` bundles: +#. Optionally, you can include additional bundles with the base image. - .. code-block:: console + #. Identify the desired bundles on the |CL| website's + :ref:`available-bundles` page or execute the + :command:`swupd bundle-list -a` command. - # touch ./base/usr/share/clear/bundles/editors - # touch ./base/usr/share/clear/bundles/network-basic + #. Create reference files for the identified bundles. For example, + to include the :file:`editors` and :file:`network-basic` bundles, + enter the commands: -#. Use `swupd` to download and install the bundles into the directory - structure created. + .. code-block:: bash - .. code-block:: console + touch ./base/usr/share/clear/bundles/editors + touch ./base/usr/share/clear/bundles/network-basic - # swupd verify --install --path="base" --manifest 17870 \ +#. Use `swupd` to download and install the bundles. + + .. code-block:: bash + + swupd verify --install --path="base" --manifest 17870 \ --url https://cdn.download.clearlinux.org/update \ --statedir "$PWD/swupd-state" --no-boot-update - .. note:: - * `verify –-install` tells `swupd` to download and install - * `–-path` specifies the root path of where the bundles are to be - installed - * `--manifest` specifies the version of the |CL| bundles to use - * `--url` specifies the URL of the bundles repository - * `--statedir` specifies the state directory where downloaded bundles - and any - state information are stored - * `--no-boot-update` tells `swupd` to skip updating boot files since - it's not needed for a container + The `swupd` example uses the following flags: - For more information on the `swupd` flags, enter the `swupd verify -h` + * :command:`verify –-install` tells `swupd` to download and install. + * :command:`--path` specifies the root path where the bundles are to be + installed. + * :command:`--manifest` specifies the version of the |CL| bundles. + * :command:`--url` specifies the URL of the bundles repository. + * :command:`--statedir` specifies the state directory where downloaded bundles + and any state information are stored. + * :command:`--no-boot-update` tells `swupd` to skip updating boot files because + boot files are not required for a container. + + For more information on `swupd` flags, enter the :command:`swupd verify -h` command. Example output: - .. code-block:: console + .. code-block:: console - # swupd verify --install --path="base" --manifest 17870 \ - --url https://cdn.download.clearlinux.org/update \ - --statedir "$PWD/swupd-state" --no-boot-update + swupd-client software verify 3.12.2 + Copyright (C) 2012-2017 Intel Corporation - swupd-client software verify 3.12.2 - Copyright (C) 2012-2017 Intel Corporation + Verifying version 17870 + Attempting to download version string to memory + Downloading packs... - Verifying version 17870 - Attempting to download version string to memory - Downloading packs... - - Extracting python-basic pack for version 17820 - ...14% - Extracting perl-basic pack for version 17790 - ...28% - Extracting openssh-server pack for version 17660 - ...42% - Extracting editors pack for version 17850 - ...57% - Extracting network-basic pack for version 17650 - ...71% - Extracting os-core pack for version 17870 - ...85% - Extracting os-core-update pack for version 17870 - ...100% - Adding any missing files - ...88% - Inspected 33982 files - 33974 files were missing - 33974 of 33974 missing files were replaced - 0 of 33974 missing files were not replaced - Calling post-update helper scripts. - WARNING: boot files update skipped due to --no-boot-update argument - Fix successful + Extracting python-basic pack for version 17820 + ...14% + Extracting perl-basic pack for version 17790 + ...28% + Extracting openssh-server pack for version 17660 + ...42% + Extracting editors pack for version 17850 + ...57% + Extracting network-basic pack for version 17650 + ...71% + Extracting os-core pack for version 17870 + ...85% + Extracting os-core-update pack for version 17870 + ...100% + Adding any missing files + ...88% + Inspected 33982 files + 33974 files were missing + 33974 of 33974 missing files were replaced + 0 of 33974 missing files were not replaced + Calling post-update helper scripts. + WARNING: boot files update skipped due to --no-boot-update argument + Fix successful .. note:: The `WARNING` message is expected and can be ignored. -#. Tar up the files and compress it. +#. Create a tarball and compress it. - .. code-block:: console + .. code-block:: bash - # tar -C base -cf base.tar . - # xz -v -T0 base.tar + tar -C base -cf base.tar . + xz -v -T0 base.tar #. Create the Dockerfile to build the image. - .. code-block:: console + .. code-block:: bash - # cat > Dockerfile << EOF + cat > Dockerfile << EOF FROM scratch MAINTAINER First Last ADD base.tar.xz / @@ -213,169 +212,167 @@ Build the base Clear Linux container image #. Build the |CL| container image. - .. code-block:: console + .. code-block:: bash - # docker build -t my-custom-clear-linux-container . + docker build -t my-custom-clear-linux-container . Example output: - .. code-block:: console + .. code-block:: console - # docker build -t my-custom-clear-linux-container . - - Sending build context to Docker daemon 806.5MB - Step 1/4 : FROM scratch - ---> - Step 2/4 : MAINTAINER First Last - ---> Running in 7238f35abcd0 - ---> ec5064287c60 - Removing intermediate container 7238f35abcd0 - Step 3/4 : ADD base.tar.xz / - ---> 2723b7d20716 - Removing intermediate container 16e3ed0df8da - Step 4/4 : CMD /bin/bash - ---> Running in efa893350647 - ---> 5414c3a12993 - Removing intermediate container efa893350647 - Successfully built 5414c3a12993 - Successfully tagged my-custom-clear-linux-container:latest + Sending build context to Docker daemon 806.5MB + Step 1/4 : FROM scratch + ---> + Step 2/4 : MAINTAINER First Last + ---> Running in 7238f35abcd0 + ---> ec5064287c60 + Removing intermediate container 7238f35abcd0 + Step 3/4 : ADD base.tar.xz / + ---> 2723b7d20716 + Removing intermediate container 16e3ed0df8da + Step 4/4 : CMD /bin/bash + ---> Running in efa893350647 + ---> 5414c3a12993 + Removing intermediate container efa893350647 + Successfully built 5414c3a12993 + Successfully tagged my-custom-clear-linux-container:latest #. List the newly created |CL| container image. - .. code-block:: console + .. code-block:: bash - # docker images + docker images Example output: - .. code-block:: console + .. code-block:: console - # docker images - - REPOSITORY TAG IMAGE ID CREATED SIZE - my-custom-clear-linux-container latest 5414c3a12993 About a minute ago 616MB + REPOSITORY TAG IMAGE ID CREATED SIZE + my-custom-clear-linux-container latest 5414c3a12993 About a minute ago 616MB #. Launch the built |CL| container. - .. code-block:: console + .. code-block:: bash - # docker run -it my-custom-clear-linux-container + docker run -it my-custom-clear-linux-container -Manage bundles in a Clear Linux based container -*********************************************** +Manage bundles in a container +***************************** -Bundles can be added and removed from an existing |CL| container by using -the `swupd` command in the Dockerfile. +You can add and remove bundles from a |CL| container using the +:command:`RUN swupd` command in the Dockerfile. -Add a bundle (`swupd bundle-add`) ---------------------------------- +Add a bundle +============ -This example Dockerfile shows how to add the `pxe-server` bundle to the -previously created |CL| Docker image: +This example Dockerfile adds the :file:`pxe-server` bundle to an existing |CL| +Docker image: - .. code-block:: console +.. code-block:: bash - # cat > Dockerfile << EOF - FROM my-customer-clear-linux-container - MAINTAINER First Last - RUN swupd bundle-add pxe-server - CMD ["/bin/bash/bash"] - EOF + cat > Dockerfile << EOF + FROM my-customer-clear-linux-container + MAINTAINER First Last + RUN swupd bundle-add pxe-server + CMD ["/bin/bash/bash"] + EOF - Example output: +Example output: - .. code-block:: console +.. code-block:: console - # docker build -t my-clearlinux-with-pxe-server-bundle . + docker build -t my-clearlinux-with-pxe-server-bundle . - Sending build context to Docker daemon 806.5MB - Step 1/4 : FROM my-custom-clear-linux-container - ---> 5414c3a12993 - Step 2/4 : MAINTAINER First Last - ---> Running in 19b4411cf4bd - ---> 08d400baffde - Removing intermediate container 19b4411cf4bd - Step 3/4 : RUN swupd bundle-add pxe-server - ---> Running in 3e634d6e0792 - swupd-client bundle adder 3.12.2 - Copyright (C) 2012-2017 Intel Corporation + Sending build context to Docker daemon 806.5MB + Step 1/4 : FROM my-custom-clear-linux-container + ---> 5414c3a12993 + Step 2/4 : MAINTAINER First Last + ---> Running in 19b4411cf4bd + ---> 08d400baffde + Removing intermediate container 19b4411cf4bd + Step 3/4 : RUN swupd bundle-add pxe-server + ---> Running in 3e634d6e0792 + swupd-client bundle adder 3.12.2 + Copyright (C) 2012-2017 Intel Corporation - Attempting to download version string to memory - Downloading packs... + Attempting to download version string to memory + Downloading packs... - Extracting pxe-server pack for version 17820 - . - Installing bundle(s) files... - .............................................................................. - .............................................................................. - .............................................................................. - .............................................................................. - .............................................................................. - .............................................................................. - Calling post-update helper scripts. - WARNING: systemctl not operable, unable to run systemd update triggers - Bundle(s) installation done. - ---> 8ead5f2c0c33 - Removing intermediate container 3e634d6e0792 - Step 4/4 : CMD /bin/bash - ---> Running in 0ceae320279b - ---> dcd9adb40611 - Removing intermediate container 0ceae320279b - Successfully built dcd9adb40611 - Successfully tagged my-clearlinux-with-pxe-server-bundle:latest + Extracting pxe-server pack for version 17820 + . + Installing bundle(s) files... + .............................................................................. + .............................................................................. + .............................................................................. + .............................................................................. + .............................................................................. + .............................................................................. + Calling post-update helper scripts. + WARNING: systemctl not operable, unable to run systemd update triggers + Bundle(s) installation done. + ---> 8ead5f2c0c33 + Removing intermediate container 3e634d6e0792 + Step 4/4 : CMD /bin/bash + ---> Running in 0ceae320279b + ---> dcd9adb40611 + Removing intermediate container 0ceae320279b + Successfully built dcd9adb40611 + Successfully tagged my-clearlinux-with-pxe-server-bundle:latest - .. note:: +.. note:: - This `WARNING` message is expected and can be ignored because Systemd - doesn't run inside a container. + The `WARNING` message can be ignored because systemd does not run inside + a container. -Remove a bundle (`swupd bundle-remove`) ---------------------------------------- -This example Dockerfile shows how to remove the `pxe-server` bundle from the -previously created |CL| Docker image: +Remove a bundle +=============== - .. code-block:: console +This example Dockerfile removes the :file:`pxe-server` bundle from an existing +|CL| Docker image: - # cat > Dockerfile << EOF - FROM my-clearlinux-with-pxe-server-bundle - MAINTAINER First Last - RUN swupd bundle-remove pxe-server - CMD ["/bin/bash/bash"] - EOF +.. code-block:: bash - Example output: + cat > Dockerfile << EOF + FROM my-clearlinux-with-pxe-server-bundle + MAINTAINER First Last + RUN swupd bundle-remove pxe-server + CMD ["/bin/bash/bash"] + EOF - .. code-block:: console +Example output: - # docker build -t my-clearlinux-remove-pxe-server-bundle . +.. code-block:: console - Sending build context to Docker daemon 806.5MB - Step 1/4 : FROM my-clearlinux-with-pxe-server-bundle - ---> dcd9adb40611 - Step 2/4 : MAINTAINER First Last - ---> Running in 71b60f15003e - ---> 742192751c1a - Removing intermediate container 71b60f15003e - Step 3/4 : RUN swupd bundle-remove pxe-server - ---> Running in ad28a3390ecc - swupd-client bundle remover 3.12.2 - Copyright (C) 2012-2017 Intel Corporation + docker build -t my-clearlinux-remove-pxe-server-bundle . - Removing bundle: pxe-server - Deleting bundle files... - Total deleted files: 92 - Untracking bundle from system... - Success: Bundle removed - 1 bundle(s) were removed successfully - ---> d6ee7903e14d - Removing intermediate container ad28a3390ecc - Step 4/4 : CMD /bin/bash - ---> Running in 7694989e97de - ---> ec23189ef954 - Removing intermediate container 7694989e97de - Successfully built ec23189ef954 - Successfully tagged my-clearlinux-remove-pxe-server-bundle:latest + Sending build context to Docker daemon 806.5MB + Step 1/4 : FROM my-clearlinux-with-pxe-server-bundle + ---> dcd9adb40611 + Step 2/4 : MAINTAINER First Last + ---> Running in 71b60f15003e + ---> 742192751c1a + Removing intermediate container 71b60f15003e + Step 3/4 : RUN swupd bundle-remove pxe-server + ---> Running in ad28a3390ecc + swupd-client bundle remover 3.12.2 + Copyright (C) 2012-2017 Intel Corporation -Also see: + Removing bundle: pxe-server + Deleting bundle files... + Total deleted files: 92 + Untracking bundle from system... + Success: Bundle removed + 1 bundle(s) were removed successfully + ---> d6ee7903e14d + Removing intermediate container ad28a3390ecc + Step 4/4 : CMD /bin/bash + ---> Running in 7694989e97de + ---> ec23189ef954 + Removing intermediate container 7694989e97de + Successfully built ec23189ef954 + Successfully tagged my-clearlinux-remove-pxe-server-bundle:latest - * :ref:`cc-getting-started` +For more details, refer to: + +* :ref:`cc-getting-started` +* :ref:`architecture-overview` diff --git a/source/clear-linux/guides/network/dpdk.rst b/source/clear-linux/guides/network/dpdk.rst index 9c339ced..2f8d4127 100644 --- a/source/clear-linux/guides/network/dpdk.rst +++ b/source/clear-linux/guides/network/dpdk.rst @@ -1,337 +1,347 @@ .. _dpdk: -Send packages between platforms -############################### +Use DPDK to send packets between platforms +########################################## -:abbr:`Data Plane Development Kit (DPDK)` is a set of libraries and drivers -for fast packet processing. This document describes how to run a basic use -case for **l3fwd DPDK example**. The objective is to *send packages between -two platforms* using a traffic generator called :ref:`pktgen `, -where the l3fwd example application will forward those packages, see -:ref:`figure 1 `. +This document describes how to send packets between two platforms in the +simple configuration shown in :ref:`Figure 1 `. The example uses the +:abbr:`Data Plane Development Kit (DPDK)`, which is a set of libraries, +drivers, sample applications, and tools for fast packet processing. .. _f1: .. figure:: ./figures/pktgen_lw3fd.png :align: center - :alt: platform A and B + :alt: Platform A and B - Figure 1: Environment for l3fwd DPDK application. + Figure 1: Environment for l3fwd DPDK application + This example uses the following DPDK components: -**Requirements:** +* pktgen: Traffic generator. See `pktgen documentation`_ for details. +* l3fwd: Layer 3 forwarding example application. See + `l3fwd documentation`_ for details. -* Two platforms using Clear Linux* for Intel® Architecture (recommended - release `13330`_ or higher). -* Both images have the **kernel-native bundle** added. -* Install of **network-basic-dev** bundle: +Prerequisites +************* - .. code-block:: bash +* Two platforms using |CLOSIA| release `13330`_ or higher. +* Both images must include the :file:`kernel-native bundle`. +* Install the :file:`network-basic-dev` bundle with the command: - # swupd bundle-add network-basic-dev + .. code-block:: bash + + sudo swupd bundle-add network-basic-dev + +* Each platform must have at least one :abbr:`NIC (Network Interface Card)`. + Check the `DPDK project`_ for the list of supported `dpdk.org NICs`_. -* The platforms must have two NICs, at least one each. It's very important to - check network card compatibility with the `DPDK project`_. You can do this - on the `dpdk.org NICS`_ site. * Two network cables. -Installing dpdk and build l3fwd example (Platform B) -==================================================== +Install dpdk and build l3fwd example (Platform B) +************************************************* -#. Move to ``l3fwd`` example. +#. Change to the :file:`l3fwd` example directory. .. code-block:: bash - # cd /usr/share/dpdk/examples/l3fwd + sudo cd /usr/share/dpdk/examples/l3fwd -#. Assign ``RTE_SDK var`` to the makefiles path. +#. Assign :envvar:`RTE_SDK` variable to the makefiles path. .. code-block:: bash - # export RTE_SDK=/usr/share/dpdk/ + sudo export RTE_SDK=/usr/share/dpdk/ -#. Assign ``RTE_TARGET var`` the value where the gcc config file is located. +#. Assign :envvar:`RTE_TARGET` variable to the location of the gcc\* config + file. .. code-block:: bash - # export RTE_TARGET=x86_64-native-linuxapp-gcc + sudo export RTE_TARGET=x86_64-native-linuxapp-gcc -#. Build the ``l3fwd`` application, then add the configuration header to - the ``CFLAGS`` var. +#. Build the `l3fwd` application and add the configuration header to + the :makevar:`CFLAGS` variable. .. code-block:: bash - # make CFLAGS+="-include /usr/include/rte_config.h" + sudo make CFLAGS+="-include /usr/include/rte_config.h" +Build pktgen (Platform A) +************************* -.. _sec_pktgen: - -Building Pktgen (Platform A) -============================ - -Since the **pktgen** project is currently not included in Clear Linux OS for -Intel Architecture, you must download it from upstream and build it: - -#. Download the `pktgen tar package`_ 3.1.2 or newer. +#. Download the `pktgen tar package`_ v3.1.2 or newer. #. Decompress packages and move to uncompressed source directory. -#. Assign ``RTE_SDK var`` the path where makefiles are located. +#. Assign :envvar:`RTE_SDK` variable to the path where makefiles are located. .. code-block:: bash - # export RTE_SDK=/usr/share/dpdk/ + sudo export RTE_SDK=/usr/share/dpdk/ -#. Assign ``RTE_TARGET var`` the value where the gcc config file is located. +#. Assign :envvar:`RTE_TARGET` to the location of the gcc config file. .. code-block:: bash - # export RTE_TARGET=x86_64-native-linuxapp-gcc + sudo export RTE_TARGET=x86_64-native-linuxapp-gcc -#. Build pktgen project, and set the ``CONFIG_RTE_BUILD_SHARED_LIB`` variable - with "n". +#. Build the `pktgen` project and set the :makevar:`CONFIG_RTE_BUILD_SHARED_LIB` variable + to "n". .. code-block:: bash - # make CONFIG_RTE_BUILD_SHARED_LIB=n + sudo make CONFIG_RTE_BUILD_SHARED_LIB=n -Binding NICs to DPDK kernel drivers (Platforms A and B) -======================================================= +Bind NICs to DPDK kernel drivers (Platforms A and B) +**************************************************** -The ``l3fwd`` application uses two NICs. DPDK has useful tools for binding +The `l3fwd` application uses two NICs. The DPDK includes tools for binding NICs to DPDK modules to run DPDK applications. -#. Load the dpdk I/O kernel module +#. Load the DPDK I/O kernel module. .. code-block:: bash - # modprobe vfio-pci + sudo modprobe vfio-pci -#. Check the status of your NICs; this will show which network cards are not - busy. When another application is using them, the status shows ``Active``, +#. Check the NIC status to determine which network cards are not + busy. When another application is using them, the status shows `Active`, and those NICs cannot be bound. .. code-block:: bash - # dpdk-devbind --status + sudo dpdk-devbind --status -#. Bind two available NICs. The general syntax for binding is - **dpdk-devbind --bind=vfio-pci **, - and the following is a working example: +#. Bind two available NICs. The general syntax for binding is: + :command:`dpdk-devbind --bind=vfio-pci `. + A working example is shown below: .. code-block:: bash - # dpdk-devbind --bind=vfio-pci 01:00.0 + sudo dpdk-devbind --bind=vfio-pci 01:00.0 -#. Check that your NICs binded correctly by checking the status; ``drv`` should - have ``igb_uio`` value; at this point, the NICs are using the DPDK modules. +#. Check the NIC status to verify that the NICs are bound correctly. If + successful, `drv` displays the value `igb_uio`, which confirms + that the NICs are using the DPDK modules. -Setting hugepages (platforms A and B) -===================================== +Set hugepages (Platforms A and B) +********************************* -Clear Linux OS for Intel Architecture supports ``hugepages`` for the large -memory pool allocation used for packet buffers. +|CLOSIA| supports `hugepages` for the large memory pool allocation used for +packet buffers. -#. Set number of hugepages. +#. Set the number of hugepages. .. code-block:: bash - # echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + sudo echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages #. Allocate pages on NUMA machines. .. code-block:: bash - # echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - # echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages + sudo echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages + sudo echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages #. Make memory available for DPDK. .. code-block:: bash - # mkdir -p /mnt/huge $ mount -t hugetlbfs nodev /mnt/huge + sudo mkdir -p /mnt/huge $ mount -t hugetlbfs nodev /mnt/huge - If you would like to know more about this, refer to the `DPDK guide`_. + For more information, refer to the `DPDK guide`_ System Requirements + section. -Setting a physical environment (Platforms A and B) -================================================== +Set up the physical environment (Platforms A and B) +*************************************************** -To achieve the model proposed in the introduction of this topic, (:ref:`f1`), -we need to connect the first Grantley’s NICs to the second Grantley’s NICs -using the network cables, see :ref:`figure 2`. +Connect the NICs on Platform A to the NICs on Platform B using the network +cables as shown in :ref:`Figure 2`. .. _f2: .. figure:: ./figures/pyshical_net.png - Figure 2: Physical network environment. + Figure 2: Physical network environment -Running l3fwd application (Platform B) -====================================== +Run l3fwd application (Platform B) +********************************** -The ``l3fwd`` application is one of the DPDK examples available when you -install the ``dpdk-dev`` bundle; this application forwards packages from one -NIC to another. +The `l3fwd` application is one of the DPDK examples available when you +install the :file:`dpdk-dev` bundle. `l3fwd` forwards packets from one +NIC to another. For details, refer to the `l3fwd documentation`_. #. Open the l3fwd example directory. .. code-block:: bash - # cd /usr/share/dpdk/examples/l3fwd + sudo cd /usr/share/dpdk/examples/l3fwd -#. **This step is very important.** DPDK needs poll drivers for work; these - poll drivers are shared objects in :file:`/usr/lib64`. DPDK supports some - NICs. The full list available at the `dpdk.org NICS`_ docs. You should know - which kernel module the NIC is using and choose a poll driver according to - your NICs. +#. **This step is very important.** -#. At this point the system must have ``hugepages`` requirements. The NICs - bound and the configuration for running ``pktgen`` depends upon network use - cases and available system resources. Use the ``-d`` flag for setting the - pull driver. For example, if the NICs are using ``e1000`` network driver, - they are going to use ``e1000`` poll driver (``librte_pmd_e1000.so``); it - should be in :file:`/usr/lib64` in Clear Linux OS for Intel Architecture, - and it should be enough to add the name. For example + #. DPDK needs poll mode drivers to operate. + #. Poll mode drivers are shared objects in :file:`/usr/lib64`. + #. See the full list of supported NICs at `dpdk.org NICs`_. + #. You must know which kernel module each NIC is using and choose a poll + mode driver that corresponds to your NICs. + +#. NIC binding and `pktgen` configuration depends upon network use cases and + available system resources. Use the :command:`-d` flag to set the poll mode + driver. + + The following example assumes that the NICs use the `e1000` network driver + and the `e1000` poll mode driver. The :file:`librte_pmd_e1000.so` is + located in :file:`/usr/lib64` in |CL|. .. code-block:: bash - # ./build/l3fwd -c 0x3 -n 2 -d librte_pmd_e1000.so -- -p 0x3 --config="(0,0,0),(1,0,1)" + sudo ./build/l3fwd -c 0x3 -n 2 -d librte_pmd_e1000.so -- -p 0x3 --config="(0,0,0),(1,0,1)" -#. When the application starts to run, it will show information about the - ``l3fwd`` running, so pay attention when the application is Initializing - ports. After port 0 initialization, you'll see a MAC address and the same - for port 1. Save this information for setting configuration to `Pktgen` - project. +#. The `l3fwd` application shows port initialization details at startup. + After port 0 initialization completes, `l3fwd` shows a MAC address and + information for port 1. -Running Pktgen application (Platform A) -=========================================== + Save the MAC address for configuring the `pktgen` project. -The `Pktgen` is network traffic generator. It measures the network packaging -performance in a forwarding use case. +Run pktgen application (Platform A) +*********************************** -#. At this point the system must have ``hugepages`` requirements and the NICs - bound. The configuration for running ``pktgen`` depends upon the network use - case and the available system resources. The following is a basic +`pktgen` is a network traffic generator included in the DPDK. + +#. `pktgen` configuration depends upon the network setup and the + available system resources. The following example shows a basic configuration. .. code-block:: bash - # ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 0xf -n 4 -- -p 0xf -P -m "1.0, 2.1" + sudo ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 0xf -n 4 -- -p 0xf -P -m "1.0, 2.1" #. Enable active colorful output (optional). - .. code-block:: console + .. code-block:: bash Pktgen> theme enable -#. The ``l3fwd`` application showed a MAC address per-port initialized; this - MAC addresses should have been set in the pktgen environment:: +#. Use the MAC addresses shown by the `l3fwd` application during initialization. + The command to set the MAC addresses in `pktgen` has the format: - > set mac + .. code-block:: bash - And a working example: + set mac - .. code-block:: console + Here is a working example: + + .. code-block:: bash Pktgen> set mac 0 00:1E:67:CB:E8:C9 Pktgen> set mac 1 00:1E:67:CB:E8:C9 -#. Start to send packages using the next command: +#. Send packets. - .. code-block:: console + .. code-block:: bash Pktgen> start 0-1 -#. If you followed these steps correctly, you'll see that ``pktgen`` is sending - and receiving packages. For more information, see the `Pktgen - documentation`_. +For more details, see the `pktgen documentation`_. - -Annex A: Using pass-through for running on virtual machines -=========================================================== +Appendix A: Use pass-through for virtual machines +************************************************* This section explains how to set up a virtual environment where virtual -machines control the host's NICs. +machines control the NICs on the host. #. Create a new directory and move to it. -#. Download or create a ``start_qemu.sh`` script for running a kvm virtual +#. Download or create a :file:`start_qemu.sh` script for running a kvm virtual machine: .. code-block:: bash - $ curl -O https://download.clearlinux.org/image/start_qemu.sh + sudo curl -O https://download.clearlinux.org/image/start_qemu.sh -#. Download a bare-metal image of Clear Linux OS for Intel Architecture and - rename it as ``clear.img``. +#. Download a bare-metal image of |CLOSIA| and rename it as :file:`clear.img`. -#. Look for an entry for device and vendor & device ID: +#. Look for an Ethernet\* device entry that contains vendor and device ID: .. code-block:: bash - $ lspci -nn | grep Ethernet + sudo lspci -nn | grep Ethernet - An output example from the last step:: + An example output: + + .. code-block:: console 03:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] - where ``8086:1521`` is ``vendor:device ID`` and ``03:00.0`` is the entry for - device. Make note of this information; it is necessary for unbinding a - host's NICs. + where `03:00.0` is the device entry and `8086:1521` is the `vendor:device + ID`. Record this information, because you need it to unbind the NICs from a + host. -#. Unbind NICs from host to do passthrough with virtual machines. Clear Linux - OS for Intel Architecture currently supports this action. You can use the - following commands:: + +#. Unbind the NICs from the host to do pass-through with virtual machines. |CLOSIA| + supports this action. The commands take the format: + + .. code-block:: bash echo "vendor device_ID" > /sys/bus/pci/drivers/pci-stub/new_id echo "entry for device" > /sys/bus/pci/drivers/igb/unbind echo "entry for device" > /sys/bus/pci/drivers/pci-stub/bind echo "vendor device_ID" > /sys/bus/pci/drivers/pci-stub/remove_id + Here is a working example: + .. code-block:: bash - $ echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/new_id - $ echo "0000:03:00.0" > /sys/bus/pci/drivers/igb/unbind - $ echo "0000:03:00.0" > /sys/bus/pci/drivers/pci-stub/bind - $ echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/remove_id + sudo echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/new_id + sudo echo "0000:03:00.0" > /sys/bus/pci/drivers/igb/unbind + sudo echo "0000:03:00.0" > /sys/bus/pci/drivers/pci-stub/bind + sudo echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/remove_id -#. Assign to the KVM virtual machine (guest) the unbound NICs previously noted. - Modify the ``start_qemu.sh`` script in ``qemu-system-x86_64`` arguments, and - add the lines with the host's NICs information:: +#. Assign the unbound NICs to the KVM virtual machine (guest). + Modify the :file:`start_qemu.sh` script in `qemu-system-x86_64` arguments, and + add the lines with the host's NICs information in the format: - -device pci-assign,host="",id=passnic0,addr=03.0 - -device pci-assign,host="",id=passnic1,addr=04.0 + .. code-block:: bash - A working example: + -device pci-assign,host="",id=passnic0,addr=03.0 + -device pci-assign,host="",id=passnic1,addr=04.0 + + Here is a working example: .. code-block:: bash -device pci-assign,host=03:00.0,id=passnic0,addr=03.0 \ -device pci-assign,host=03:00.3,id=passnic1,addr=04.0 \ -#. If you would like to add more NUMA machines to the virtual machine, you can - add the next line in the Makefile boot target:: +#. Add more NUMA machines to the virtual machine by adding lines to the + Makefile boot target in the format: + + .. code-block:: bash -numa node,mem=,cpus= - As a working example for a virtual machine with 4096 of memory and four CPUs, the configuration - would look like this:: + Here is a working example for a virtual machine with 4096 memory and four + CPUs: + + .. code-block:: bash -numa node,mem=2048,cpus=0-1 \ -numa node,mem=2048,cpus=2-3 \ - This means that each NUMA machine has to use the same quantity of memory. + .. note:: Each NUMA machine must use the same quantity of memory. -#. Finally, run the ``start_qemu.sh`` script. +#. Run the :file:`start_qemu.sh` script. .. _13330: https://download.clearlinux.org/releases/13330/ .. _DPDK project: http://dpdk.org -.. _dpdk.org NICS: http://dpdk.org/doc/nics +.. _dpdk.org NICs: http://dpdk.org/doc/nics .. _pktgen tar package: http://dpdk.org/browse/apps/pktgen-dpdk/refs .. _DPDK guide: http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html -.. _Pktgen documentation: `Pktgen documentation`_ https://media.readthedocs.org/pdf/pktgen/latest/pktgen.pdf +.. _l3fwd documentation: http://dpdk.org/doc/guides/sample_app_ug/l3_forward.html +.. _pktgen documentation: http://pktgen-dpdk.readthedocs.io/en/latest/index.html diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-1.png b/source/clear-linux/guides/network/figures/vnc/vnc-1.png new file mode 100644 index 00000000..bb496aaf Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-1.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-10.png b/source/clear-linux/guides/network/figures/vnc/vnc-10.png new file mode 100644 index 00000000..be6751fc Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-10.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-11.png b/source/clear-linux/guides/network/figures/vnc/vnc-11.png new file mode 100644 index 00000000..0ba816b9 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-11.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-2.png b/source/clear-linux/guides/network/figures/vnc/vnc-2.png new file mode 100644 index 00000000..aa7561f7 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-2.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-3.png b/source/clear-linux/guides/network/figures/vnc/vnc-3.png new file mode 100644 index 00000000..29365e08 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-3.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-4.png b/source/clear-linux/guides/network/figures/vnc/vnc-4.png new file mode 100644 index 00000000..b3608289 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-4.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-5.png b/source/clear-linux/guides/network/figures/vnc/vnc-5.png new file mode 100644 index 00000000..ff56d089 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-5.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-6.png b/source/clear-linux/guides/network/figures/vnc/vnc-6.png new file mode 100644 index 00000000..5a808e34 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-6.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-7.png b/source/clear-linux/guides/network/figures/vnc/vnc-7.png new file mode 100644 index 00000000..a20e6c97 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-7.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-8.png b/source/clear-linux/guides/network/figures/vnc/vnc-8.png new file mode 100644 index 00000000..fa702cb4 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-8.png differ diff --git a/source/clear-linux/guides/network/figures/vnc/vnc-9.png b/source/clear-linux/guides/network/figures/vnc/vnc-9.png new file mode 100644 index 00000000..a8cafba4 Binary files /dev/null and b/source/clear-linux/guides/network/figures/vnc/vnc-9.png differ diff --git a/source/clear-linux/guides/network/ipxe-install.rst b/source/clear-linux/guides/network/ipxe-install.rst index 50e5c158..2a6585ef 100644 --- a/source/clear-linux/guides/network/ipxe-install.rst +++ b/source/clear-linux/guides/network/ipxe-install.rst @@ -1,70 +1,73 @@ .. _ipxe-install: Install Clear Linux over the network with iPXE -############################################## +################################################ -This guide shows how to install |CL| through :abbr:`PXE (Pre-boot Execution Environment)`. +This guide describes how to install Clear Linux\* using :abbr:`PXE (Pre-boot +Execution Environment)`. -PXE is an industry standard describing the client-server interaction with network-boot software using -the DHCP and TFTP protocols. This guide shows one possible use of this -environment to automatically install |CL|. +PXE is an industry standard that describes client-server interaction with +network-boot software and uses the DHCP and TFTP protocols. This guide shows one +method of using the PXE environment to install |CL|. -The PXE extension known as `iPXE`_\* adds support for additional protocols -such as HTTP, :abbr:`iSCSI (Internet Small Computer Systems Interface)`, :abbr:`AoE (ATA over Ethernet)`, and -:abbr:`FCoE (Fiber Channel over Ethernet)`. iPXE can also be used to enable -network booting on computers which lack built-in PXE support. +The PXE extension called `iPXE`_ adds support for additional protocols such as +HTTP, :abbr:`iSCSI (Internet Small Computer Systems Interface)`, :abbr:`AoE +(ATA over Ethernet\*)`, and :abbr:`FCoE (Fiber Channel over Ethernet\*)`. iPXE +enables network booting on computers with no built-in PXE support. -Figure 1 depicts the flow of information between a PXE server and a PXE -client we must create to install |CL| through iPXE. +To install |CL| through iPXE, you must create a PXE client. Figure 1 depicts +the flow of information between a PXE server and a PXE client. .. figure:: ./figures/network-boot-flow.png :alt: PXE information flow - Figure 1: PXE information flow + Figure 1: PXE information flow. .. caution:: - The |CL| image that boots through the PXE process automatically erases all data and partitions on the PXE client system and - creates 3 new partitions to install onto. + The |CL| image that boots through the PXE process automatically erases all + data and partitions on the PXE client system and creates 3 new partitions + to install onto. Prerequisites ************* -Before booting with iPXE, the following preparations must be made: +Before booting with iPXE, make the following preparations. -* Your PXE server has an Ethernet/LAN boot option. -* Your PXE server has at least two network adapters. -* Your PXE server is connected to a public network. -* Your PXE server and PXE clients are connected to a switch on a private - network. -* Your PXE server has the secure boot option disabled. -* Your PXE clients have a boot order where the network boot option is - prioritized before the disk boot option. +Connect the PXE server and PXE clients to a switch on a private network, as +shown in Figure 2. + +.. figure:: ./figures/network-boot-setup.png + :alt: Network topology + + Figure 2: Network topology. + +Your PXE client must have a boot order where the network boot option is +prioritized before the disk boot option. + +Your PXE server must have: + +* Ethernet/LAN boot option. +* At least two network adapters. +* Connection to a public network. +* Secure boot option disabled. .. note:: - The ``Secure Boot`` option in the BIOS must be disabled because the UEFI binaries used to - boot |CL| are not signed. + You must disable the secure boot option in the BIOS because the UEFI + binaries used to boot |CL| are not signed. -The required computer and network setup is shown in figure 2. - -.. figure:: ./figures/network-boot-setup.png - :alt: NAT network topology - - Figure 2: NAT network topology Configuration ************* -The configuration process to install |CL| using iPXE has been automated with -the :file:`configure-ipxe.sh` script included with -:abbr:`ICIS (Ister Cloud Init Service)`, thus quickly enabling a bulk -provisioning setup. For additional instructions on how to get started with the -script, refer to the guide on the `ICIS GitHub repository`_. Otherwise, to -setup manually, follow the steps below. +To set up |CL| using iPXE automatically, use the :file:`configure-ipxe.sh` +script included with :abbr:`ICIS (Ister Cloud Init Service)`. For additional +instructions on the script, refer to the guide on the `ICIS GitHub repository`_. -#. Define the variables used to parameterize the configuration of an iPXE - boot. +To set up |CL| manually, perform the steps below. + +#. Define the variables used for iPXE boot configuration. .. code-block:: console @@ -82,28 +85,28 @@ setup manually, follow the steps below. #. Log in and get root privilege. - .. code-block:: console + .. code-block:: bash - $ sudo -s + sudo -s -#. Add the ``pxe-server`` bundle to your |CL| system. This bundle has all the +#. Add the `pxe-server` bundle to your |CL| system. The bundle contains all files needed to run a PXE server. - .. code-block:: console + .. code-block:: bash - # swupd bundle-add pxe-server + sudo swupd bundle-add pxe-server #. Download the latest network-bootable release of |CL| and extract the files. - .. code-block:: console + .. code-block:: bash - # mkdir -p $ipxe_root - # curl -o /tmp/clear-pxe.tar.xz \ + sudo mkdir -p $ipxe_root + sudo curl -o /tmp/clear-pxe.tar.xz \ https://download.clearlinux.org/current/clear-$(curl \ https://download.clearlinux.org/latest)-pxe.tar.xz - # tar -xJf /tmp/clear-pxe.tar.xz -C $ipxe_root - # ln -sf $(ls $ipxe_root | grep 'org.clearlinux.*') $ipxe_root/linux + sudo tar -xJf /tmp/clear-pxe.tar.xz -C $ipxe_root + sudo ln -sf $(ls $ipxe_root | grep 'org.clearlinux.*') $ipxe_root/linux .. note:: @@ -118,8 +121,8 @@ setup manually, follow the steps below. .. code-block:: console - # cat > $ipxe_root/ipxe_boot_script.ipxe << EOF - #!ipxe + sudo cat > $ipxe_root/ipxe_boot_script.ipxe << EOF + sudo!ipxe kernel linux quiet init=/usr/lib/systemd/systemd-bootchart \ initcall_debug tsc=reliable no_timer_check noreplace-smp rw \ initrd=initrd @@ -127,14 +130,14 @@ setup manually, follow the steps below. boot EOF -#. The ``pxe-server`` bundle contains a lightweight web-server known as - ``nginx``. Create a configuration file for ``nginx`` to serve |CL| to PXE +#. The `pxe-server` bundle contains a lightweight web-server known as + `nginx`. Create a configuration file for `nginx` to serve |CL| to PXE clients with the following contents: .. code-block:: console - # mkdir -p /etc/nginx/conf.d - # cat > /etc/nginx/conf.d/$ipxe_app_name.conf << EOF + sudo mkdir -p /etc/nginx/conf.d + sudo cat > /etc/nginx/conf.d/$ipxe_app_name.conf << EOF server { listen $ipxe_port; server_name localhost; @@ -145,42 +148,42 @@ setup manually, follow the steps below. } EOF - # cp /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf + sudo cp /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf .. note:: - Creating a separate configuration file for ``nginx`` to serve - network-bootable images on a non-standard port number preserves - existing `nginx` configurations. + Create a separate `nginx` configuration file to serve network-bootable + images on a non-standard port number. This action saves existing `nginx` + configurations. -#. Start ``nginx`` and enable the startup on boot option. +#. Start `nginx` and enable the startup on boot option. + + .. code-block:: bash + + sudo systemctl start nginx + sudo systemctl enable nginx + +#. The `pxe-server` bundle contains a lightweight DNS server which + conflicts with the DNS stub listener provided in `systemd-resolved`. + Disable the DNS stub listener and temporarily stop `systemd-resolved`. .. code-block:: console - # systemctl start nginx - # systemctl enable nginx - -#. The ``pxe-server`` bundle contains a lightweight DNS server which - conflicts with the DNS stub listener provided by ``systemd-resolved``. - Disable the DNS stub listener and temporarily stop ``systemd-resolved``. - - .. code-block:: console - - # mkdir -p /etc/systemd - # cat > /etc/systemd/resolved.conf << EOF + sudo mkdir -p /etc/systemd + sudo cat > /etc/systemd/resolved.conf << EOF [Resolve] DNSStubListener=no EOF - # systemctl stop systemd-resolved + sudo systemctl stop systemd-resolved #. Assign a static IP address to the network adapter for the private network - and restart ``systemd-networkd`` with the following commands: + and restart `systemd-networkd` with the following commands: .. code-block:: console - # mkdir -p /etc/systemd/network - # cat > /etc/systemd/network/70-internal-static.network << EOF + sudo mkdir -p /etc/systemd/network + sudo cat > /etc/systemd/network/70-internal-static.network << EOF [Match] Name=$internal_iface [Network] @@ -188,74 +191,73 @@ setup manually, follow the steps below. Address=$pxe_internal_ip/$pxe_subnet_bitmask EOF - # systemctl restart systemd-networkd + sudo systemctl restart systemd-networkd -#. Configure NAT to route traffic from the private network to the public - network, effectively turning the PXE server into a router. To keep these - changes in spite of reboots, save the changes to the firewall with the - following commands: +#. Configure :abbr:`NAT (Network Address Translation)` to route traffic from + the private network to the public network. This action makes the PXE + server act as a router. To make these changes persistent during reboots, save the + changes to the firewall with the following commands: - .. code-block:: console + .. code-block:: bash - # iptables -t nat -F POSTROUTING - # iptables -t nat -A POSTROUTING -o $external_iface -j MASQUERADE - # systemctl enable iptables-save.service - # systemctl restart iptables-save.service - # systemctl enable iptables-restore.service - # systemctl restart iptables-restore.service + sudo iptables -t nat -F POSTROUTING + sudo iptables -t nat -A POSTROUTING -o $external_iface -j MASQUERADE + sudo systemctl enable iptables-save.service + sudo systemctl restart iptables-save.service + sudo systemctl enable iptables-restore.service + sudo systemctl restart iptables-restore.service .. note:: - The firewall masks or translates packets to make them appear as - coming from the PXE server. Thus, it hides the PXE clients from the - public network. + The firewall masks packets to make them appear as coming from the PXE + server and hides PXE clients from the public network. #. Configure the kernel to forward network packets to different interfaces. Otherwise, NAT will not work. - .. code-block:: console + .. code-block:: bash - # mkdir -p /etc/sysctl.d - # echo net.ipv4.ip_forward=1 > /etc/sysctl.d/80-nat-forwarding.conf - # echo 1 > /proc/sys/net/ipv4/ip_forward + sudo mkdir -p /etc/sysctl.d + sudo echo net.ipv4.ip_forward=1 > /etc/sysctl.d/80-nat-forwarding.conf + sudo echo 1 > /proc/sys/net/ipv4/ip_forward -#. The ``pxe-server`` bundle contains iPXE firmware images that allow computers +#. The `pxe-server` bundle contains iPXE firmware images that allow computers without an iPXE implementation to perform an iPXE boot. Create a TFTP - hosting directory and populate it with the iPXE firmware images with the - following commands: + hosting directory and populate the directory with the iPXE firmware images + with the following commands: - .. code-block:: console + .. code-block:: bash - # mkdir -p $tftp_root - # ln -sf /usr/share/ipxe/undionly.kpxe $tftp_root/undionly.kpxe + sudo mkdir -p $tftp_root + sudo ln -sf /usr/share/ipxe/undionly.kpxe $tftp_root/undionly.kpxe -#. The ``pxe-server`` bundle contains a lightweight TFTP, DNS, and DHCP - server known as ``dnsmasq``. Create a configuration file for ``dnsmasq`` +#. The `pxe-server` bundle contains a lightweight TFTP, DNS, and DHCP + server known as `dnsmasq`. Create a configuration file for `dnsmasq` to listen on a dedicated IP address for those functions. PXE clients on - the private network will use this IP address to access those functions. + the private network will use this IP address. .. code-block:: console - # cat > /etc/dnsmasq.conf << EOF + sudo cat > /etc/dnsmasq.conf << EOF listen-address=$pxe_internal_ip EOF #. Add the options to serve iPXE firmware images to PXE clients over TFTP to - the ``dnsmasq`` configuration file. + the `dnsmasq` configuration file. .. code-block:: console - # cat >> /etc/dnsmasq.conf << EOF + sudo cat >> /etc/dnsmasq.conf << EOF enable-tftp tftp-root=$tftp_root EOF -#. Add the options to host a DHCP server for PXE clients to the ``dnsmasq`` +#. Add the options to host a DHCP server for PXE clients to the `dnsmasq` configuration file. .. code-block:: console - # cat >> /etc/dnsmasq.conf << EOF + sudo cat >> /etc/dnsmasq.conf << EOF dhcp-leasefile=/var/db/dnsmasq.leases dhcp-authoritative @@ -272,7 +274,7 @@ setup manually, follow the steps below. EOF - This configuration provides the following important functions: + The configuration provides the following important functions: * Directs PXE clients without an iPXE implementation to the TFTP server to acquire architecture-specific iPXE firmware images that allow them @@ -281,44 +283,42 @@ setup manually, follow the steps below. defined subnet. * Directs PXE clients to the DNS server. * Directs PXE clients to the PXE server for routing via NAT. - * Divides the private network into two pools of IP addresses, one for - network booting and another for usage after boot, each with their own - lease times. + * Divides the private network into two pools of IP addresses. One pool + is for network boot and one pool is used after boot. Each pool has + their own lease times. -#. Create a file where ``dnsmasq`` can record the IP addresses it provides +#. Create a file for `dnsmasq` to record the IP addresses it provides to PXE clients. - .. code-block:: console + .. code-block:: bash - # mkdir -p /var/db - # touch /var/db/dnsmasq.leases + sudo mkdir -p /var/db + sudo touch /var/db/dnsmasq.leases -#. Start ``dnsmasq`` and enable startup on boot. +#. Start `dnsmasq` and enable startup on boot. - .. code-block:: console + .. code-block:: bash - # systemctl enable dnsmasq - # systemctl restart dnsmasq + sudo systemctl enable dnsmasq + sudo systemctl restart dnsmasq -#. Start ``systemd-resolved``. +#. Start `systemd-resolved`. - .. code-block:: console + .. code-block:: bash - # systemctl start systemd-resolved + sudo systemctl start systemd-resolved .. note:: - Using the ``dnsmasq`` DNS server allows ``systemd-resolved`` to dynamically - update the list of DNS servers for the private network from the public - network. This setup effectively creates a pass-through DNS server which - relies on the DNS servers listed in :file:`/etc/resolv.conf`. + `systemd-resolved` dynamically updates the list of DNS servers for the + private network if you use the `dnsmasq` DNS server. The setup creates a + pass-through DNS server that relies on the DNS servers listed in + :file:`/etc/resolv.conf`. -#. Power on the PXE client and watch it boot and install |CL|. +#. Power on the PXE client and watch the client boot and install |CL|. - .. note:: - - After booting, |CL| will automatically partition the hard drive, - install itself, update to the latest version, and reboot. + After booting, |CL| automatically partitions the hard drive, + installs itself, updates to the latest version, and reboots. **Congratulations!** You have successfully installed and configured a PXE diff --git a/source/clear-linux/guides/network/network-bonding.rst b/source/clear-linux/guides/network/network-bonding.rst index 6b562280..1b601748 100644 --- a/source/clear-linux/guides/network/network-bonding.rst +++ b/source/clear-linux/guides/network/network-bonding.rst @@ -1,31 +1,48 @@ .. _network-bonding: -Combine multiple interfaces -########################### +Combine multiple interfaces with network bonding +################################################ -Network bonding is a technique for combining multiple network interfaces into -a single, logical interface, providing some redundancy and bandwidth -aggregation. +Network bonding combines multiple network interfaces into a single logical +interface to provide redundancy and bandwidth aggregation. -|CLOSIA| includes the bonding_ and team_ drivers. The guide example provided -below shows how to configure systemd to use the ``bonding`` driver. +|CLOSIA| includes Linux bonding_ and team_ drivers. This guide describes how +to configure systemd to use the `bonding` driver. + +The example demonstrates how to: + +* Bond all four ports of a quad-port NIC in `802.3ad` mode. + +* Enable jumbo frames to optimize large data transfers on the local network. + +Your NICs and network switch must support `802.3ad` mode and jumbo frames. The +example explains how to configure your NICs for both features. Your switch may +require additional configuration. See your switch documentation for details. .. note:: - All commands in this guide must be run as root. -1. Create the ``/etc/systemd/network`` directory (if it doesn't already exist): + You must run all commands in this guide as root. + +#. Log in and get root privileges. .. code-block:: console - # mkdir -p /etc/systemd/network + sudo -s - This directory contains the configuration files and network settings - for the virtual device and its underlying physical interfaces. +#. Create the :file:`/etc/systemd/network` directory. -2. Configure systemd to create a virtual network device, ``bond1``. Use a text - editor to create a file named ``30-bond1.netdev`` as shown here: + .. code-block:: bash - .. code-block:: ini + mkdir -p /etc/systemd/network + + The :file:`/etc/systemd/network` directory contains configuration files and + network settings for the virtual device and its underlying physical + interfaces. + +#. Configure systemd to create a virtual network device called `bond1`. Use a + text editor to create a file named :file:`30-bond1.netdev`. + + .. code-block:: console [NetDev] Name=bond1 @@ -37,19 +54,15 @@ below shows how to configure systemd to use the ``bonding`` driver. MIIMonitorSec=1s LACPTransmitRate=fast - The syntax for this file is defined in the systemd.netdev_ manpage. - `This example`__ may be used verbatim, or tuned to your particular - requirements. Note that ``802.3ad`` mode requires explicit support from - your NICs and network switch. This and other modes may also require - additional configuration of your network switch. + Refer to the systemd.netdev_ manpage for :file:`30-bond1.netdev` file + syntax. This example is based on Example 9 on the manpage. Modify the + example for your configuration. -__ https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#id-1.20.10 +#. Configure the slave interfaces. Create a text file named + :file:`30-bond1-enp1s0.network`. Assign the slave interfaces to the virtual + `bond1` device and use the syntax shown in systemd.network_. -3. Configure the slave interfaces, assigning them to the new ``bond1`` device, - using the syntax in systemd.network_, and in a text file named - ``30-bond1-enp1s0.network`` as shown here: - - .. code-block:: ini + .. code-block:: console [Match] Name=enp1s0f* @@ -60,21 +73,20 @@ __ https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#id-1.20. [Link] MTUBytes=9000 - This example demonstrates bonding all four ports of a quad-port NIC, with - names in the range ``enp1s0f0-enp1s0f3``, allowing the use of a single file - with a wildcard match. You may also create a separate file for each NIC, - particularly if they have names that are not wildcard-friendly. This - configuration assigns each NIC as a slave of ``bond1``. For best results, - do not assign addresses or DHCP support to the individual NICs. + The example bonds all four ports of a quad-port NIC as a slave of `bond1`. + The example uses a wildcard match because the NIC names are in the range + `enp1s0f0-enp1s0f3`. If your NIC names are not wildcard-compatible, create + a separate :file:`.network` file for each NIC. - This example also enables jumbo frames of up to 9000 bytes to optimize large - data transfers on the local network. Again, your NICs and switch must - support jumbo frames, and your switch may require additional configuration. + For best results, do not assign addresses or DHCP support to the individual + NICs. -4. Define the network configuration for the bonded interface in a file named - ``30-bond1.network`` as shown here: + The `MTUBytes` setting enables jumbo frames of up to 9000 bytes. Your + switch may require additional configuration to support this setting. - .. code-block:: ini +#. Configure the bonded interface in a file named :file:`30-bond1.network`. + + .. code-block:: console [Match] Name=bond1 @@ -86,25 +98,26 @@ __ https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#id-1.20. [Link] MTUBytes=9000 - Since ``bond1`` is a virtual interface, it has no concept of physical link - status. The ``BindCarrier`` directive indicates that the link status of this - interface is determined by the status of the listed slave devices. + `bond1` is a virtual interface with no physical link status. - This is the logical interface, so assign it an IP address. DHCP is more - complicated with bonded interfaces, and is not covered in this example. + `BindCarrier` indicates that the `bond1` link status is determined by the + status of the listed slave devices. - This file also enables jumbo frames of up to 9000 bytes. This option must be - enabled for all slave interfaces *and* the bonded interface, in order to take - effect. + `Address` contains an IP address that you assign to the logical interface. + DHCP bonded interfaces are complex and outside the scope of this example. -5. Apply the new network configuration: + `MTUBytes` must be set to 9000 on all slave interfaces and on the bonded + interface for successful jumbo frames operation. If `MTUBytes` is not the + same on all interfaces, then the lowest value is used. - .. code-block:: console +#. Apply the new network configuration with the command: - # systemctl restart systemd-networkd + .. code-block:: bash - The MTU settings will not take effect until a reboot, or if you explicitly - apply them via ``ifconfig``, for example. + systemctl restart systemd-networkd + + The `MTUBytes` settings do not take effect until you reboot or manually + apply the settings with a utility such as `ifconfig`. .. _bonding: https://www.kernel.org/doc/Documentation/networking/bonding.txt diff --git a/source/clear-linux/guides/network/network.rst b/source/clear-linux/guides/network/network.rst index d1c7357a..7c765728 100644 --- a/source/clear-linux/guides/network/network.rst +++ b/source/clear-linux/guides/network/network.rst @@ -7,10 +7,11 @@ This guide provides step-by-step instructions for common tasks associated with the configuration, administration, and use of networks in the |CLOSIA|. .. toctree:: - :maxdepth: 1 + :maxdepth: 1 - ipxe-install - dpdk - ovs-dpdk - network-bonding - custom-clear-container \ No newline at end of file + ipxe-install + dpdk + ovs-dpdk + network-bonding + custom-clear-container + vnc \ No newline at end of file diff --git a/source/clear-linux/guides/network/vnc.rst b/source/clear-linux/guides/network/vnc.rst new file mode 100644 index 00000000..e2f1e21d --- /dev/null +++ b/source/clear-linux/guides/network/vnc.rst @@ -0,0 +1,790 @@ +.. _vnc: + +Remote-desktop to a Clear Linux host using VNC +############################################## + +:abbr:`VNC (Virtual Network Computing)` is a client-server GUI-based tool +that allows you to connect via remote-desktop to your |CLOSIA| host. + +This guide shows you how to: + +* Install the VNC server and misc. components on your |CL| host. +* Configure a VNC-server-start method on your |CL| host. +* Install a VNC viewer app and an SSH client on your client system. +* Establish a VNC connection to your |CL| host. +* Terminate a VNC connection to your |CL| host. +* Encrypt VNC traffic through an SSH tunnel. + +Install the VNC server and misc. components on your Clear Linux host +******************************************************************** + +To configure VNC to work on your |CL| host, install these bundles: + +* `desktop-autostart`: Installs :abbr:`GDM (Gnome Desktop Manager)`, sets + it to start automatically on boot, and installs TigerVNC Viewer. +* `vnc-server`: Installs the TigerVNC server. + +Follow these steps: + +#. Log into your |CL| host and get root privileges. + + .. code-block:: console + + $ sudo -s + +#. Install the |CL| bundles. + + .. code-block:: console + + # swupd bundle-add desktop-autostart vnc-server + +#. Reboot your |CL| host. + +Configure a VNC-server-start method on your Clear Linux host +************************************************************ + +There are three methods you can use to configure and start the VNC server on +your host: + +.. list-table:: Table 1: VNC-server-start Configuration Methods + :widths: 10,20,20,20 + :header-rows: 1 + + * - Attribute + - `Method 1`: Manually start a VNC session + - `Method 2`: Automatically start a VNC session via a systemd service script + - `Method 3`: Create multi-user logins with authentication through GDM + * - Description + - This is the traditional method where you SSH into the |CL| host, manually + start a VNC session to get a display ID, and connect to it by + supplying the display ID. + - The system administrator sets up a systemd service script for you with + a pre-assigned display ID. You make a VNC connection and supply + your pre-assigned display ID. + - The system adminstrator configures GDM to accept connection requests. + When you make a VNC connection to the |CL| host, you see + the GDM login screen and authenticate as if you are local. + * - Who configures VNC settings? + - You + - System adminstrator + - System adminstrator + * - Who starts VNC session? + - You + - Set to start automatically on boot by system administrator + - Set to start automatically on boot by system administrator + * - Who ends VNC sesssion? + - You + - You + - System administrator can disable VNC service altogether + * - Requires VNC password to authenticate? + - Yes + - Yes + - No. Use |CL| account username and password through GDM + + +Although all three methods can coexist on the same |CL| host, we recommend +you pick a method that suits your needs. + +For simplicity, the rest of this guide refers to these methods as +`Method 1`, `Method 2`, and `Method 3`. + +Method 1: Manually start a VNC session +====================================== + +You (and each user) must perform these steps to initialize your VNC settings. + +#. Log in. +#. Open a terminal emulator. +#. Start VNC with the :command:`vncserver` command. Since this is your + first time starting VNC, it adds default configuration files and asks you + to set a VNC password. + + .. code-block:: console + + $ vncserver + + Example output: + + .. code-block:: console + + $ vncserver + + You will require a password to access your desktops. + + Password: + Verify: + Would you like to enter a view-only password (y/n)? n + xauth: file /home/vnc-user-a/.Xauthority does not exist + + New 'clr-linux:2 (vnc-user-a)' desktop is clr-linux:2 + + Creating default startup script /home/vnc-user-a/.vnc/xstartup + Creating default config /home/vnc-user-a/.vnc/config + Starting applications specified in /home/vnc-user-a/.vnc/xstartup + Log file is /home/vnc-user-a/.vnc/clr-linux:2.log + + Upon completion, you can find the default configuration files and the + password file hidden in the `.vnc` directory in your home directory. + + Also, a VNC session starts and shows a unique display ID, which is the + number following the hostname and the colon `:`. In the above example, the display ID is 2. In a later step, you will supply the display ID to + your VNC viewer app for connection. + +#. Kill the active VNC session for the time being with the + :command:`vncserver -kill :[display ID]` command. Substitute [display ID] + with your active VNC session display ID. For example: + + .. code-block:: console + + $ vncserver -kill :2 + + .. note:: + + If you do not recall the active session display ID, use the + :command:`vncserver -list` command to find it. + +#. Optional configurations: + + * To customize settings such as screen size, security type, etc., + modify the :file:`$HOME/.vnc/config` file. + * To customize the applications to run at startup, modify the + :file:`$HOME/.vnc/xstartup` file. + +Method 2: Automatically start a VNC session via a systemd service script +======================================================================== + +To configure VNC for this method, you must have root privileges. You will +set up a systemd service file for all intended VNC users with their own +preassigned unique display ID. + +#. Log in and get root privileges. + + .. code-block:: console + + $ sudo -s + +#. Make sure the user accounts already exist. Use the following command to + list all users. + + + .. code-block:: console + + # cut -d: -f1 /etc/passwd + +#. Create the path :file:`/etc/systemd/system`. + + .. code-block:: console + + # mkdir -p /etc/systemd/system + +#. Create a systemd service script file :file:`vncserver@:[X].service`, + where [X] is the display ID, for each user in :file:`/etc/systemd/system` + Each user must be assigned a unique display ID. Be sure the correct + username is entered in the `User` field. The example below shows user + `vnc-user-b` who is assigned the display ID `5`. + + .. code-block:: console + + # cat > /etc/systemd/system/vncserver@:5.service << EOF + + [Unit] + Description=VNC Remote Desktop Service for "vnc-user-b" with display ID "5" + After=syslog.target network.target + + [Service] + Type=simple + User=vnc-user-b + PAMName=login + PIDFile=/home/%u/.vnc/%H%i.pid + ExecStartPre=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :' + ExecStart=/usr/bin/vncserver %i -geometry 2000x1200 -alwaysshared -fg + ExecStop=/usr/bin/vncserver -kill %i + + [Install] + WantedBy=multi-user.target + + EOF + +#. Have each user log into their account and set a VNC password with + the :command:`vncpasswd` command before proceeding to the next step. + +#. Start the VNC service script and set it to start automatically on + boot for each user. Replace the [X] with the display ID. + + .. code-block:: console + + # systemctl daemon-reload + # systemctl start vncserver@:[X].service + # systemctl enable vncserver@:[X].service + +#. After starting the services, verify they are running. + + .. code-block:: console + + # systemctl | grep vnc + + The example below shows 2 VNC sessions that were successfully started for + users `vnc-user-b` with display ID 5 and `vnc-user-c` with display ID 6. + + .. code-block:: console + + # systemctl | grep vnc + + vncserver@:5.services loaded active running VNC Remote Desktop Service for "vnc-user-b" with display ID "5" + vncserver@:6.services loaded active running VNC Remote Desktop Service for "vnc-user-c" with display ID "6" + system-vncserver.slice loaded active active system-vncserver.slice + +Method 3: Multi-user logins with authentication through GDM +=========================================================== + +For this method, VNC is configured as a systemd service that listens on port +5900 and GDM is configured to accept access requests from VNC. When you +make a VNC connection to your |CL| host, you are presented with the GDM login screen and you authenticate as if you are local. You must have root privileges to perform this configuration. + +#. Log in and get root privileges. + + .. code-block:: console + + $ sudo -s + +#. Create the path :file:`/etc/systemd/system`. + + .. code-block:: console + + # mkdir -p /etc/systemd/system + +#. Create a systemd socket file :file:`xvnc.socket` and add the following: + + .. code-block:: console + + # cat > /etc/systemd/system/xvnc.socket << EOF + + [Unit] + Description=XVNC Server on port 5900 + + [Socket] + ListenStream=5900 + Accept=yes + + [Install] + WantedBy=sockets.target + + EOF + +#. Create a systemd service file :file:`xvnc@.service` and add the following: + + .. code-block:: console + + # cat > /etc/systemd/system/xvnc@.service << EOF + + [Unit] + Description=Daemon for each XVNC connection + + [Service] + ExecStart=-/usr/bin/Xvnc -inetd -query localhost -geometry 2000x1200 -once -SecurityTypes=None + User=nobody + StandardInput=socket + StandardError=syslog + + EOF + +#. Create the path :file:`/etc/gdm`. + + .. code-block:: console + + # mkdir -p /etc/gdm + + +#. Create a GDM :file:`custom.conf` file and add the following: + + .. code-block:: console + + # cat > /etc/gdm/custom.conf << EOF + + [xdmcp] + Enable=true + Port=177 + + EOF + +#. Start the VNC socket script and set it to start automatically on boot. + + .. code-block:: console + + # systemctl daemon-reload + # systemctl start xvnc.socket + # systemctl enable xvnc.socket + +#. After starting the socket, verify it is running. + + .. code-block:: console + + # systemctl | grep vnc + + The example below shows the xvnc.socket is running. + + .. code-block:: console + + # systemctl | grep vnc + + xvnc.socket loaded active listening XVNC Server on port 5900 + system-xvnc.slice loaded active active system-xvnc.slice + +See the `vncserver` Man page for additional information. + +Install a VNC viewer app and an SSH client on your client system +**************************************************************** + +You need a VNC viewer app on your client system to connect to your |CL| host. +An SSH client is only needed if you chose to use `Method 1` or you plan to +encrypt your VNC traffic, which is discussed later in this guide. + +Perform the steps below to add these apps to your client system. + +Install a VNC viewer app +======================== + +On |CL|: + +.. code-block:: console + + # swupd bundle-add desktop-autostart + +On Ubuntu, Mint: + +.. code-block:: console + + # apt-get install xtightvncviewer + +On Fedora: + +.. code-block:: console + + # dnf install tigervnc + +On Windows: + +* Install `RealVNC for Windows`_ + +On macOS: + +* Install `RealVNC for macOS`_ + +Install an SSH client +===================== + +* On most Linux distros (Clear Linux, Ubuntu, Mint, Fedora, etc.) and macOS, + SSH is built-in so you don't need to install it. +* On Windows, you can install `Putty`_. + +Establish a VNC connection to your Clear Linux host +*************************************************** + +Depending on the VNC-server-configuration method chosen, use the appropriate VNC connection: + +If you chose `Method 1`, you must take a few extra steps by +using SSH to connect to your |CL| host and then manually launching VNC. + +If you chose `Method 2`, get your preassigned VNC display ID from your +system administrator first and then proceed to the +:ref:`connect-to-vnc-session` section below. + +If you chose `Method 3`, proceed to the +:ref:`connect-to-vnc-session` below. + + +SSH into your Clear Linux host and launch VNC +============================================= + +#. SSH into your Clear Linux host + + #. On Linux distros and macOS: + + .. code-block:: console + + $ ssh [username]@[clear-linux-host-ip-address] + + #. On Windows: + + #. Launch Putty. + #. Under the :guilabel:`Category` section, select :guilabel:`Session`. + See Figure 1. + #. Enter the IP address of your Clear Linux host in the + :guilabel:`Host Name (or IP address)` field. + #. Set the :guilabel:`Connection type` option to :guilabel:`SSH`. + #. Click the :guilabel:`Open` button. + + .. figure:: figures/vnc/vnc-1.png + :scale: 90 % + :alt: Putty - configure SSH session settings + + Figure 1: Putty - configure SSH session settings + +#. Log in with your |CL| username and password. Do not use your VNC + password. +#. Start a VNC session. + + .. code-block:: console + + $ vncserver + + Example output: + + .. code-block:: console + + $ vncserver + + New 'clr-linux:3 (vnc-user-c)' desktop is clr-linux:3 + + Starting applications specified in /home/vnc-user-c/.vnc/xstartup + Log file is /home/vnc-user-c/.vnc/clr-linux:3.log + +#. Take note of the generated display ID because you will input it into + the VNC viewer app to establish the connection. The above example shows + the display ID is 3. + + .. note:: + + VNC automatically picks a unique display ID unless you specify one. + To specify a display ID, enter a unique number that is not already + in use after the colon. For example: + + .. code-block:: console + + $ vncserver :8 + +#. You can now end the SSH connection by logging out. This does + not terminate your active VNC session. + +.. _connect-to-vnc-session: + +Connect to your VNC session +=========================== + +For `Method 1` and `Method 2`, you must connect to a specific active session +or display ID using one of two options: + +* Use a fully-qualified VNC port number, which consists of the default VNC + server port (5900) plus the display ID +* Use the display ID + +For example, if the display ID is 3, it can be specified as `5903` or just +as `3`. For `Method 3`, VNC does not expect a display ID. Use `5900`. For simplicity, the instructions below use the fully-qualified VNC port +number. + +**On Linux distros:** + +#. Open a terminal emulator and enter: + + .. code-block:: console + + $ vncviewer [clear-linux-host-ip-address]:[fully-qualified VNC port number] + +#. Enter your credentials. + + * For `Method 1` and `Method 2`, enter your VNC password. No username is + required. + * For `Method 3`, enter your |CL| account username and password through + GDM. + + .. note:: + + With `Method 3`, you cannot remotely log into your |CL| host through + VNC if you are logged in locally and vice versa. + +**On Windows and macOS using `RealVNC` app:** + +#. Start the RealVNC viewer app. See Figure 2. +#. Enter the IP address of the Clear Linux host and the fully-qualified + VNC port number. + + The following screenshot shows connecting to |CL| host + 192.168.25.54 with a fully-qualified VNC port number 5902. + + .. figure:: figures/vnc/vnc-2.png + :scale: 90 % + :alt: RealVNC Viewer + + Figure 2: RealVNC Viewer + +#. Press the :kbd:`Enter` key. + +#. Enter your credentials. + + * For `Method 1` and `Method 2`, enter your VNC password. No username is + required. + * For `Method 3`, enter your |CL| account username and password through + GDM. + + .. note:: + + With `Method 3`, you cannot remotely log into your |CL| host through + VNC if you are logged in locally and vice versa. + +`Optional: Configure RealVNC Image Quality` +------------------------------------------- + +To increase the RealVNC viewer image quality, manually change the `ColorLevel` value. Follow these steps: + +#. Right-click a connection node and select :guilabel:`Properties...`. + See Figure 3. + + .. figure:: figures/vnc/vnc-3.png + :scale: 90 % + :alt: RealVNC Viewer - change connection node properties + + Figure 3: RealVNC Viewer - change connection node properties + +#. Select the :guilabel:`Expert` tab. See Figure 4. + +#. Select the :guilabel:`ColorLevel` setting and change it to your + preferred setting. + + .. figure:: figures/vnc/vnc-4.png + :scale: 90 % + :alt: RealVNC Viewer - change ColorLevel + + Figure 4: RealVNC Viewer - change :guilabel:`ColorLevel` + +Terminate a VNC connection to your Clear Linux host +*************************************************** + +For `Method 1` and `Method 2`, once started, a VNC session remains active +on your |CL| host even if you close your VNC viewer app. If you want to +truly terminate an active VNC session, follow these steps: + +#. SSH into your Clear Linux host. +#. Open a terminal emulator. +#. Find the active VNC session display ID with the command + :command:`vncserver -list`. + + .. code-block:: console + + $ vncserver -list + +#. Terminate it with the :command:`vncserver -kill` command followed by a + colon and the display ID. + + .. code-block:: console + + $ vncserver -kill :[display ID] + +#. For `Method 3`, only the system administrator can stop and disable the + VNC service by using these commands: + + .. code-block:: console + + # systemctl stop xvnc.socket + # systemctl disable xnvc.socket + + +Encrypt VNC traffic through an SSH tunnel +***************************************** + +By default, VNC traffic is not encrypted. Figure 6 shows an example warning +from RealVNC Viewer. + +.. figure:: figures/vnc/vnc-6.png + :scale: 90 % + :alt: RealVNC Viewer - Connection not encrypted warning + + Figure 6: RealVNC Viewer - Connection not encrypted warning + +To add security, VNC traffic can be routed through an SSH tunnel. This is accomplished by following these steps: + +#. Configure the VNC server to only accept connection from localhost by + adding the `-localhost` option. +#. Set up an SSH tunnel between your client system and your |CL| host. + Your client system will forward traffic from the localhost (the client) + destined for a specified fully-qualified VNC port number (on the client) + to your |CL| host with the same port number. +#. The VNC viewer app on your client system will now connect to localhost, + instead of the IP address of your |CL| host. + +Configure VNC to only accept connection from localhost +====================================================== + +For `Method 1`: + +#. Edit the :file:`config` file located in :file:`$HOME/.vnc` and uncomment + the `# localhost` line. It should look like this: + + .. code-block:: console + + ## Supported server options to pass to vncserver upon invocation can be listed + ## in this file. See the following manpages for more: vncserver(1) + Xvnc(1). + ## Several common ones are shown below. Uncomment and modify to your liking. + ## + # securitytypes=vncauth,tlsvnc + # desktop=sandbox + # geometry=2000x1200 + localhost + # alwaysshared + +#. If an active session exists, kill it, and then restart it. + +For `Method 2`: + +#. Edit the systemd service script :file:`vncserver@:[X].service` located in + :file:`/etc/systemd/system` and add `-localhost` to the `ExecStart` + line. The example below uses vncserver@:5.service: + + .. code-block:: console + + [Unit] + Description=VNC Remote Desktop Service for "vnc-user-b" with display ID "5" + After=syslog.target network.target + + [Service] + Type=simple + User=vnc-user-b + PAMName=login + PIDFile=/home/%u/.vnc/%H%i.pid + ExecStartPre=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :' + ExecStart=/usr/bin/vncserver %i -geometry 2000x1200 -localhost -alwaysshared -fg + ExecStop=/usr/bin/vncserver -kill %i + + [Install] + WantedBy=multi-user.target + +#. Restart the service script: + + .. code-block:: console + + # systemctl daemon-load + # systemctl restart vncserver@:5.service + +For `Method 3`: + +#. No change is needed to the :file:`xvnc@service` script. + + After you have restarted your VNC session, you can verify that it only + accepts connections from localhost by using the :command:`netstat` + command like this: + + .. code-block:: console + + $ netstat -plant + + .. note:: + + Add the |CL| `network-basic` bundle to get the :command:`netstat` + command. + +Figure 7 shows two VNC sessions (5901 and 5905) accepting connections from +any host as specified by the `0.0.0.0`'s. This is before the `-localhost` option was used. + +.. figure:: figures/vnc/vnc-7.png + :scale: 100 % + :alt: VNC session accepting connection from any host + + Figure 7: VNC sessions (5901 and 5905) accepting connections from any host + +Figure 8 shows two VNC sessions (5901 and 5905) only accepting connections from localhost as specified by `127.0.0.1`'s. This is after the `-localhost` option was used. + +.. figure:: figures/vnc/vnc-8.png + :scale: 100 % + :alt: VNC session only accepting connection from localhost + + Figure 8: VNC sessions (5901 and 5905) only accepting connections from localhost + +Set up an SSH tunnel from your client system to your |CL| host +============================================================== + +**On Linux distros and macOS:** + +#. Open terminal emulator and enter: + + .. code-block:: console + + $ ssh -L [client port number]:localhost:[fully-qualified VNC port number] \ + -N -f -l [username] [clear-linux-host-ip-address] + +#. Enter your |CL| account password (not your VNC password). + + .. note:: + + * `-L` specifies that [client port number] on the localhost (on the + client side) is forwarded to [fully-qualified VNC port number] + (on the server side). + * Replace `[client port number]` with an available client port number + (for example: 1234). For simplicity, you can make the + `[client port number]` the same as the `[fully-qualified VNC port number]`. + * Replace `[fully-qualified VNC port number]` with 5900 (default VNC + port) plus the display ID. For example, if the display ID is 2, + the fully-qualified VNC port number is is 5902. + * `-N` tells SSH to only forward ports and not execute a remote + command. + * `-f` tells SSH to go into the background before command execution. + * `-l` specifies the username to log in as. + +**On Windows:** + +#. Launch Putty. +#. Specify the |CL| VNC host to connect to. + + #. Under the :guilabel:`Category` section, select :guilabel:`Session`. + See Figure 1. + #. Enter the IP address of your Clear Linux host in the + :guilabel:`Host Name (or IP address)` field. + #. Set the :guilabel:`Connection type` option to :guilabel:`SSH`. + +#. Configure the SSH tunnel. See Figure 9 for an example. + + #. Under the :guilabel:`Category` section, go to + :guilabel:`Connection` > :guilabel:`SSH` > :guilabel:`Tunnels`. + + #. In the :guilabel:`Source port` field, enter an available client + port number (for example: 1234). For simplicity, you can make the + `Source port` the same as the fully-qualified VNC port number. + + #. In the :guilabel:`Destination` field, enter + `localhost:` plus the fully-qualified VNC port number. + + #. Click the :guilabel:`Add` button. + + .. figure:: figures/vnc/vnc-9.png + :scale: 100 % + :alt: Putty - configure SSH tunnel + + Figure 9: Putty - configure SSH tunnel + +#. Click the :guilabel:`Open` button. +#. Enter your |CL| account password (not your VNC password). + +Connect to a VNC session through an SSH tunnel +============================================== + +After you have set up an SSH tunnel, follow these instructions to connect to +your VNC session. + +**On Linux distros:** + +#. Open terminal emulator and enter: + + .. code-block:: console + + $ vncviewer localhost:[client port number] + +**On Windows and macOS using `RealVNC`:** + +#. Start the RealVNC viewer app. +#. Enter `localhost` and the fully-qualified VNC port number. See Figure 10 + for an example. + + .. figure:: figures/vnc/vnc-10.png + :scale: 100 % + :alt: RealVNC viewer app connecting to localhost:1234 + + Figure 10: RealVNC viewer app connecting to `localhost:1234` + + .. note:: + + RealVNC will still warn that the connection is not encrypted even + though its traffic is going through the SSH tunnel. You can ignore + this warning. + +.. _RealVNC for Windows: https://www.realvnc.com/en/connect/download/viewer/windows/ +.. _RealVNC for macOS: https://www.realvnc.com/en/connect/download/viewer/macos/ +.. _Putty: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html diff --git a/source/clear-linux/reference/collaboration/documentation/images.rst b/source/clear-linux/reference/collaboration/documentation/images.rst index ee46e817..6ce6ad66 100644 --- a/source/clear-linux/reference/collaboration/documentation/images.rst +++ b/source/clear-linux/reference/collaboration/documentation/images.rst @@ -45,8 +45,8 @@ Follow these guidelines when creating graphics for the |CLOSIA|: * Use only approved image formats. Use either PNG or JPEG bitmap files for screenshots and SVG files for vector graphics. If a figure is not a - photograph or screenshot, please provide figure as a vector graphic to - ensure it can be changed later on. + photograph or screenshot, use the vector graphic file format to ensure + the figure can be changed later. Examples diff --git a/source/clear-linux/reference/collaboration/documentation/language.rst b/source/clear-linux/reference/collaboration/documentation/language.rst index f086ce07..1d7d0303 100644 --- a/source/clear-linux/reference/collaboration/documentation/language.rst +++ b/source/clear-linux/reference/collaboration/documentation/language.rst @@ -3,10 +3,11 @@ Language reference guide ######################## -This section provides you with the accepted use of the English language. It -includes information about words use, punctuation, and grammar. This guide -does not replace a professional writer's review but is intended to help -collaborators submit consistent contributions. +This section describes acceptable usage of the English language in the +|CLOSIA| documentation. It includes information about words use, +punctuation, and grammar. This guide does not replace a professional +writer's review but is intended to help collaborators submit consistent +contributions. To make translations easier and to make the content accessible to non-native speakers, |CLOSIA| uses Simple English. However, we have not implemented any diff --git a/source/clear-linux/reference/collaboration/documentation/simple.rst b/source/clear-linux/reference/collaboration/documentation/simple.rst index 7815e6b1..a67a7e19 100644 --- a/source/clear-linux/reference/collaboration/documentation/simple.rst +++ b/source/clear-linux/reference/collaboration/documentation/simple.rst @@ -13,8 +13,8 @@ makes translation easier, and improves comprehension for people whose first language is not English. |CLOSIA| does not use controlled language, which restricts the writer's -vocabulary to a list of approved words. Some preferences are evidently in -place. +vocabulary to a list of approved words. However, we do strongly recommend +using the language principles described below. Short sentences and paragraphs ****************************** @@ -26,7 +26,7 @@ principle of one main idea in a sentence, plus one additional point if needed. Similarly, restrict your paragraph length to about six sentences. -Remember the basic structure of a paragraph: Introduction, body and +Remember the basic structure of a paragraph: Introduction, body, and conclusion. Both the introduction and the conclusion should be one sentence long. The body of a paragraph should never exceed four sentences. Here less is more. diff --git a/source/clear-linux/reference/collaboration/documentation/tables.rst b/source/clear-linux/reference/collaboration/documentation/tables.rst index fe82c527..3ad59298 100644 --- a/source/clear-linux/reference/collaboration/documentation/tables.rst +++ b/source/clear-linux/reference/collaboration/documentation/tables.rst @@ -17,8 +17,8 @@ Follow these general guidelines: * Indent the contents correctly. This allows the content to be read even if it is not rendered. -* Only create a table if the body of the table is larger than 6, that means - at least 2x3 or 3x2. +* Only create a table if the body of the table contains six or more cells, +which is a minimum table size of at least 2x3 or 3x2. ReST supports several types of tables. |CL| uses grid and :abbr:`CSV-tables (Comma Separated Values tables)`. Grid tables are only @@ -71,9 +71,9 @@ support several layout options. For example: crunchy, now would it?" "Gannet Ripple", 1.99, "On a stick!" -CSV-tables can have a title, the header row is optional and separate from the -rest of the table, each column's width is customizable among others. See the -Sphinx `CSV-tables documentation`_ to learn all the possible options +Some of the options available with CSV-tables are table titles, an optional +header row separate from the rest of the table, and customizable column width. +See the Sphinx `CSV-tables documentation`_ to learn all the possible options available. This template can help you create CSV-tables: diff --git a/source/clear-linux/tutorials/aws-web/aws-web.rst b/source/clear-linux/tutorials/aws-web/aws-web.rst index 2a5c656d..8c433212 100644 --- a/source/clear-linux/tutorials/aws-web/aws-web.rst +++ b/source/clear-linux/tutorials/aws-web/aws-web.rst @@ -198,13 +198,13 @@ other instances available, they are also listed but not selected. chmod 400 AWSClearTestKey.pem #. Copy the text highlighted in the :guilabel:`Example:` section that is - shown in :ref:`figure 11`. Paste the copied text into your terminal and add - `-l clear` text to the end of the command string. Press the :kbd:`Enter` - key to execute the command. + shown in :ref:`figure 11`. Paste the copied text into your + terminal, change the text before the `@` sign to the username `clear`, and + press the :kbd:`Enter` key to execute the command. .. code-block:: console - ssh -i "AWSClearTestKey.pem" root@ec2-34-209-39-184.us-west-2.compute.amazonaws.com -l clear + ssh -i "AWSClearTestKey.pem" clear@ec2-34-209-39-184.us-west-2.compute.amazonaws.com #. A message appears on the terminal stating the authenticty of the host can't be established and prompts you with the message: diff --git a/source/clear-linux/tutorials/spark.rst b/source/clear-linux/tutorials/spark.rst new file mode 100644 index 00000000..7941de1d --- /dev/null +++ b/source/clear-linux/tutorials/spark.rst @@ -0,0 +1,134 @@ + .. _spark: + +Set up a standalone cluster system using Apache\* Spark\* +######################################################### + +This tutorial describes how to install, configure, and run Apache Spark on +|CLOSIA|. Apache Spark is a fast general-purpose cluster computing system with +the following features: + +* Provides high-level APIs in Java\*, Scala\*, Python\*, and R\*. +* Includes an optimized engine that supports general execution graphs. +* Supports high-level tools including Spark SQL, MLlib, GraphX, and Spark + Streaming. + +In this tutorial, you will install Spark on a single machine running the +master daemon and a worker daemon. + +Prerequisites +************* + +This tutorial assumes you have installed |CL| on your host system. +For detailed instructions on installing |CL| on a bare metal system, visit +the :ref:`bare metal installation tutorial`. + +Before you install any new packages, update |CL| with the following command: + +.. code-block:: bash + + sudo swupd update + +Install Apache Spark +******************** + +Apache Spark is included in the :file:`big-data-basic` bundle. To install the +framework, enter: + +.. code-block:: bash + + sudo swupd bundle-add big-data-basic + +Configure Apache Spark +********************** + +#. Create the configuration directory with the command: + + .. code-block:: bash + + sudo mkdir /etc/spark + +#. Copy the default templates from :file:`/usr/share/defaults/spark` to + :file:`/etc/spark` with the command: + + .. code-block:: bash + + sudo cp /usr/share/defaults/spark/* /etc/spark + + .. note:: Since |CL| is a stateless system, you should never modify the + files under the :file:`/usr/share/defaults` directory. The software + updater overwrites those files. + + +#. Copy the template files below to create custom configuration files: + + .. code-block:: bash + + sudo cp /etc/spark/spark-defaults.conf.template /etc/spark/spark-defaults.conf + sudo cp /etc/spark/spark-env.sh.template /etc/spark/spark-env.sh + sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties + +#. Edit the :file:`/etc/spark/spark-env.sh` file and add the + :envvar:`SPARK_MASTER_HOST` variable. Replace the example address below + with your localhost IP address. View your IP address using the + :command:`hostname -I` command. + + .. code-block:: + + SPARK_MASTER_HOST="10.300.200.100" + + .. note:: This optional step enables the master's web user interface to + view information needed later in this tutorial. + +#. Edit the :file:`/etc/spark/spark-defaults.conf` file and update the + `spark.master` variable with the `SPARK_MASTER_HOST` address and port `7077`. + + .. code-block:: + + spark.master spark://10.300.200.100:7077 + +Start the master server and a worker daemon +******************************************* + +#. Start the master server using: + + .. code-block:: bash + + sudo /usr/share/apache-spark/sbin/./start-master.sh + +#. Start one worker daemon and connect it to the master using the + `spark.master` variable defined earlier: + + .. code-block:: bash + + sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077 + +#. Open an internet browser and view the worker daemon information using + the master's IP address and port `8080`: + + .. code-block:: + + http://10.300.200.100:8080 + +Run the Spark wordcount example +******************************* + +#. Run the wordcount example using a file on your local host and output the + results to a new file with the following command: + + .. code-block:: bash + + sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results + +#. Open an internet browser and view the application information using + the master's IP address and port `8080`: + + .. code-block:: + + http://10.300.200.100:8080 + +#. View the results of the wordcount application in the :file:`~/Documents/results` file. + +**Congratulations!** + +You successfully installed and set up a standalone Apache Spark cluster. +Additionally, you ran a simple wordcount example. diff --git a/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst b/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst index 0d89b00b..54bddc12 100644 --- a/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst +++ b/source/clear-linux/tutorials/telemetry-backend/telemetry-backend.rst @@ -13,6 +13,7 @@ also redirect where records go if they wish to collect records for themselves by setting up and using their own telemetry backend server. A telemetry backend server consists of two Flask applications: + * The :guilabel:`collector` is an ingestion app for records received from the :guilabel:`telemetrics-client` probes. * The :guilabel:`telemetryui` web app exposes several visualizations of the diff --git a/source/clear-linux/tutorials/tutorials.rst b/source/clear-linux/tutorials/tutorials.rst index 044f813a..3cbb1f17 100644 --- a/source/clear-linux/tutorials/tutorials.rst +++ b/source/clear-linux/tutorials/tutorials.rst @@ -18,4 +18,5 @@ specific |CLOSIA| use cases. fmv aws-web/aws-web telemetry-backend/telemetry-backend - smb-file-sharing/smb-file-sharing + smb/smb + spark