Revert "Merge of latest changes into RTD theme to enable multi-language support (#604)"
This reverts commit 186f1e02b8.
@@ -0,0 +1,276 @@
|
||||
.. _aws-web:
|
||||
|
||||
Create and launch |CL-ATTR| from Amazon Web Services
|
||||
######################################################
|
||||
|
||||
This tutorial walks you through creating and launching a |CL|
|
||||
:abbr:`AMI (Amazon Machine Image)` instance from the
|
||||
:abbr:`AWS (Amazon Web Services)` console and complete the following tasks:
|
||||
|
||||
#. Locate and select the |CL| OS Basic AMI in the AWS Marketplace.
|
||||
#. Create a new public and private key pair to allow you to connect to your
|
||||
|CL| instance securely.
|
||||
#. Launch the new |CL| instance and connect to it.
|
||||
#. Update your instance of |CL| using the :command:`swupd` command.
|
||||
#. Stop the |CL| instance.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
This tutorial assumes the following statements are true:
|
||||
|
||||
* You are using a linux-based system to access AWS and can run :command:`SSH`
|
||||
to access the remote |CL| AWS image.
|
||||
* Your browser puts downloaded files in the :file:`$HOME/Downloads`
|
||||
directory.
|
||||
* You have already set up an AWS user account and logged into the AWS
|
||||
console.
|
||||
|
||||
.. note::
|
||||
This tutorial uses a |CL| AMI t2.micro instance that is eligible for the
|
||||
AWS free tier. To learn more about AWS and setting up an account, visit the
|
||||
AWS website at http://aws.amazon.com.
|
||||
|
||||
Locate, select, and launch the |CL| Basic AMI
|
||||
*********************************************
|
||||
|
||||
#. Start from your main AWS services console menu in your browser and select the
|
||||
:guilabel:`EC2` text as shown in figure 1:
|
||||
|
||||
.. figure:: figures/aws-web-1.png
|
||||
:scale: 50 %
|
||||
:alt: AWS Console
|
||||
|
||||
Figure 1: :guilabel:`AWS Console`
|
||||
|
||||
This selection brings up your :guilabel:`EC2 Dashboard` menu.
|
||||
|
||||
#. To create a new |CL| instance from the :guilabel:`EC2 Dashboard` menu,
|
||||
select the :guilabel:`Launch Instance` button as shown in figure 2:
|
||||
|
||||
.. figure:: figures/aws-web-2.png
|
||||
:scale: 50 %
|
||||
:alt: EC2 Dashboard
|
||||
|
||||
Figure 2: :guilabel:`EC2 Dashboard`
|
||||
|
||||
This selection takes you to the
|
||||
:guilabel:`Step 1: Choose an Amazon Machine Image (AMI)` menu.
|
||||
|
||||
#. To find the :guilabel:`Clear Linux OS Basic` AMI in the
|
||||
:guilabel:`Step 1: Choose an Amazon Machine Image (AMI)` menu, do the
|
||||
following:
|
||||
|
||||
#. In the lefthand navigation window, select the
|
||||
:guilabel:`AWS Marketplace` menu item to bring up the search bar to
|
||||
:guilabel:`Search AWS Marketplace Products`.
|
||||
|
||||
#. In the search bar, type "clear linux os" and hit the :kbd:`Enter` key to
|
||||
search for and locate the :guilabel:`Clear Linux OS Basic` AMI.
|
||||
|
||||
#. Select the :guilabel:`Clear Linux OS Basic` AMI by clicking on the
|
||||
:guilabel:`Select` button as shown in figure 3:
|
||||
|
||||
.. figure:: figures/aws-web-3.png
|
||||
:scale: 50 %
|
||||
:alt: Step 1: Choose AMI
|
||||
|
||||
Figure 3: :guilabel:`Step 1: Choose AMI`
|
||||
|
||||
#. A pop-up dialog box appears showing you more information about the
|
||||
:guilabel:`Clear Linux OS Basic` AMI along with the pricing details for running
|
||||
|CL| on different platform configurations as shown in figure 4. Select
|
||||
the :guilabel:`Continue` button.
|
||||
|
||||
.. figure:: figures/aws-web-4.png
|
||||
:scale: 50 %
|
||||
:alt: Clear Linux OS Basic
|
||||
|
||||
Figure 4: :guilabel:`Clear Linux OS Basic`
|
||||
|
||||
#. The :guilabel:`Choose Instance Type` menu appears as shown in figure 5.
|
||||
|
||||
.. figure:: figures/aws-web-5.png
|
||||
:scale: 50 %
|
||||
:alt: Choose an Instance Type
|
||||
|
||||
Figure 5: :guilabel:`Choose an Instance Type`
|
||||
|
||||
Select the :guilabel:`t2.micro` type by clicking the box on the left side
|
||||
of the instance and then select the :guilabel:`Review and Launch` button to
|
||||
move to the :guilabel:`Step 7: Review the Instance Launch` menu.
|
||||
|
||||
.. note::
|
||||
|
||||
You can configure the instance details, add additional storage, add
|
||||
tags, and configure the security group before selecting the
|
||||
:guilabel:`Review and Launch` button if you want to further customize
|
||||
this |CL| instance.
|
||||
|
||||
#. The :guilabel:`Step 7: Review the Instance Launch` menu, shown in figure 6,
|
||||
allows you to
|
||||
:guilabel:`Cancel` the process, return to the :guilabel:`Previous` screen
|
||||
to change the configuration or :guilabel:`Launch` the instance defined.
|
||||
|
||||
.. figure:: figures/aws-web-6.png
|
||||
:scale: 50 %
|
||||
:alt: Step 7: Review the Instance Launch
|
||||
|
||||
Figure 6: :guilabel:`Step 7: Review the Instance Launch`
|
||||
|
||||
#. Select the :guilabel:`Launch` button. A dialog box appears, as shown in
|
||||
figure 7, asking you to
|
||||
:guilabel:`Select an existing key pair or create a new pair`.
|
||||
|
||||
.. figure:: figures/aws-web-7.png
|
||||
:scale: 50 %
|
||||
:alt: Select an existing key pair or create a new pair
|
||||
|
||||
Figure 7: :guilabel:`Select an existing key pair or create a new pair`
|
||||
|
||||
#. Select the :guilabel:`Create a new key pair` option.
|
||||
|
||||
#. For the :guilabel:`Key pair name` field, enter `AWSClearTestKey`.
|
||||
|
||||
#. Select the :guilabel:`Download Key Pair` button to download the
|
||||
:file:`AWSClearTestKey.pem` to your browser's defined
|
||||
:file:`Downloads` directory.
|
||||
|
||||
#. When the file finishes downloading, select the
|
||||
:guilabel:`Launch Instances` button to proceed to the
|
||||
:guilabel:`Launch Status` menu shown in figure 8.
|
||||
|
||||
.. figure:: figures/aws-web-8.png
|
||||
:scale: 50 %
|
||||
:alt: Launch Status
|
||||
|
||||
Figure 8: :guilabel:`Launch Status`
|
||||
|
||||
#. Once the :guilabel:`Launch Status` page changes to what is shown in
|
||||
figure 9, select the :guilabel:`View Instances` button to view your
|
||||
:guilabel:`Instances` dashboard.
|
||||
|
||||
.. figure:: figures/aws-web-9.png
|
||||
:scale: 50 %
|
||||
:alt: View Instance
|
||||
|
||||
Figure 9: :guilabel:`View Instance`
|
||||
|
||||
Connect to your Clear Linux OS Basic instance
|
||||
*********************************************
|
||||
|
||||
Your :guilabel:`Instances` Dashboard is shown in figure 10 with the new |CL|
|
||||
OS basic instance already selected and in the running state. If there are
|
||||
other instances available, they are also listed but not selected.
|
||||
|
||||
.. figure:: figures/aws-web-10.png
|
||||
:scale: 50 %
|
||||
:alt: Instance Dashboard
|
||||
|
||||
Figure 10: :guilabel:`Instance Dashboard`
|
||||
|
||||
#. To connect to your running instance, click the :guilabel:`Connect` button
|
||||
located at the top of your dashboard. AWS brings up the pop-up dialog
|
||||
box shown in figure 11 describing how to connect to your running instance.
|
||||
|
||||
.. _fig-aws-web-11:
|
||||
|
||||
.. figure:: figures/aws-web-11.png
|
||||
:scale: 50 %
|
||||
:alt: Connect to Your Instance
|
||||
|
||||
Figure 11: :guilabel:`Connect to Your Instance`
|
||||
|
||||
#. Open a terminal on your system. You should be in your :file:`$HOME`
|
||||
directory.
|
||||
|
||||
#. Copy the previously downloaded keyfile from the :file:`Downloads`
|
||||
directory to the current directory.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
cp Downloads/AWSClearTestKey.pem .
|
||||
|
||||
#. Change the attributes of the :file:`AWSClearTestKey.pem` using the
|
||||
:command:`chmod` command as instructed in the dialog box shown in figure
|
||||
11.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
chmod 400 AWSClearTestKey.pem
|
||||
|
||||
#. Copy the text highlighted in the :guilabel:`Example:` section that is
|
||||
shown in :ref:`figure 11<fig-aws-web-11>`. Paste the copied text into your
|
||||
terminal, change the text before the `@` sign to the username `clear`, and
|
||||
press the :kbd:`Enter` key to execute the command.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ssh -i "AWSClearTestKey.pem" clear@ec2-34-209-39-184.us-west-2.compute.amazonaws.com
|
||||
|
||||
#. A message appears on the terminal stating the authenticity of the host can't
|
||||
be established and prompts you with the message:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
The authenticity of host 'ec2-34-209-39-184.us-west-2.compute.amazonaws.com (34.209.39.184)' can't be established.
|
||||
ECDSA key fingerprint is SHA256:LrziT5Ar66iBTfia8qmiIsrfBUm/UGam76U8bDR6yJc.
|
||||
Are you sure you want to continue connecting (yes/no)?
|
||||
|
||||
#. Type `yes` and hit the :kbd:`Enter` key. Another warning is printed to the
|
||||
terminal and you are now at the command prompt of your new |CL| instance.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
Warning: Permanently added 'ec2-34-209-39-184.us-west-2.compute.amazonaws.com,34.209.39.184' (ECDSA) to the list of known hosts.
|
||||
clear@clr-96a8565d0ca54b0c80364a1e5e7b0f88 ~ $
|
||||
|
||||
Update the |CL| instance
|
||||
************************
|
||||
|
||||
Run the :command:`sudo swupd update` command to update the operating
|
||||
system as shown in figure 12:
|
||||
|
||||
.. figure:: figures/aws-web-12.png
|
||||
:scale: 50 %
|
||||
:alt: sudo swupd update
|
||||
|
||||
Figure 12: :guilabel:`sudo swupd update`
|
||||
|
||||
In this example we updated from version 18940 to 19100.
|
||||
|
||||
Stop the |CL| instance
|
||||
**********************
|
||||
|
||||
When you are finished using your AWS |CL| instance you need to stop it using
|
||||
the :guilabel:`Instances` dashboard to stop accruing charges. Complete the
|
||||
following steps from the :guilabel:`Instances` dashboard to stop your AWS |CL|
|
||||
instance from running.
|
||||
|
||||
#. Select the :guilabel:`Actions` button to bring up a pull-down menu.
|
||||
|
||||
#. Select the :guilabel:`Instance State` menu item to expand the options.
|
||||
|
||||
#. Select :guilabel:`Stop` menu item to shut down the running instance.
|
||||
|
||||
Figure 13 illustrates these steps.
|
||||
|
||||
.. figure:: figures/aws-web-13.png
|
||||
:scale: 50 %
|
||||
:alt: Stop Instance
|
||||
|
||||
Figure 13: :guilabel:`Stop Instance`
|
||||
|
||||
#. A pop-up dialog box appears warning you that any ephemeral storage of
|
||||
your instance will be lost. Select the :guilabel:`Yes, Stop` button to stop
|
||||
your |CL| instance.
|
||||
|
||||
.. figure:: figures/aws-web-14.png
|
||||
:scale: 50 %
|
||||
:alt: Stop Instances
|
||||
|
||||
Figure 14: :guilabel:`Stop Instances`
|
||||
|
||||
Congratulations! You are up and running with |CL| on AWS. To see what you
|
||||
can do with your |CL| instance, visit our :ref:`tutorials <tutorials>`
|
||||
section for examples on using your |CL| system.
|
||||
|
After Width: | Height: | Size: 255 KiB |
|
After Width: | Height: | Size: 213 KiB |
|
After Width: | Height: | Size: 307 KiB |
|
After Width: | Height: | Size: 60 KiB |
|
After Width: | Height: | Size: 221 KiB |
|
After Width: | Height: | Size: 275 KiB |
|
After Width: | Height: | Size: 295 KiB |
|
After Width: | Height: | Size: 161 KiB |
|
After Width: | Height: | Size: 208 KiB |
|
After Width: | Height: | Size: 272 KiB |
|
After Width: | Height: | Size: 207 KiB |
|
After Width: | Height: | Size: 277 KiB |
|
After Width: | Height: | Size: 77 KiB |
|
After Width: | Height: | Size: 213 KiB |
@@ -0,0 +1,544 @@
|
||||
.. _azure:
|
||||
|
||||
Run |CL-ATTR| using Microsoft Azure CLI 2.0
|
||||
###########################################
|
||||
|
||||
|CL-ATTR| is available for you to use in the Microsoft* Azure* marketplace and
|
||||
is offered with three different images, also known as a
|
||||
:abbr:`SKU (Stock Keeping Unit)`.
|
||||
|
||||
* |CL| Basic - This SKU consists of a bare-bones system which can be used as a
|
||||
starting point for those wanting to explore and build out a system with
|
||||
additional software bundles of their choosing.
|
||||
|
||||
* |CL| Containers - This SKU comes with the containers-basic software bundle
|
||||
already installed.
|
||||
|
||||
* |CL| Machine-learning - This SKU comes pre-loaded with popular open source
|
||||
tools for developing machine learning applications.
|
||||
|
||||
You can access these images directly from your MS Azure dashboard through the
|
||||
`Azure portal`_ or by using the MS Azure :abbr:`CLI (Command Line Interface)`
|
||||
2.0. If you do not already have an account set up with MS Azure, you can sign
|
||||
up for a `MS Azure free account`_ to access the |CL|
|
||||
:abbr:`VM(Virtual Machine)` images.
|
||||
|
||||
The Azure CLI offers the ability to create and manage resources in MS Azure
|
||||
from the command line. In this tutorial you learn to:
|
||||
|
||||
#. Install the latest MS Azure CLI on your |CL| machine.
|
||||
|
||||
#. Log into MS Azure using the CLI 2.0 interface.
|
||||
|
||||
#. Create a MS Azure resource group.
|
||||
|
||||
#. Locate the |CL| images.
|
||||
|
||||
#. Create and log into the |CL| virtual machine.
|
||||
|
||||
#. Stop and deallocate the |CL| VM and resources.
|
||||
|
||||
To use the MS Azure CLI 2.0 on your |CL| system, your system must have the
|
||||
following packages installed:
|
||||
|
||||
* Python 2.7 or later
|
||||
|
||||
* libffi
|
||||
|
||||
* OpenSSL 1.0.2
|
||||
|
||||
You can check to see what versions you have installed on your system by
|
||||
running the individual commands as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python --version
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
python 2.7.12
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
openssl version
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
OpenSSL 1.0.2n 7 Dec 2017
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ls -l /usr/lib64/libffi*
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
lrwxrwxrwx 1 root root 15 Jan 12 2017 /usr/lib64/libffi.so.6 -> libffi.so.6.0.4
|
||||
-rwxr-xr-x 1 root root 38792 Jan 12 2017 /usr/lib64/libffi.so.6.0.4
|
||||
|
||||
If you do not have these packages installed on your |CL| system, install the
|
||||
sysadmin-basic software bundle using the :command:`swupd` command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add sysadmin-basic
|
||||
|
||||
.. note::
|
||||
|
||||
These instructions are for installing the MS Azure CLI 2.0 tools on a |CL|
|
||||
system. If you are installing the CLI on another platform, follow the
|
||||
instructions in the `MS Azure Install Azure CLI tutorial`_ for your
|
||||
specific operating system.
|
||||
|
||||
Install MS Azure CLI 2.0 on |CL|
|
||||
********************************
|
||||
|
||||
#. To install the MS Azure CLI 2.0 on |CL|, use the :command:`curl` command as
|
||||
follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -L https://aka.ms/InstallAzureCli | bash
|
||||
|
||||
If you get an error message from :command:`curl` related to the -L
|
||||
parameter or an error message is generated that includes the text "Object
|
||||
Moved", use the full URL instead of the aka.ms redirect address:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl https://azurecliprod.blob.core.windows.net/install | bash
|
||||
|
||||
#. The installation script begins and prompts you several times during
|
||||
execution for information.
|
||||
|
||||
.. note::
|
||||
|
||||
The console output from the script displays your username instead of the
|
||||
**[user]** variable shown on this tutorial.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
===> In what directory would you like to place the install? (leave blank to use '/home/[user]/lib/azure-cli'):
|
||||
|
||||
Press the :kbd:`Enter` key to accept the default or you can chose another
|
||||
directory to install the MS Azure CLI 2.0 tools into.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
===> In what directory would you like to place the 'az' executable? (leave blank to use '/home/[user]/bin'):
|
||||
|
||||
Press the :kbd:`Enter` key to accept the default or you can chose another
|
||||
directory to install the :command:`az` executable in.
|
||||
|
||||
#. The installation downloads and builds all required tools and when complete
|
||||
prompts you with:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
===> Modify profile to update your $PATH and enable shell/tab completion now? (Y/n): Y
|
||||
|
||||
Type :kbd:`Y` and press the :kbd:`Enter` key to allow this modification.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
===> Enter a path to an rc file to update (leave blank to use '/home/[user]/.bashrc'):
|
||||
|
||||
Press the :kbd:`Enter` key to accept the default or enter the pathname to
|
||||
your :file:`.bashrc` file. The installation completes with the final output
|
||||
shown below:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
-- Backed up '/home/[user].bashrc' to '/home/[user]/.bashrc.backup'
|
||||
-- Tab completion set up complete.
|
||||
-- If tab completion is not activated, verify that '/home/[user]/.bashrc' is sourced by your shell.
|
||||
--
|
||||
-- ** Run `exec -l $SHELL` to restart your shell. **
|
||||
--
|
||||
-- Installation successful.
|
||||
-- Run the CLI with /home/[user]/bin/az --help
|
||||
|
||||
#. The installation program finishes and you must restart your shell for
|
||||
the changes to take effect. When the installation is successful, run the
|
||||
following command to restart your shell.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
exec -l $SHELL
|
||||
|
||||
With the MS Azure CLI 2.0 executable successfully built and installed, run
|
||||
the :command:`az` command.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
az
|
||||
|
||||
The output from the :command:`az` command is shown below:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
|
||||
/\
|
||||
/ \ _____ _ _ __ ___
|
||||
/ /\ \ |_ / | | | \'__/ _ \
|
||||
/ ____ \ / /| |_| | | | __/
|
||||
/_/ \_\/___|\__,_|_| \___|
|
||||
|
||||
|
||||
Welcome to the cool new Azure CLI!
|
||||
|
||||
Here are the base commands:
|
||||
|
||||
account : Manage Azure subscription information.
|
||||
acr : Manage Azure Container Registries.
|
||||
acs : Manage Azure Container Services.
|
||||
ad : Synchronize on-premises directories and manage Azure Active Directory
|
||||
resources.
|
||||
advisor : (PREVIEW) Manage Azure Advisor.
|
||||
aks : Manage Kubernetes clusters.
|
||||
appservice : Manage App Service plans.
|
||||
backup : Commands to manage Azure Backups.
|
||||
batch : Manage Azure Batch.
|
||||
batchai : Batch AI.
|
||||
billing : Manage Azure Billing.
|
||||
cdn : Manage Azure Content Delivery Networks (CDNs).
|
||||
cloud : Manage registered Azure clouds.
|
||||
cognitiveservices: Manage Azure Cognitive Services accounts.
|
||||
configure : Display and manage the Azure CLI 2.0 configuration. This command is
|
||||
interactive.
|
||||
consumption : Manage consumption of Azure resources.
|
||||
container : (PREVIEW) Manage Azure Container Instances.
|
||||
cosmosdb : Manage Azure Cosmos DB database accounts.
|
||||
disk : Manage Azure Managed Disks.
|
||||
dla : (PREVIEW) Manage Data Lake Analytics accounts, jobs, and catalogs.
|
||||
dls : (PREVIEW) Manage Data Lake Store accounts and filesystems.
|
||||
eventgrid : Manage Azure Event Grid topics and subscriptions.
|
||||
extension : Manage and update CLI extensions.
|
||||
feature : Manage resource provider features.
|
||||
feedback : Loving or hating the CLI? Let us know!
|
||||
find : Find Azure CLI commands.
|
||||
functionapp : Manage function apps.
|
||||
group : Manage resource groups and template deployments.
|
||||
image : Manage custom virtual machine images.
|
||||
interactive : Start interactive mode.
|
||||
iot : (PREVIEW) Manage Internet of Things (IoT) assets.
|
||||
keyvault : Safeguard and maintain control of keys, secrets, and certificates.
|
||||
lab : Manage Azure DevTest Labs.
|
||||
lock : Manage Azure locks.
|
||||
login : Log in to Azure.
|
||||
logout : Log out to remove access to Azure subscriptions.
|
||||
managedapp : Manage template solutions provided and maintained by Independent Software
|
||||
Vendors (ISVs).
|
||||
monitor : Manage the Azure Monitor Service.
|
||||
mysql : Manage Azure Database for MySQL servers.
|
||||
network : Manage Azure Network resources.
|
||||
policy : Manage resource policies.
|
||||
postgres : Manage Azure Database for PostgreSQL servers.
|
||||
provider : Manage resource providers.
|
||||
redis : Access to a secure, dedicated Redis cache for your Azure applications.
|
||||
reservations : Manage Azure Reservations.
|
||||
resource : Manage Azure resources.
|
||||
role : Manage user roles for access control with Azure Active Directory and service
|
||||
principals.
|
||||
sf : Manage and administer Azure Service Fabric clusters.
|
||||
snapshot : Manage point-in-time copies of managed disks, native blobs, or other
|
||||
snapshots.
|
||||
sql : Manage Azure SQL Databases and Data Warehouses.
|
||||
storage : Manage Azure Cloud Storage resources.
|
||||
tag : Manage resource tags.
|
||||
vm : Provision Linux or Windows virtual machines.
|
||||
vmss : Manage groupings of virtual machines in an Azure Virtual Machine Scale Set
|
||||
(VMSS).
|
||||
webapp : Manage web apps.
|
||||
|
||||
Log into your Microsoft Azure account
|
||||
*************************************
|
||||
|
||||
#. With the :command:`az` command properly installed and functional, login to
|
||||
your MS Azure account using the :command:`az login` command shown below:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
az login
|
||||
|
||||
The output from this command is:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code BQ9MG442B to authenticate.
|
||||
|
||||
#. Open your browser and enter the page `https://aka.ms/devicelogin` as shown
|
||||
in figure 1:
|
||||
|
||||
.. figure:: azure/figures/azure-1.png
|
||||
:scale: 50 %
|
||||
:alt: Microsoft Device Login
|
||||
|
||||
Figure 1: :guilabel:`Microsoft Device Login`
|
||||
|
||||
#. Enter the code `BQ9MG442B` to authenticate your device as shown in figure
|
||||
2. The code `BQ9MG442B` is a random authentication code generated for each
|
||||
session login and will be different each time you log into MS Azure using
|
||||
the :command:`az login` command.
|
||||
|
||||
.. figure:: azure/figures/azure-2.png
|
||||
:scale: 50 %
|
||||
:alt: Microsoft Device Login - Azure CLI
|
||||
|
||||
Figure 2: :guilabel:`Microsoft Device Login - Azure CLI`
|
||||
|
||||
#. Once you enter the authentication code, the website changes to a screen to
|
||||
enter your existing Microsoft Azure credentials.
|
||||
|
||||
#. Log in with your existing MS Azure account credentials. Once complete, the
|
||||
browser screen changes again as shown in figure 3. The screen shows you
|
||||
have signed into the Microsoft Cross-platform Command Line Interface
|
||||
application on your device and you can close the window.
|
||||
|
||||
.. figure:: azure/figures/azure-3.png
|
||||
:scale: 50 %
|
||||
:alt: Microsoft Azure Cross-platform CLI
|
||||
|
||||
Figure 3: :guilabel:`Microsoft Azure Cross-platform CLI`
|
||||
|
||||
The MS Azure CLI 2.0 interface is now active using your existing MS Azure
|
||||
account credentials.
|
||||
|
||||
Create a MS Azure resource group
|
||||
********************************
|
||||
|
||||
To learn more about MS Azure resource groups, visit the
|
||||
`Azure Resource Manager overview`_ for an overview and detailed description
|
||||
of resources within MS Azure.
|
||||
|
||||
#. To create our new resource group, run the :command:`az group create ...`
|
||||
command shown below to create a resource group named `ClearResourceGroup`
|
||||
using the `-n` parameter and locate it in the `westus` region using the
|
||||
`-l` parameter.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
az group create -n ClearResourceGroup -l westus
|
||||
|
||||
#. When the command has completed, the output from this command is similar to
|
||||
the following:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
{
|
||||
"id": "/subscriptions/{unique-id}/resourceGroups/ClearResourceGroup",
|
||||
"location": "westus",
|
||||
"managedBy": null,
|
||||
"name": "ClearResourceGroup",
|
||||
"properties": {
|
||||
"provisioningState": "Succeeded"
|
||||
},
|
||||
"tags": null
|
||||
}
|
||||
|
||||
Create and log into the |CL| virtual machine
|
||||
********************************************
|
||||
|
||||
For this tutorial, we are using the |CL| Basic SKU for our VM.
|
||||
|
||||
#. To locate the available |CL| Basic VM SKU images in the MS Azure
|
||||
marketplace, run the :command:`az vm image ...` command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
az vm image list --offer clear-linux --sku basic --all --output table
|
||||
|
||||
This command may take some time to finish. The output lists all available
|
||||
|CL| Basic images available in the Microsoft Azure marketplace as shown
|
||||
below:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
Offer Publisher Sku Urn Version
|
||||
-------------- ------------------- ---------------- ------------------------------------------------------------- ---------
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:15780.0.0 15780.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:15960.0.0 15960.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:16050.0.0 16050.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:16150.0.0 16150.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:16500.0.0 16500.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:16810.0.0 16810.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:18080.0.0 18080.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:18620.0.0 18620.0.0
|
||||
clear-linux-os clear-linux-project basic clear-linux-project:clear-linux-os:basic:18860.0.0 18860.0.0
|
||||
|
||||
.. note::
|
||||
|
||||
The output list shows current offerings. New versions are added to the
|
||||
MS Azure marketplace all the time. To reference the latest version of an
|
||||
image, you can use the version label `latest` when specifying an image.
|
||||
|
||||
#. The information shown in the `Urn` column lists the
|
||||
`Publisher:Offer:Sku:Version` for each image available and this is the
|
||||
information we want to create the |CL| Basic VM. Since we are creating a
|
||||
|CL| Basic VM, highlight the `clear-linux-project:clear-linux-os:basic:`
|
||||
string and copy it to your clipboard. Use the label
|
||||
`latest` instead of referencing a specific version.
|
||||
|
||||
#. Create the new |CL| Basic VM. Run the :command:`az vm create ...`
|
||||
command using the URN `:clear-linux-project:clear-linux-os:basic:latest`
|
||||
that we copied to the clipboard on the previous step.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
az vm create --resource-group ClearResourceGroup --name ClearVM --image clear-linux-project:clear-linux-os:basic:latest --generate-ssh-keys
|
||||
|
||||
.. note::
|
||||
|
||||
If you have already defined your public/private SSH key pair and they
|
||||
are stored in your :file:`$HOME/.ssh` directory, you do not need to
|
||||
include the *--generate-ssh-keys* option.
|
||||
|
||||
Your output from this command will look similar to this output, where
|
||||
[user] is your user name:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
SSH key files '/home/[user]/.ssh/id_rsa' and '/home/[user]/.ssh/id_rsa.pub' have been generated under ~/.ssh to allow SSH access to the VM. If using machines without permanent storage, back up your keys to a safe location.
|
||||
|
||||
running...
|
||||
|
||||
{
|
||||
"fqdns": "",
|
||||
"id": "/subscriptions/{unique-id}/resourceGroups/ClearResourceGroup/providers/Microsoft.Compute/virtualMachines/ClearVM",
|
||||
"location": "westus",
|
||||
"macAddress": "00-0D-3A-37-C7-59",
|
||||
"powerState": "VM running",
|
||||
"privateIpAddress": "10.0.0.4",
|
||||
"publicIpAddress": "13.91.4.245",
|
||||
"resourceGroup": "ClearResourceGroup",
|
||||
"zones": ""
|
||||
}
|
||||
|
||||
Take note of the public IP address from the output.
|
||||
|
||||
#. Login into the new |CL| Basic VM, run the :command:`ssh` command with the
|
||||
public IP address obtained from the previous step as shown:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ssh [user]@13.91.4.245
|
||||
|
||||
You may see the following message about the authenticity of the host. If
|
||||
this appears, type `yes` to proceed connecting to your new |CL| VM.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
The authenticity of host '13.91.4.245 (13.91.4.245)' can't be established.
|
||||
RSA key fingerprint is SHA256:{unique-number}.
|
||||
Are you sure you want to continue connecting (yes/no)? yes
|
||||
Warning: Permanently added '13.91.4.245' (RSA) to the list of known hosts.
|
||||
|
||||
[user]@ClearVM ~ $
|
||||
|
||||
You are now logged into your new |CL| VM as [user], where [user] is your
|
||||
user name. To check which software bundles are included with
|
||||
this VM image, run the :command:`sudo swupd bundle-list` command inside the VM:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-list
|
||||
|
||||
The output shown should be similar to:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
swupd-client bundle list 3.14.1
|
||||
Copyright (C) 2012-2017 Intel Corporation
|
||||
|
||||
bootloader
|
||||
editors
|
||||
kernel-hyperv
|
||||
network-basic
|
||||
openssh-server
|
||||
os-cloudguest-azure
|
||||
os-core
|
||||
os-core-update
|
||||
perl-basic
|
||||
python-basic
|
||||
python3-basic
|
||||
storage-utils
|
||||
sysadmin-basic
|
||||
Current OS version: 19600
|
||||
|
||||
When you are finished using your new |CL| VM, type :command:`exit` to close
|
||||
the :command:`SSH` terminal and logout.
|
||||
|
||||
Stop and deallocate the |CL| VM and resources
|
||||
*********************************************
|
||||
|
||||
When you finish using your new |CL| instance, you must stop the VM and
|
||||
deallocate the resources in your resource group. If you only stop a VM, the OS
|
||||
image shuts down but the resources associated with it in your resource group
|
||||
remain allocated and incurring charges. For instance, if you stop and then
|
||||
later start the VM using the :command:`az vm start...` without deallocating
|
||||
the resources, the IP address is retained and you can access the VM using that
|
||||
same IP address. To release the resources associated with the VM and stop
|
||||
incurring charges for them, you must deallocate the resources as well.
|
||||
|
||||
#. At the command prompt, enter the :command:`az vm stop...` command as
|
||||
follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
az vm stop --resource-group ClearResourceGroup --name ClearVM
|
||||
|
||||
This will stop the VM and then output text similar to:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
{
|
||||
"endTime": "2017-12-13T23:04:02.346676+00:00",
|
||||
"error": null,
|
||||
"name": "{unique-name}",
|
||||
"startTime": "2017-12-13T23:03:59.018536+00:00",
|
||||
"status": "Succeeded"
|
||||
}
|
||||
|
||||
#. Once the VM stops, deallocate the VM resources to stop incurring
|
||||
charges for the |CL| instance. Enter the following command:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
az vm deallocate --resource-group ClearResourceGroup --name ClearVM
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You are up and running with |CL| on MS Azure using the Azure
|
||||
CLI 2.0 command line tools.
|
||||
|
||||
Next steps
|
||||
**********
|
||||
|
||||
To see use cases you can fulfill with your |CL| instance, visit our
|
||||
:ref:`tutorials <tutorials>` section.
|
||||
|
||||
For additional information visit the |CL|
|
||||
`Azure Partner Mini Case Study`_ and the `Azure Partner Datasheet`_.
|
||||
|
||||
To learn more about the MS Azure CLI 2.0 tool and options that are available,
|
||||
visit the `MS Azure documentation and tutorials`_ website.
|
||||
|
||||
.. _`Azure Portal`:
|
||||
https://portal.azure.com
|
||||
|
||||
.. _`MS Azure free account`:
|
||||
https://azure.microsoft.com/en-us/free/
|
||||
|
||||
.. _`MS Azure documentation and tutorials`:
|
||||
https://docs.microsoft.com/en-us/cli/azure/overview?view=azure-cli-latest
|
||||
|
||||
.. _`MS Azure Install Azure CLI tutorial`:
|
||||
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
|
||||
|
||||
.. _`Azure Resource Manager overview`:
|
||||
https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview
|
||||
|
||||
.. _Azure Partner Datasheet:
|
||||
http://download.microsoft.com/download/D/9/E/D9E22342-96D9-4455-BB15-99A1AF514DDD/Microsoft%20Azure%20Partner%20Datasheet%20-%20Intel%20Clear%20Linux.pdf
|
||||
|
||||
.. _Azure Partner Mini Case Study:
|
||||
http://download.microsoft.com/download/D/9/E/D9E22342-96D9-4455-BB15-99A1AF514DDD/Microsoft%20Azure%20Partner%20Mini%20Case%20Study%20-%20Intel%20Clear%20Linux.pdf
|
||||
|
After Width: | Height: | Size: 116 KiB |
|
After Width: | Height: | Size: 133 KiB |
|
After Width: | Height: | Size: 128 KiB |
@@ -0,0 +1,106 @@
|
||||
.. _dars:
|
||||
|
||||
Data Analytics Reference Stack
|
||||
##############################
|
||||
|
||||
This tutorial shows you how to use the Data Analytics Reference Stack
|
||||
(DARS), and to optionally build your own images with the baseline Dockerfiles
|
||||
provided in the `DARS repository`_. Our assumption is that |CL-ATTR| is the
|
||||
host. However, any system that supports Docker\* containers can be used to
|
||||
follow these steps.
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
The Data Analytics Reference Stack release
|
||||
******************************************
|
||||
|
||||
The Data Analytics Reference Stack (DARS) provides developers and enterprises a straightforward, highly optimized software stack for storing and processing large amounts of data. More detail is available on the `DARS architecture and performance benchmarks`_.
|
||||
|
||||
The Data Analytics Reference Stack provides two pre-built Docker images, available on `Docker Hub`_:
|
||||
|
||||
* A |CL|-derived `DARS with OpenBlas`_ stack optimized for `OpenBLAS`_
|
||||
* A |CL|-derived `DARS with MKL`_ stack optimized for `MKL`_
|
||||
|
||||
We recommend you view the latest component versions for each image in the
|
||||
:file:`README` found in the `DARS repository`_. Because |CL| is a rolling
|
||||
distribution, the package version numbers in the |CL|-based containers may
|
||||
not be the latest released by |CL|.
|
||||
|
||||
|
||||
|
||||
Using the Docker Images
|
||||
***********************
|
||||
|
||||
To immediately start using the latest stable DARS images, pull directly
|
||||
from `Docker Hub`_. For this tutorial we'll use the `Dars with MKL`_ version of the stack.
|
||||
|
||||
Once you have downloaded the image, you can run it with
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker run -it --ulimit nofile=1000000:1000000 --name mkl <name of image>
|
||||
|
||||
This will launch the image and drop you into a bash shell inside the
|
||||
container. You will see output similar to the following:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
root@fd5155b89857 /root # spark-shell
|
||||
spark-shell
|
||||
Config directory: /usr/share/defaults/spark/
|
||||
Welcome to
|
||||
____ __
|
||||
/ __/__ ___ _____/ /__
|
||||
_\ \/ _ \/ _ `/ __/ '_/
|
||||
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
|
||||
/_/
|
||||
|
||||
Using Scala version 2.12.7 (OpenJDK 64-Bit Server VM, Java 1.8.0-internal)
|
||||
Type in expressions to have them evaluated.
|
||||
Type :help for more information.
|
||||
|
||||
scala>
|
||||
|
||||
The :command:`--ulimit nofile` parameter is currently required in order to
|
||||
increase the number of open files opened at certain point by the spark
|
||||
engine.
|
||||
|
||||
Building DARS Images
|
||||
********************
|
||||
|
||||
If you choose to build your own DARS container images, you can customize
|
||||
them as needed. Use the provided Dockerfile as a baseline. To construct
|
||||
images with |CL|, start with a |CL| development platform that
|
||||
has the :command:`containers-basic-dev` bundle installed. Learn more about
|
||||
bundles and installing them by using :ref:`swupd-guide`.
|
||||
|
||||
First, clone the `DARS repository`_ from GitHub.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/clearlinux/dockerfiles/tree/master/stacks/dars -b master
|
||||
|
||||
Then, inside the DARS directory, run :command:`make` to build OpenBLAS and
|
||||
MKL images, and run :command:`make baseline` to build the baseline CentOS
|
||||
image. Depending on the system, it may take a while to finish building.
|
||||
Once completed, check the resulting images with :command:`Docker`
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker images | grep dars
|
||||
|
||||
You can use any of the resulting images to launch fully functional
|
||||
containers. If you need to customize the containers, you can edit the
|
||||
provided :file:`Dockerfile`.
|
||||
|
||||
|
||||
.. _DARS repository: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dars
|
||||
.. _Docker Hub: https://hub.docker.com/
|
||||
.. _OpenBLAS: http://www.openblas.net/
|
||||
.. _MKL: https://software.intel.com/en-us/mkl
|
||||
.. _CentOS: https://www.centos.org/
|
||||
.. _DARS with OpenBLAS: https://hub.docker.com/r/clearlinux/stacks-dars-openblas/
|
||||
.. _DARS with MKL: https://hub.docker.com/r/clearlinux/stacks-dars-mkl/
|
||||
.. _DARS architecture and performance benchmarks: https://clearlinux.org/stacks/data-analytics-stack-v1
|
||||
@@ -0,0 +1,446 @@
|
||||
.. _dlrs:
|
||||
|
||||
Deep Learning Reference Stack
|
||||
#############################
|
||||
|
||||
This tutorial describes how to run benchmarking workloads for TensorFlow\*,
|
||||
PyTorch\*, and Kubeflow in |CL-ATTR| using the Deep Learning Reference Stack.
|
||||
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
Overview
|
||||
********
|
||||
|
||||
We created the Deep Learning Reference Stack to help AI developers deliver the
|
||||
best experience on Intel® Architecture. This stack reduces complexity common
|
||||
with deep learning software components, provides flexibility for customized
|
||||
solutions, and enables you to quickly prototype and deploy Deep Learning
|
||||
workloads. Use this tutorial to run benchmarking workloads on your solution.
|
||||
|
||||
The Deep Learning Reference Stack is available in the following versions:
|
||||
|
||||
* `Intel MKL-DNN-VNNI`_, which is optimized using Intel® Math Kernel Library
|
||||
for Deep Neural Networks (Intel® MKL-DNN) primitives and introduces support
|
||||
for Intel® AVX-512 Vector Neural Network Instructions (VNNI).
|
||||
* `Intel MKL-DNN`_, which includes the TensorFlow framework optimized using
|
||||
Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) primitives.
|
||||
* `Eigen`_, which includes `TensorFlow`_ optimized for Intel® architecture.
|
||||
* `PyTorch with OpenBLAS`_, which includes PyTorch with OpenBlas.
|
||||
* `PyTorch with Intel MKL-DNN`_, which includes PyTorch optimized using Intel®
|
||||
Math Kernel Library (Intel® MKL) and Intel MKL-DNN.
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
To take advantage of the Intel® AVX-512 and VNNI functionality with the Deep
|
||||
Learning Reference Stack, you must use the following hardware:
|
||||
|
||||
* Intel® AVX-512 images require an Intel® Xeon® Scalable Platform
|
||||
* VNNI requires a 2nd generation Intel® Xeon® Scalable Platform
|
||||
|
||||
|
||||
Stack features
|
||||
==============
|
||||
|
||||
* Deep Learning Reference Stack `V3.0 release announcement`_.
|
||||
* Deep Learning Reference Stack v2.0 including current `PyTorch benchmark results`_.
|
||||
* Deep Learning Reference Stack v1.0 including current `TensorFlow benchmark results`_.
|
||||
* `Release notes on Github\*`_ for the latest release of Deep Learning Reference Stack.
|
||||
|
||||
.. note::
|
||||
|
||||
Performance test results for the Deep Learning Reference Stack were
|
||||
obtained using `runc` as the runtime.
|
||||
|
||||
Prerequisites
|
||||
=============
|
||||
|
||||
* :ref:`Install <bare-metal-install-desktop>` |CL| on your host system.
|
||||
* :command:`containers-basic` bundle
|
||||
* :command:`cloud-native-basic` bundle
|
||||
|
||||
In |CL|, :command:`containers-basic` includes Docker\*, which is required for
|
||||
TensorFlow and PyTorch benchmarking. Use the :command:`swupd` utility to
|
||||
check if :command:`containers-basic` and :command:`cloud-native-basic` are present:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-list
|
||||
|
||||
To install the :command:`containers-basic` or :command:`cloud-native-basic` bundles, enter:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add containers-basic cloud-native-basic
|
||||
|
||||
Docker is not started upon installation of the :command:`containers-basic`
|
||||
bundle. To start Docker, enter:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl start docker
|
||||
|
||||
To ensure that Kubernetes is correctly installed and configured, follow the
|
||||
instructions in :ref:`kubernetes`.
|
||||
|
||||
Version compatibility
|
||||
=====================
|
||||
|
||||
We validated these steps against the following software package versions:
|
||||
|
||||
* |CL| 26240 (Lower version not supported.)
|
||||
* Docker 18.06.1
|
||||
* Kubernetes 1.11.3
|
||||
* Go 1.11.12
|
||||
|
||||
TensorFlow single and multi-node benchmarks
|
||||
*******************************************
|
||||
|
||||
This section describes running the `TensorFlow benchmarks`_ in single node.
|
||||
For multi-node testing, replicate these steps for each node. These steps
|
||||
provide a template to run other benchmarks, provided that they can invoke
|
||||
TensorFlow.
|
||||
|
||||
#. Download either the `Eigen`_ or the `Intel MKL-DNN`_ Docker image
|
||||
from `Docker Hub`_.
|
||||
|
||||
#. Run the image with Docker:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker run --name <image name> --rm -i -t <clearlinux/
|
||||
stacks-dlrs-TYPE> bash
|
||||
|
||||
.. note::
|
||||
|
||||
Launching the Docker image with the :command:`-i` argument starts
|
||||
interactive mode within the container. Enter the following commands in
|
||||
the running container.
|
||||
|
||||
#. Clone the benchmark repository in the container:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone http://github.com/tensorflow/benchmarks -b cnn_tf_v1.12_compatible
|
||||
|
||||
#. Execute the benchmark script:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --device=cpu --model=resnet50 --data_format=NHWC
|
||||
|
||||
.. note::
|
||||
|
||||
You can replace the model with one of your choice supported by the
|
||||
TensorFlow benchmarks.
|
||||
|
||||
PyTorch single and multi-node benchmarks
|
||||
****************************************
|
||||
|
||||
This section describes running the `PyTorch benchmarks`_ for Caffe2 in
|
||||
single node.
|
||||
|
||||
#. Download either the `PyTorch with OpenBLAS`_ or the `PyTorch with Intel
|
||||
MKL-DNN`_ Docker image from `Docker Hub`_.
|
||||
|
||||
#. Run the image with Docker:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker run --name <image name> --rm -i -t <clearlinux/stacks-dlrs-TYPE> bash
|
||||
|
||||
.. note::
|
||||
|
||||
Launching the Docker image with the :command:`-i` argument starts
|
||||
interactive mode within the container. Enter the following commands in
|
||||
the running container.
|
||||
|
||||
#. Clone the benchmark repository:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/pytorch/pytorch.git
|
||||
|
||||
#. Execute the benchmark script:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd pytorch/caffe2/python
|
||||
python convnet_benchmarks.py --batch_size 32 \
|
||||
--cpu \
|
||||
--model AlexNet
|
||||
|
||||
Kubeflow multi-node benchmarks
|
||||
******************************
|
||||
|
||||
The benchmark workload runs in a Kubernetes cluster. The tutorial uses
|
||||
`Kubeflow`_ for the Machine Learning workload deployment on three nodes.
|
||||
|
||||
Kubernetes setup
|
||||
================
|
||||
|
||||
Follow the instructions in the :ref:`kubernetes` tutorial to get set up on
|
||||
|CL|. The Kubernetes community also has
|
||||
`instructions for creating a cluster`_.
|
||||
|
||||
Kubernetes networking
|
||||
=====================
|
||||
|
||||
We used `flannel`_ as the network provider for these tests. If you
|
||||
prefer a different network layer, refer to the Kubernetes
|
||||
`networking documentation`_ for setup.
|
||||
|
||||
Images
|
||||
======
|
||||
|
||||
You must add `launcher.py` to the Docker image to include the Deep
|
||||
Learning Reference Stack and put the benchmarks repo in the correct
|
||||
location. From the Docker image, run the following:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mkdir -p /opt
|
||||
git clone https://github.com/tensorflow/benchmarks.git /opt/tf-benchmarks
|
||||
cp launcher.py /opt
|
||||
chmod u+x /opt/*
|
||||
|
||||
Your entry point becomes: :file:`/opt/launcher.py`
|
||||
|
||||
This builds an image that can be consumed directly by TFJob from Kubeflow.
|
||||
|
||||
ksonnet\*
|
||||
=========
|
||||
|
||||
Kubeflow uses ksonnet\* to manage deployments, so you must install it
|
||||
before setting up Kubeflow.
|
||||
|
||||
ksonnet was added to the :command:`cloud-native-basic` bundle in |CL| version 27550. If
|
||||
you are using an older |CL| version (not recommended), you must manually
|
||||
install ksonnet as described below.
|
||||
|
||||
On |CL|, follow these steps:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
swupd bundle-add go-basic-dev
|
||||
export GOPATH=$HOME/go
|
||||
export PATH=$PATH:$GOPATH/bin
|
||||
go get github.com/ksonnet/ksonnet
|
||||
cd $GOPATH/src/github.com/ksonnet/ksonnet
|
||||
make install
|
||||
|
||||
After the ksonnet installation is complete, ensure that binary `ks` is
|
||||
accessible across the environment.
|
||||
|
||||
Kubeflow
|
||||
========
|
||||
|
||||
Once you have Kubernetes running on your nodes, set up `Kubeflow`_ by
|
||||
following these instructions from the `quick start guide`_.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export KUBEFLOW_SRC=$HOME/kflow
|
||||
export KUBEFLOW_TAG="v0.4.1"
|
||||
export KFAPP="kflow_app"
|
||||
export K8S_NAMESPACE="kubeflow"
|
||||
|
||||
mkdir ${KUBEFLOW_SRC}
|
||||
cd ${KUBEFLOW_SRC}
|
||||
ks init ${KFAPP}
|
||||
cd ${KFAPP}
|
||||
ks registry add kubeflow github.com/kubeflow/kubeflow/tree/${KUBEFLOW_TAG}/kubeflow
|
||||
ks pkg install kubeflow/common
|
||||
ks pkg install kubeflow/tf-training
|
||||
|
||||
Next, deploy the primary package for our purposes: tf-job-operator.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ks env rm default
|
||||
kubectl create namespace ${K8S_NAMESPACE}
|
||||
ks env add default --namespace "${K8S_NAMESPACE}"
|
||||
ks generate tf-job-operator tf-job-operator
|
||||
ks apply default -c tf-job-operator
|
||||
|
||||
This creates the CustomResourceDefinition (CRD) endpoint to launch a TFJob.
|
||||
|
||||
Run a TFJob
|
||||
===========
|
||||
|
||||
#. Select this link for the `ksonnet registries for deploying TFJobs`_.
|
||||
|
||||
#. Install the TFJob components as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ks registry add dlrs-tfjob github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow/dlrs-tfjob
|
||||
|
||||
ks pkg install dlrs-tfjob/dlrs-bench
|
||||
|
||||
#. Export the image name to use for the deployment:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export DLRS_IMAGE=<docker_name>
|
||||
|
||||
.. note::
|
||||
|
||||
Replace <docker_name> with the image name you specified in previous steps.
|
||||
|
||||
#. Generate Kubernetes manifests for the workloads and apply them using these commands:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ks generate dlrs-resnet50 dlrsresnet50 --name=dlrsresnet50 --image=${DLRS_IMAGE}
|
||||
ks generate dlrs-alexnet dlrsalexnet --name=dlrsalexnet --image=${DLRS_IMAGE}
|
||||
ks apply default -c dlrsresnet50
|
||||
ks apply default -c dlrsalexnet
|
||||
|
||||
This replicates and deploys three test setups in your Kubernetes cluster.
|
||||
|
||||
Results of running this tutorial
|
||||
================================
|
||||
|
||||
You must parse the logs of the Kubernetes pod to retrieve performance
|
||||
data. The pods will still exist post-completion and will be in
|
||||
‘Completed’ state. You can get the logs from any of the pods to inspect the
|
||||
benchmark results. More information about `Kubernetes logging`_ is available
|
||||
from the Kubernetes community.
|
||||
|
||||
Use Jupyter Notebook
|
||||
********************
|
||||
|
||||
This example uses the `PyTorch with OpenBLAS`_ container image. After it is
|
||||
downloaded, run the Docker image with :command:`-p` to specify the shared port
|
||||
between the container and the host. This example uses port 8888.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker run --name pytorchtest --rm -i -t -p 8888:8888 clearlinux/stacks-pytorch-oss bash
|
||||
|
||||
After you start the container, launch the Jupyter Notebook. This
|
||||
command is executed inside the container image.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
jupyter notebook --ip 0.0.0.0 --no-browser --allow-root
|
||||
|
||||
After the notebook has loaded, you will see output similar to the following:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
To access the notebook, open this file in a browser: file:///.local/share/jupyter/runtime/nbserver-16-open.html
|
||||
Or copy and paste one of these URLs:
|
||||
http://(846e526765e3 or 127.0.0.1):8888/?token=6357dbd072bea7287c5f0b85d31d70df344f5d8843fbfa09
|
||||
|
||||
From your host system, or any system that can access the host's IP address,
|
||||
start a web browser with the following. If you are not running the browser on
|
||||
the host system, replace :command:`127.0.0.1` with the IP address of the host.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
http://127.0.0.1:8888/?token=6357dbd072bea7287c5f0b85d31d70df344f5d8843fbfa09
|
||||
|
||||
Your browser displays the following:
|
||||
|
||||
.. figure:: figures/dlrs-fig-1.png
|
||||
:scale: 50 %
|
||||
:alt: Jupyter Notebook
|
||||
|
||||
Figure 1: :guilabel:`Jupyter Notebook`
|
||||
|
||||
|
||||
To create a new notebook, click :guilabel:`New` and select :guilabel:`Python 3`.
|
||||
|
||||
.. figure:: figures/dlrs-fig-2.png
|
||||
:scale: 50%
|
||||
:alt: Create a new notebook
|
||||
|
||||
Figure 2: Create a new notebook
|
||||
|
||||
A new, blank notebook is displayed, with a cell ready for input.
|
||||
|
||||
.. figure:: figures/dlrs-fig-3.png
|
||||
:scale: 50%
|
||||
:alt: New blank notebook
|
||||
|
||||
|
||||
To verify that PyTorch is working, copy the following snippet into the blank cell, and run the cell.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
from __future__ import print_function
|
||||
import torch
|
||||
x = torch.rand(5, 3)
|
||||
print(x)
|
||||
|
||||
.. figure:: figures/dlrs-fig-4.png
|
||||
:scale: 50%
|
||||
:alt: Sample code snippet
|
||||
|
||||
When you run the cell, your output will look something like this:
|
||||
|
||||
.. figure:: figures/dlrs-fig-5.png
|
||||
:scale: 50%
|
||||
:alt: code output
|
||||
|
||||
You can continue working in this notebook, or you can download existing
|
||||
notebooks to take advantage of the Deep Learning Reference Stack's optimized
|
||||
deep learning frameworks. Refer to `Jupyter Notebook`_ for details.
|
||||
|
||||
Related topics
|
||||
**************
|
||||
|
||||
* Deep Learning Reference Stack `V3.0 release announcement`_
|
||||
* `TensorFlow benchmarks`_
|
||||
* `PyTorch benchmarks`_
|
||||
* `Kubeflow`_
|
||||
* :ref:`kubernetes` tutorial
|
||||
* `Jupyter Notebook`_
|
||||
|
||||
|
||||
.. _TensorFlow: https://www.tensorflow.org/
|
||||
|
||||
.. _Kubeflow: https://www.kubeflow.org/
|
||||
|
||||
.. _Docker Hub: https://hub.docker.com/
|
||||
|
||||
.. _TensorFlow benchmarks: https://www.tensorflow.org/guide/performance/benchmarks
|
||||
|
||||
.. _PyTorch benchmarks: https://github.com/pytorch/pytorch/blob/master/caffe2/python/convnet_benchmarks.py
|
||||
|
||||
.. _instructions for creating a cluster: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
|
||||
|
||||
.. _flannel: https://github.com/coreos/flannel
|
||||
|
||||
.. _networking documentation: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
|
||||
|
||||
.. _quick start guide: https://www.kubeflow.org/docs/started/getting-started/
|
||||
|
||||
.. _Eigen: https://hub.docker.com/r/clearlinux/stacks-dlrs-oss/
|
||||
|
||||
.. _Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-dlrs-mkl/
|
||||
|
||||
.. _PyTorch with OpenBLAS: https://hub.docker.com/r/clearlinux/stacks-pytorch-oss
|
||||
|
||||
.. _PyTorch with Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-pytorch-mkl
|
||||
|
||||
.. _Intel MKL-DNN-VNNI: https://hub.docker.com/r/clearlinux/stacks-dlrs-mkl-vnni
|
||||
|
||||
.. _V3.0 release announcement: https://clearlinux.org/stacks/deep-learning-reference-stack-v3
|
||||
|
||||
.. _ksonnet registries for deploying TFJobs: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow/dlrs-tfjob
|
||||
|
||||
.. _Kubernetes logging: https://kubernetes.io/docs/concepts/cluster-administration/logging/
|
||||
|
||||
.. _TensorFlow benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack
|
||||
|
||||
.. _PyTorch benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack-pytorch
|
||||
|
||||
.. _Jupyter Notebook: https://jupyter.org/
|
||||
|
||||
.. _Release notes on Github\*: https://github.com/clearlinux/dockerfiles/blob/master/stacks/dlrs/releasenote.md
|
||||
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 51 KiB |
|
After Width: | Height: | Size: 57 KiB |
|
After Width: | Height: | Size: 67 KiB |
|
After Width: | Height: | Size: 81 KiB |
@@ -0,0 +1,247 @@
|
||||
.. _docker:
|
||||
|
||||
Run Docker\* on |CL-ATTR|
|
||||
#########################
|
||||
|
||||
|CL-ATTR| supports multiple containerization platforms, including a Docker\*
|
||||
solution. |CL| has many `unique features`_ including a minimal default
|
||||
installation, which makes it compelling to use as a host for container
|
||||
workloads, management, and orchestration.
|
||||
|
||||
This tutorial covers:
|
||||
|
||||
.. contents:: :local:
|
||||
:depth: 1
|
||||
|
||||
.. note::
|
||||
|
||||
This tutorial focuses on the installaton of the Docker ecosystem.
|
||||
If you want to use |CL| as a Docker container image, refer to the
|
||||
official |CL| container image
|
||||
`published on Docker* Hub <https://hub.docker.com/_/clearlinux/>`_
|
||||
and our guide to :ref:`custom-clear-container`.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
This tutorial assumes you have installed |CL| on your host system.
|
||||
For detailed instructions on installing |CL| on a bare metal system, follow
|
||||
the :ref:`bare metal installation instructions<bare-metal-install-server>`.
|
||||
|
||||
Before you install any new packages, update |CL| with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
Additionally, you should have:
|
||||
|
||||
* A basic understanding of Linux and Docker.
|
||||
|
||||
* |CL| environment that has transparent network access to the Internet.
|
||||
If you are behind a HTTP proxy server, in a corporate setting for example,
|
||||
please refer to the `Docker proxy instructions`_ .
|
||||
|
||||
Install the containers-basic bundle
|
||||
***********************************
|
||||
|
||||
Software in |CL| is offered in the form of `bundles`_ to provide a
|
||||
complete function. The *containers-basic* provides all the required software
|
||||
packages to run Docker images as containers.
|
||||
|
||||
#. First, install the *containers-basic* bundle by running this
|
||||
:command:`swupd` command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add containers-basic
|
||||
|
||||
#. Start the Docker daemon through systemd manager by running this command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl start docker
|
||||
|
||||
If you want Docker to start automatically on boot, enable the
|
||||
systemd service by running this command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl enable docker
|
||||
|
||||
#. Finally, verify :command:`docker` has been installed by running this
|
||||
command and checking the version output for both *client* and *server*:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker version
|
||||
|
||||
Congratulations! At this point, you have a working installation of Docker
|
||||
on |CL|. You are ready to start using container images on your system.
|
||||
|
||||
Integration with Kata Containers (optional)
|
||||
********************************************
|
||||
|
||||
`Kata Containers`_, is an open source project aiming to increase security
|
||||
of containers by using a hardware-backed virtual machine container runtime
|
||||
rather than software namespace containers that are provided by the standard
|
||||
Docker *runc* runtime.
|
||||
|
||||
|CL| provides easy integration of the *kata-runtime* with Docker.
|
||||
More information on installing and using the *kata-runtime* may be found at :ref:`kata`.
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
The proceeding sections of this tutorial are standard to Docker setup
|
||||
and configuration. If you are familiar with Docker basics, you do not
|
||||
need to continue reading. The following sections are provided here for
|
||||
sake of completeness.
|
||||
|
||||
.. _additional-docker-configuration:
|
||||
|
||||
Additional Docker configuration
|
||||
*******************************
|
||||
|
||||
Perform additional Docker daemon configuration via a configuration file
|
||||
typically located at :file:`/etc/docker/daemon.json`. |CL| features a
|
||||
`stateless system`_ so the configuration file :file:`daemon.json` does *NOT*
|
||||
exist by default.
|
||||
|
||||
#. Create the :file:`daemon.json` by running this command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo touch /etc/docker/daemon.json
|
||||
|
||||
.. note::
|
||||
|
||||
Refer to the `Docker documentation on daemon configuration`_ for the
|
||||
full list of available configuration options and examples.
|
||||
|
||||
#. For production systems, we follow Docker's recommendation to use the
|
||||
`OverlayFS storage driver`_ `overlay2`, shown below:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"storage-driver": "overlay2"
|
||||
}
|
||||
|
||||
.. note::
|
||||
|
||||
A testing version is found in `Docker Device Mapper storage driver`_.
|
||||
If using this storage driver, a warning message may appear: "usage of
|
||||
loopback devices is strongly discouraged for production use".
|
||||
|
||||
#. Save and close :file:`daemon.json`.
|
||||
|
||||
#. Once you've made any required changes, be sure to restart the
|
||||
Docker daemon through systemd manager by running this command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl restart docker
|
||||
|
||||
Pulling and running an image from Docker Hub
|
||||
********************************************
|
||||
|
||||
`Docker Hub`_ is a publically available container image repository which
|
||||
comes preconfigured with Docker. In the example below we will pull and run
|
||||
an the official Docker image for nginx, an open source reverse proxy server.
|
||||
|
||||
#. First, pull a container image from Docker Hub using the
|
||||
:command:`docker pull` command. Download the latest nginx Docker
|
||||
container image by running this command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker pull nginx
|
||||
|
||||
#. Create and launch a new container using the :command:`docker run`
|
||||
command. Launch a nginx container by running this command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker run --name test-nginx -d -p 8080:80 nginx
|
||||
|
||||
.. note::
|
||||
|
||||
Below is an explanation of switches used in the command above. For
|
||||
detailed :command:`docker run` switches and syntax, refer to the
|
||||
`Docker Documentation`_ .
|
||||
|
||||
* The *--name* switch lets you provide a friendly name to
|
||||
target the container for future operations
|
||||
|
||||
* The *-d* switch launches the container in the background
|
||||
|
||||
* The *-p* switch allows the container's HTTP port (80) to be
|
||||
accessible from the |CL| host on port 8080
|
||||
|
||||
#. You can access the Welcome to Nginx! splash page running in the container
|
||||
by browsing to \http://127.0.0.1:8080 or by running this :command:`curl`
|
||||
command from your |CL| machine:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl 127.0.0.1:8080
|
||||
|
||||
#. Finally, stop and delete the nginx container by running the
|
||||
:command:`docker stop` and :command:`docker rm` commands.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker stop test-nginx
|
||||
sudo docker rm test-nginx
|
||||
|
||||
Congratulations! At this point, you have successfully pulled a nginx
|
||||
container image from `Docker Hub`_ and have run an example container.
|
||||
|
||||
Creating a Docker swarm cluster
|
||||
*******************************
|
||||
|
||||
Clusters of Docker hosts are referred to as *swarms*.
|
||||
|
||||
The process in this tutorial can be repeated to install Docker on multiple
|
||||
|CL| hosts with the intent to form a Docker swarm cluster.
|
||||
|
||||
The `Docker documentation on swarm key concepts`_ and
|
||||
`Docker documentation on creating a swarm`_ can be referenced
|
||||
for further instructions on setting up a swarm.
|
||||
|
||||
Related topics
|
||||
**************
|
||||
|
||||
* `Docker Home`_
|
||||
* `Docker Documentation`_
|
||||
* `Docker Hub`_
|
||||
* `Kata Containers`_
|
||||
|
||||
.. _unique features: https://clearlinux.org/features
|
||||
|
||||
.. _Docker proxy instructions: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
|
||||
|
||||
.. _bundles: https://clearlinux.org/documentation/clear-linux/concepts/bundles-about#related-concepts
|
||||
|
||||
.. _stateless system: https://clearlinux.org/features/stateless
|
||||
|
||||
.. _Docker documentation on daemon configuration: https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file
|
||||
|
||||
.. _Kata Containers: https://katacontainers.io/
|
||||
|
||||
.. _Docker Home: https://www.docker.com/
|
||||
|
||||
.. _Docker Documentation: https://docs.docker.com/
|
||||
|
||||
.. _Docker Hub: https://hub.docker.com/
|
||||
|
||||
.. _Docker documentation on swarm key concepts: https://docs.docker.com/engine/swarm/key-concepts/
|
||||
|
||||
.. _Docker documentation on creating a swarm: https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/
|
||||
|
||||
.. _Configure direct lvm mode for production: https://docs.docker.com/storage/storagedriver/device-mapper-driver/
|
||||
|
||||
.. _OverlayFS storage driver: https://docs.docker.com/storage/storagedriver/overlayfs-driver/
|
||||
|
||||
.. _Docker Device Mapper storage driver: https://docs.docker.com/storage/storagedriver/device-mapper-driver/
|
||||
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 823 KiB |
@@ -0,0 +1,103 @@
|
||||
.. _flatpak:
|
||||
|
||||
Use Flatpak\* to install applications on |CL-ATTR|
|
||||
##################################################
|
||||
|
||||
This tutorial provides all the required steps to install Flatpak as well as
|
||||
downloading, installing, and running LibreOffice\* on |CL-ATTR|.
|
||||
|
||||
Please visit the `Flatpak website`_ for more information about Flatpak and
|
||||
how to use it. You can also `download it here`_.
|
||||
|
||||
Before you begin
|
||||
****************
|
||||
|
||||
This tutorial assumes you have installed |CL| on your host system.
|
||||
For detailed instructions on installing |CL| on a bare metal system, visit
|
||||
our :ref:`bare metal installation guide <bare-metal-install-desktop>`.
|
||||
|
||||
Install Flatpak on your host system
|
||||
===================================
|
||||
|
||||
Flatpak is included as part of the bundle `desktop`. To install the
|
||||
application, log in to your user account and enter the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add desktop
|
||||
|
||||
Install and run the LibreOffice Flatpak image
|
||||
=============================================
|
||||
|
||||
Application developers have the option to bundle their applications using
|
||||
Flatpak to allow the installation of a single distribution of their
|
||||
application on different distributions of Linux, including |CL|.
|
||||
Flatpak provides a `list of applications`_ available through Flathub.
|
||||
|
||||
|CL| enables the Flathub repository by default.
|
||||
|
||||
|
||||
Installing using gnome software
|
||||
-------------------------------
|
||||
|
||||
All you need to do is to launch `gnome software`, search for the LibreOffice
|
||||
app, and click the install button.
|
||||
|
||||
.. figure:: figures/01-install-libreoffice.gif
|
||||
:alt: install libreoffice step by step
|
||||
|
||||
Figure 1: Installing LibreOffice using gnome-software
|
||||
|
||||
|
||||
|
||||
Installing using the command line
|
||||
---------------------------------
|
||||
|
||||
Open the `gnome-terminal` and type the following command to install the
|
||||
LibreOffice app.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
flatpak install --user flathub org.libreoffice.LibreOffice
|
||||
Installing in user:
|
||||
org.libreoffice.LibreOffice/x86_64/stable flathub 2aff77bd5cf1
|
||||
permissions: ipc, network, pulseaudio, wayland, x11, dri
|
||||
file access: host, xdg-run/dconf
|
||||
dbus access: ca.desrt.dconf, org.gtk.vfs.*
|
||||
dbus ownership: org.libreoffice.LibreOfficeIpc0
|
||||
org.libreoffice.LibreOffice.Locale/x86_64/stable flathub 924157b3b009
|
||||
Is this ok [y/n]: y
|
||||
Installing for user: org.libreoffice.LibreOffice/x86_64/stable from flathub
|
||||
[####################] 403 metadata, 4661 content objects fetched; 222574 KiB transferred in 99 seconds
|
||||
Now at 2aff77bd5cf1.
|
||||
Installing for user: org.libreoffice.LibreOffice.Locale/x86_64/stable from flathub
|
||||
[####################] 10 metadata, 71 content objects fetched; 1013 KiB transferred in 3 seconds
|
||||
Now at 924157b3b009.
|
||||
|
||||
|
||||
Launch LibreOffice
|
||||
==================
|
||||
A new set of icons will appear in your Gnome applications list titled :guilabel:`LibreOffice.` To
|
||||
execute the application, highlight the application and click on the :guilabel:`LibreOffice` icon.
|
||||
LibreOffice will start normally.
|
||||
|
||||
.. figure:: figures/02-openlibreoffice.gif
|
||||
:alt: Opening LibreOffice app
|
||||
|
||||
Figure 2: Select :guilabel:`LibreOffice` app
|
||||
|
||||
Using the command line
|
||||
----------------------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
flatpak run org.libreoffice.LibreOffice
|
||||
|
||||
|
||||
.. _Flatpak website: http://flatpak.org
|
||||
|
||||
.. _list of applications: http://flatpak.org/apps.html
|
||||
|
||||
.. _download it here:
|
||||
http://download.documentfoundation.org/libreoffice/flatpak/latest/LibreOffice.flatpak
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
.. _fmv:
|
||||
|
||||
Use the function multi-version patch generator
|
||||
##############################################
|
||||
|
||||
CPU architectures often gain interesting new instructions as they evolve but
|
||||
application developers find it difficult to take advantage of those
|
||||
instructions. The reluctance to lose backward-compatibility is one of the
|
||||
main roadblocks slowing developers from using advancements in newer computing
|
||||
architectures. :abbr:`FMV (Function Multi-Versioning)`, which first appeared
|
||||
in `GCC`_ 4.8, is a way to have multiple implementations of a function, each
|
||||
using a different architecture specialized instruction-set extensions. GCC
|
||||
6 introduces changes to FMV to make it even easier to bring architecture-
|
||||
based optimizations to the application code.
|
||||
|
||||
In this tutorial we will use FMV on general code and on
|
||||
:abbr:`FFT (Fast Fourier Transform)` library code. Upon completing the
|
||||
tutorial, you will be able to use this technology on your code and use the
|
||||
libraries to deploy architecture-based optimizations to your application code.
|
||||
|
||||
Install and configure a |CL| host on bare metal
|
||||
***********************************************
|
||||
First, follow our guide to :ref:`bare-metal-install-desktop`. Once the bare
|
||||
metal installation and initial configuration are complete, add the
|
||||
`desktop-dev` bundle to the system. `desktop-dev`: contains the necessary
|
||||
development tools like GCC and Perl\*.
|
||||
|
||||
To install the bundles, run the following command in the :file:`$HOME`
|
||||
directory:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add desktop-dev
|
||||
|
||||
Detect loop vectorization candidates
|
||||
************************************
|
||||
|
||||
Now, we need to detect the loop vectorization candidates to be cloned for
|
||||
multiple platforms with FMV. As an example, we will use the following
|
||||
simple C code:
|
||||
|
||||
.. code-block:: c
|
||||
:linenos:
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <sys/time.h>
|
||||
#define MAX 1000000
|
||||
|
||||
int a[256], b[256], c[256];
|
||||
|
||||
void foo(){
|
||||
int i,x;
|
||||
for (x=0; x<MAX; x++){
|
||||
for (i=0; i<256; i++){
|
||||
a[i] = b[i] + c[i];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
int main(){
|
||||
foo();
|
||||
return 0;
|
||||
}
|
||||
|
||||
Save the example code as :file:`example.c` in the current directory and build
|
||||
with the following flags:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
gcc -O3 -fopt-info-vec example.c -o example
|
||||
|
||||
The build generates the following output:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
example.c:11:9: note: loop vectorized
|
||||
example.c:11:9: note: loop vectorized
|
||||
|
||||
The output shows that line 11 is a good candidate for vectorization:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
for (i=0; i<256; i++){
|
||||
a[i] = b[i] + c[i];
|
||||
|
||||
Generate the FMV patch
|
||||
**********************
|
||||
|
||||
To generate the FMV patch with the `make-fmv-patch`_ project, we
|
||||
must clone the project and generate a log file with the loop vectorized
|
||||
information:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/clearlinux/make-fmv-patch.git
|
||||
gcc -O3 -fopt-info-vec example.c -o example &> log
|
||||
|
||||
To generate the patch files, execute:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
perl ./make-fmv-patch/make-fmv-patch.pl log .
|
||||
|
||||
The :file:`make-fmv-patch.pl` script takes two arguments: `<buildlog>` and
|
||||
`<sourcecode>`. Replace `<buildlog>` and `<sourcecode>` with the proper
|
||||
values and execute:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
perl make-fmv-patch.pl <buildlog> <sourcecode>
|
||||
|
||||
The command generates the following :file:`example.c.patch` patch:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
--- ./example.c 2017-09-27 16:05:42.279505430 +0000
|
||||
+++ ./example.c~ 2017-09-27 16:19:11.691544026 +0000
|
||||
@@ -5,6 +5,7 @@
|
||||
|
||||
int a[256], b[256], c[256];
|
||||
|
||||
+__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
void foo(){
|
||||
int i,x;
|
||||
for (x=0; x<MAX; x++){
|
||||
|
||||
We recommend you use the :file:`make-fmv-patch` script to add the attribute
|
||||
generating the target clones on the function `foo`. Thus, we can have the
|
||||
following code:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <sys/time.h>
|
||||
#define MAX 1000000
|
||||
|
||||
int a[256], b[256], c[256];
|
||||
|
||||
__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
void foo(){
|
||||
int i,x;
|
||||
for (x=0; x<MAX; x++){
|
||||
for (i=0; i<256; i++){
|
||||
a[i] = b[i] + c[i];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
int main(){
|
||||
foo();
|
||||
return 0;
|
||||
}
|
||||
|
||||
Changing the value of the `$avx2` variable, we can change the target
|
||||
clones when adding the patches or in the :file:`make-fmv-patch.pl` script:
|
||||
|
||||
.. code-block:: perl
|
||||
|
||||
my $avx2 = '__attribute__((target_clones("avx2","arch=atom","default")))'."\n";
|
||||
|
||||
Compile the code again with FMV and add the option to analyze the `objdump`
|
||||
log:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
gcc -O3 example.c -o example -g
|
||||
objdump -S example | less
|
||||
|
||||
You can see the multiple clones of the `foo` function:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
foo
|
||||
foo.avx2.0
|
||||
foo.arch_atom.1
|
||||
|
||||
The cloned functions use AVX2 registers and vectorized instructions. To
|
||||
verify, enter the following commands:
|
||||
|
||||
::
|
||||
|
||||
vpaddd (%r8,%rax,1),%ymm0,%ymm0
|
||||
vmovdqu %ymm0,(%rcx,%rax,1)
|
||||
|
||||
FTT project example
|
||||
*******************
|
||||
|
||||
To follow the same approach with a package like FFT, we must use the
|
||||
`-fopt-info-vec` flag to get a build log file similar to:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
~/make-fmv-patch/make-fmv-patch.pl results/build.log fftw-3.3.6-pl2/
|
||||
|
||||
patching fftw-3.3.6-pl2/libbench2/verify-lib.c @ lines (36 114 151 162 173 195 215 284)
|
||||
patching fftw-3.3.6-pl2/tools/fftw-wisdom.c @ lines (150)
|
||||
patching fftw-3.3.6-pl2/libbench2/speed.c @ lines (26)
|
||||
patching fftw-3.3.6-pl2/tests/bench.c @ lines (27)
|
||||
patching fftw-3.3.6-pl2/libbench2/util.c @ lines (181)
|
||||
patching fftw-3.3.6-pl2/libbench2/problem.c @ lines (229)
|
||||
patching fftw-3.3.6-pl2/tests/fftw-bench.c @ lines (101 147 162 249)
|
||||
patching fftw-3.3.6-pl2/libbench2/mp.c @ lines (79 190 215)
|
||||
patching fftw-3.3.6-pl2/libbench2/caset.c @ lines (5)
|
||||
patching fftw-3.3.6-pl2/libbench2/verify-r2r.c @ lines (44 187 197 207 316 333 723)
|
||||
|
||||
For example, the :file:`fftw-3.3.6-pl2/tools/fftw-wisdom.c.patch` file
|
||||
generates the following patches:
|
||||
|
||||
.. code-block:: diff
|
||||
:linenos:
|
||||
|
||||
--- fftw-3.3.6-pl2/libbench2/verify-lib.c 2017-01-27 21:08:13.000000000 +0000
|
||||
+++ fftw-3.3.6-pl2/libbench2/verify-lib.c~ 2017-09-27 17:49:21.913802006 +0000
|
||||
@@ -33,6 +33,7 @@
|
||||
|
||||
double dmax(double x, double y) { return (x > y) ? x : y; }
|
||||
|
||||
+__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
static double aerror(C *a, C *b, int n)
|
||||
{
|
||||
if (n > 0) {
|
||||
@@ -111,6 +112,7 @@
|
||||
}
|
||||
|
||||
/* make array hermitian */
|
||||
+__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
void mkhermitian(C *A, int rank, const bench_iodim *dim, int stride)
|
||||
{
|
||||
if (rank == 0)
|
||||
@@ -148,6 +150,7 @@
|
||||
}
|
||||
|
||||
/* C = A + B */
|
||||
+__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
void aadd(C *c, C *a, C *b, int n)
|
||||
{
|
||||
int i;
|
||||
@@ -159,6 +162,7 @@
|
||||
}
|
||||
|
||||
/* C = A - B */
|
||||
+__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
void asub(C *c, C *a, C *b, int n)
|
||||
{
|
||||
int i;
|
||||
@@ -170,6 +174,7 @@
|
||||
}
|
||||
|
||||
/* B = rotate left A (complex) */
|
||||
+__attribute__((target_clones("avx2","arch=atom","default")))
|
||||
void arol(C *b, C *a, int n, int nb, int na)
|
||||
{
|
||||
int i, ib, ia;
|
||||
@@ -192,6 +197,7 @@
|
||||
}
|
||||
}
|
||||
|
||||
With these patches, we can select where to apply the FMV technology making
|
||||
bringing architecture-based optimizations to application code even easier.
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You have successfully installed an FMV development environment on |CL|.
|
||||
Furthermore, you used cutting edge compiler technology to improve the
|
||||
performance of your application based on Intel Architecture technology and
|
||||
profiling of the specific execution of your application.
|
||||
|
||||
.. _GCC: https://gcc.gnu.org
|
||||
.. _make-fmv-patch: https://github.com/clearlinux/make-fmv-patch
|
||||
@@ -0,0 +1,420 @@
|
||||
.. _greengrass:
|
||||
|
||||
Enable AWS Greengrass\* and OpenVINO™ toolkit
|
||||
#############################################
|
||||
|
||||
Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers to
|
||||
deploy inference functionalities [1] on Intel® IoT edge devices with
|
||||
accelerators (CPU, Integrated GPU, Intel® FPGA, and Intel® Movidius™
|
||||
technology). These functions provide a great developer experience and seamless
|
||||
migration of visual analytics from cloud to edge in a secure manner using a
|
||||
containerized environment. Hardware-accelerated FaaS provides the best-in-class
|
||||
performance by accessing optimized deep learning libraries on Intel® IoT
|
||||
edge devices with accelerators.
|
||||
|
||||
This tutorial demonstrates how to:
|
||||
|
||||
* Set up the Intel® edge device with |CL-ATTR|
|
||||
* Install the OpenVINO™ toolkit and Amazon Web Services\* (AWS\*)
|
||||
Greengrass\* software stacks
|
||||
* Use AWS Greengrass\* and AWS Lambda\* to deploy the FaaS samples from the cloud
|
||||
|
||||
Refer to the following topics:
|
||||
|
||||
.. contents:: :local:
|
||||
:depth: 1
|
||||
|
||||
|
||||
Supported platforms
|
||||
*******************
|
||||
|
||||
* Operating System: |CL| latest release
|
||||
* Hardware: Intel® core platforms (This tutorial supports inference on CPU only.)
|
||||
|
||||
Sample description
|
||||
==================
|
||||
|
||||
The AWS Greengrass samples are located at `Edge-Analytics-FaaS`_. This
|
||||
tutorial uses the 1.0 version of the source code.
|
||||
|
||||
|CL| provides the following AWS Greengrass samples:
|
||||
|
||||
* `greengrass_classification_sample.py`_
|
||||
|
||||
This AWS Greengrass sample classifies a video stream using classification
|
||||
networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS\*
|
||||
IoT Cloud every second.
|
||||
|
||||
* `greengrass_object_detection_sample_ssd.py`_
|
||||
|
||||
This AWS Greengrass sample detects objects in a video stream and
|
||||
classifies them using single-shot multi-box detection (SSD) networks such
|
||||
as SSD Squeezenet, SSD Mobilenet, and SSD300. This sample publishes
|
||||
detection outputs such as class label, class confidence, and bounding box
|
||||
coordinates on AWS IoT Cloud every second.
|
||||
|
||||
|
||||
Install the OS on the edge device
|
||||
*********************************
|
||||
|
||||
Start with a clean installation of |CL| on a new system, using the
|
||||
:ref:`bare-metal-install-desktop`, found in :ref:`get-started`.
|
||||
|
||||
Create user accounts
|
||||
====================
|
||||
|
||||
After |CL| is installed, create two user accounts. Create an administrative
|
||||
user in |CL| and create a user account for the Greengrass services to use (
|
||||
see Greengrass user below).
|
||||
|
||||
#. Create a new user and set a password for that user. Enter the following
|
||||
commands as ``root``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
useradd <userid>
|
||||
passwd <userid>
|
||||
|
||||
#. Next, enable the :command:`sudo` command for your new ``<userid>``. Add
|
||||
``<userid>`` to the *wheel* group:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
usermod -G wheel -a <userid>
|
||||
|
||||
#. Create a :file:`/etc/fstab` file.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
touch /etc/fstab
|
||||
|
||||
.. note::
|
||||
|
||||
By default, |CL| does not create an :file:`/etc/fstab` file.
|
||||
You must create this file before the Greengrass service runs.
|
||||
|
||||
Add required bundles
|
||||
====================
|
||||
|
||||
Use the :command:`swupd` software updater utility to add the prerequisite bundles
|
||||
for the OpenVINO software stack:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
swupd bundle-add os-clr-on-clr desktop-autostart computer-vision-basic
|
||||
|
||||
.. note::
|
||||
|
||||
Learn more about how to :ref:`swupd-guide`.
|
||||
|
||||
The :command:`computer-vision-basic` bundle installs the OpenVINO™ toolkit,
|
||||
and the sample models optimized for Intel® edge platforms.
|
||||
|
||||
Convert deep learning models
|
||||
============================
|
||||
|
||||
Locate sample models
|
||||
--------------------
|
||||
|
||||
There are two types of provided models that can be used in conjunction with
|
||||
AWS Greengrass for this tutorial: classification or object detection.
|
||||
|
||||
To complete this tutorial using an image classification model,
|
||||
download the BVLC AlexNet model files `bvlc_alexnet.caffemodel`_ and `deploy.prototxt`_
|
||||
to the default model_location at :file:`/usr/share/openvino/models`.
|
||||
Any custom pre-trained classification models can be used with the
|
||||
classification sample.
|
||||
|
||||
For object detection, the sample models optimized for Intel® edge platforms
|
||||
are included with the computer-vision-basic bundle installation at :file:`/usr/share/openvino/models`.
|
||||
These models are provided as an example; however, you may also use a custom SSD model
|
||||
with the Greengrass object detection sample.
|
||||
|
||||
Run model optimizer
|
||||
-------------------
|
||||
|
||||
Follow these instructions for `converting deep learning models to Intermediate Representation using Model Optimizer`_. To optimize either of the sample models described above, run one of the following commands.
|
||||
|
||||
For classification using BVLC AlexNet model:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python3 mo.py --framework caffe --input_model
|
||||
<model_location>/bvlc_alexnet.caffemodel --input_proto
|
||||
<model_location>/deploy.prototxt --data_type <data_type> --output_dir
|
||||
<output_dir> --input_shape [1,3,227,227]
|
||||
|
||||
For object detection using SqueezeNetSSD-5Class model:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python3 mo.py --framework caffe --input_model
|
||||
<model_location>/'SqueezeNet 5-Class detection'/SqueezeNetSSD-5Class.caffemodel
|
||||
--input_proto <model_location>/'SqueezeNet 5-Class detection'/SqueezeNetSSD-5Class.prototxt
|
||||
--data_type <data_type> --output_dir <output_dir>
|
||||
|
||||
In these examples:
|
||||
|
||||
* ``<model_location>`` is :file:`/usr/share/openvino/models`.
|
||||
|
||||
* ``<data_type>`` is FP32 or FP16, depending on target device.
|
||||
|
||||
* ``<output_dir>`` is the directory where the Intermediate Representation
|
||||
(IR) is stored. IR contains .xml format corresponding to the network
|
||||
structure and .bin format corresponding to weights. This .xml file should be
|
||||
passed to :command:`<PARAM_MODEL_XML>`.
|
||||
|
||||
* In the BVLC AlexNet model, the prototxt defines the input shape with
|
||||
batch size 10 by default. In order to use any other batch size, the
|
||||
entire input shape must be provided as an argument to the model
|
||||
optimizer. For example, to use batch size 1, you must provide:
|
||||
``--input_shape [1,3,227,227]``
|
||||
|
||||
|
||||
Configure AWS Greengrass group
|
||||
******************************
|
||||
|
||||
For each Intel® edge platform, you must create a new AWS Greengrass group
|
||||
and install AWS Greengrass core software to establish the connection between
|
||||
cloud and edge.
|
||||
|
||||
#. To create an AWS Greengrass group, follow the instructions in
|
||||
`Configure AWS IoT Greengrass on AWS IoT`_.
|
||||
|
||||
#. To install and configure AWS Greengrass core on edge platform, follow
|
||||
the instructions in `Start AWS Greengrass on the Core Device`_. In
|
||||
step 8(b), download the x86_64 Ubuntu\* configuration of the AWS Greengrass
|
||||
core software.
|
||||
|
||||
.. note::
|
||||
|
||||
You do not need to run the :file:`cgroupfs-mount.sh` script in step #6
|
||||
of Module 1 of the `AWS Greengrass developer guide`_ because this is
|
||||
enabled already in |CL|.
|
||||
|
||||
#. Be sure to download both the security resources and the AWS Greengrass
|
||||
core software.
|
||||
|
||||
.. note::
|
||||
|
||||
Security certificates are linked to your AWS account.
|
||||
|
||||
|
||||
Create and package Lambda function
|
||||
**********************************
|
||||
|
||||
#. Complete steps 1-4 of the AWS Greengrass tutorial at `Create and Package a Lambda Function`_.
|
||||
|
||||
.. note::
|
||||
|
||||
This creates the tarball needed to create the AWS Greengrass
|
||||
environment on the edge device.
|
||||
|
||||
|
||||
#. In step 5, replace :file:`greengrassHelloWorld.py` with the classification or object detection
|
||||
Greengrass sample from `Edge-Analytics-Faas`_:
|
||||
|
||||
* Classification: `greengrass_classification_sample.py`_
|
||||
|
||||
* Object Detection: `greengrass_object_detection_sample_ssd.py`_
|
||||
|
||||
#. Zip the selected Greengrass sample with the extracted Greengrass SDK folders from the previous
|
||||
step into :file:`greengrass_sample_python_lambda.zip`.
|
||||
|
||||
The zip should contain:
|
||||
|
||||
* greengrasssdk
|
||||
|
||||
* greengrass classification or object detection sample
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
zip -r greengrass_lambda.zip greengrasssdk
|
||||
greengrass_object_detection_sample_ssd.py
|
||||
|
||||
#. Return to the AWS documentation section called `Create and Package a Lambda Function`_
|
||||
and complete the procedure.
|
||||
|
||||
.. note::
|
||||
|
||||
In step 9(a) of the AWS documentation, while uploading the zip file,
|
||||
make sure to name the handler to one of the following, depending on the
|
||||
AWS Greengrass sample you are using:
|
||||
|
||||
* greengrass_object_detection_sample_ssd.function_handler
|
||||
* greengrass_classification_sample.function_handler
|
||||
|
||||
|
||||
Configure Lambda function
|
||||
*************************
|
||||
|
||||
After creating the Greengrass group and the Lambda function, start
|
||||
configuring the Lambda function for AWS Greengrass.
|
||||
|
||||
#. Follow steps 1-8 in `Configure the Lambda Function for AWS IoT Greengrass`_ in the AWS
|
||||
documentation.
|
||||
|
||||
#. In addition to the details mentioned in step 8, change the Memory limit
|
||||
to 2048 MB to accommodate large input video streams.
|
||||
|
||||
#. Add the following environment variables as key-value pairs when editing
|
||||
the Lambda configuration and click on update:
|
||||
|
||||
.. list-table:: **Table 1. Environment variables: Lambda configuration**
|
||||
:widths: 20 80
|
||||
:header-rows: 1
|
||||
|
||||
* - Key
|
||||
- Value
|
||||
* - PARAM_MODEL_XML
|
||||
- <MODEL_DIR>/<IR.xml>, where <MODEL_DIR> is user specified and
|
||||
contains IR.xml, the Intermediate Representation file from Intel® Model Optimizer.
|
||||
For this tutorial, <MODEL_DIR> should be set to '/usr/share/openvino/models'
|
||||
or one of its subdirectories.
|
||||
* - PARAM_INPUT_SOURCE
|
||||
- <DATA_DIR>/input.webm to be specified by user. Holds both input and
|
||||
output data. For webcam, set PARAM_INPUT_SOURCE to ‘/dev/video0’
|
||||
* - PARAM_DEVICE
|
||||
- "CPU"
|
||||
* - PARAM_CPU_EXTENSION_PATH
|
||||
- /usr/lib64/libcpu_extension.so
|
||||
* - PARAM_OUTPUT_DIRECTORY
|
||||
- <DATA_DIR> to be specified by user. Holds both input and output
|
||||
data
|
||||
* - PARAM_NUM_TOP_RESULTS
|
||||
- User specified for classification sample.
|
||||
(e.g. 1 for top-1 result, 5 for top-5 results)
|
||||
|
||||
#. Add subscription to subscribe, or publish messages from AWS Greengrass
|
||||
Lambda function by completing the procedure in `Configure the Lambda Function for AWS IoT Greengrass`_.
|
||||
|
||||
.. note::
|
||||
|
||||
The optional topic filter field is the topic mentioned inside the Lambda
|
||||
function. In this tutorial, sample topics include the following:
|
||||
:command:`openvino/ssd` or :command:`openvino/classification`
|
||||
|
||||
Add local resources
|
||||
===================
|
||||
|
||||
Refer to the AWS documentation for details about `local resources and access privileges`_.
|
||||
|
||||
The following table describes the local resources needed for the CPU:
|
||||
|
||||
.. list-table:: **Local resources**
|
||||
:widths: 20, 20, 20, 20
|
||||
:header-rows: 1
|
||||
|
||||
* - Name
|
||||
- Resource type
|
||||
- Local path
|
||||
- Access
|
||||
|
||||
* - ModelDir
|
||||
- Volume
|
||||
- <MODEL_DIR> to be specified by user
|
||||
- Read-Only
|
||||
|
||||
* - Webcam
|
||||
- Device
|
||||
- /dev/video0
|
||||
- Read-Only
|
||||
|
||||
* - DataDir
|
||||
- Volume
|
||||
- <DATA_DIR> to be specified by user. Holds both input and output
|
||||
data.
|
||||
- Read and Write
|
||||
|
||||
Deploy Lambda function
|
||||
**********************
|
||||
|
||||
Refer to the AWS documentation for instructions on how to
|
||||
`deploy the lambda function to AWS Greengrass core device`_. Select
|
||||
*Deployments* on the group page and follow the instructions.
|
||||
|
||||
Output consumption
|
||||
==================
|
||||
|
||||
There are four options available for output consumption. These options are
|
||||
used to report, stream, upload, or store inference output at an interval
|
||||
defined by the variable :command:`reporting_interval` in the AWS Greengrass samples.
|
||||
|
||||
a. IoT cloud output:
|
||||
|
||||
This option is enabled by default in the AWS Greengrass samples using the
|
||||
:command:`enable_iot_cloud_output` variable. You can use it to verify the lambda
|
||||
running on the edge device. It enables publishing messages to IoT cloud
|
||||
using the subscription topic specified in the lambda. (For example, topics
|
||||
may include :command:`openvino/classification` for classification and :command:`openvino/ssd`
|
||||
for object detection samples.) For classification, top-1 result with class
|
||||
label are published to IoT cloud. For SSD object detection, detection
|
||||
results such as bounding box coordinates of objects, class label, and
|
||||
class confidence are published.
|
||||
|
||||
Follow the instructions here to `view the output on IoT cloud`_.
|
||||
|
||||
b. Kinesis streaming:
|
||||
|
||||
This option enables inference output to be streamed from the edge device
|
||||
to cloud using Kinesis [3] streams when :command:`enable_kinesis_output` is set
|
||||
to True. The edge devices act as data producers and continually push
|
||||
processed data to the cloud. You must set up and specify
|
||||
Kinesis stream name, Kinesis shard, and AWS region in the AWS Greengrass
|
||||
samples.
|
||||
|
||||
c. Cloud storage using AWS S3 bucket:
|
||||
|
||||
When the :command:`enable_s3_jpeg_output` variable is set to True, it enables
|
||||
uploading and storing processed frames (in jpeg format) in an AWS S3
|
||||
bucket. You must set up and specify the S3 bucket name in the AWS
|
||||
Greengrass samples to store the JPEG images. The images are named using the
|
||||
timestamp and uploaded to S3.
|
||||
|
||||
d. Local storage:
|
||||
|
||||
When the :command:`enable_s3_jpeg_output` variable is set to True, it enables
|
||||
storing processed frames (in jpeg format) on the edge device. The images
|
||||
are named using the timestamp and stored in a directory specified by
|
||||
:command:`PARAM_OUTPUT_DIRECTORY`.
|
||||
|
||||
References
|
||||
**********
|
||||
|
||||
1. AWS Greengrass: https://aws.amazon.com/greengrass/
|
||||
2. AWS Lambda: https://aws.amazon.com/lambda/
|
||||
3. AWS Kinesis: https://aws.amazon.com/kinesis/
|
||||
|
||||
.. _Edge-Analytics-FaaS: https://github.com/intel/Edge-Analytics-FaaS/tree/v1.0/AWS%20Greengrass
|
||||
|
||||
.. _bvlc_alexnet.caffemodel: http://dl.caffe.berkeleyvision.org/bvlc_alexnet.caffemodel
|
||||
|
||||
.. _deploy.prototxt: https://github.com/BVLC/caffe/blob/master/models/bvlc_alexnet/deploy.prototxt
|
||||
|
||||
.. _greengrass_classification_sample.py: https://github.com/intel/Edge-Analytics-FaaS/blob/v1.0/AWS%20Greengrass/greengrass_classification_sample.py
|
||||
|
||||
.. _greengrass_object_detection_sample_ssd.py: https://github.com/intel/Edge-Analytics-FaaS/blob/v1.0/AWS%20Greengrass/greengrass_object_detection_sample_ssd.py
|
||||
|
||||
.. _converting deep learning models to Intermediate Representation using Model Optimizer: https://software.intel.com/en-us/articles/OpenVINO-ModelOptimizer
|
||||
|
||||
.. _AWS Greengrass Developer Guide: https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html
|
||||
|
||||
.. _Configure AWS IoT Greengrass on AWS IoT: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html
|
||||
|
||||
.. _Start AWS Greengrass on the Core Device: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-device-start.html
|
||||
|
||||
.. _Configure the Lambda Function for AWS IoT Greengrass: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html
|
||||
|
||||
.. _local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
|
||||
|
||||
.. _deploy the lambda function to AWS Greengrass core device: https://docs.aws.amazon.com/greengrass/latest/developerguide/configs-core.html
|
||||
|
||||
.. _Edge-optmized models repository: https://github.com/intel/Edge-optimized-models
|
||||
|
||||
.. _view the output on IoT cloud: https://docs.aws.amazon.com/greengrass/latest/developerguide/lambda-check.html
|
||||
|
||||
.. _this link to add local resources and access privileges: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
|
||||
|
||||
.. _Create and Package a Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html
|
||||
@@ -0,0 +1,256 @@
|
||||
.. _hadoop:
|
||||
|
||||
Set up a single node cluster with Hadoop\*
|
||||
##########################################
|
||||
|
||||
This tutorial walks you through the process of installing, configuring, and
|
||||
running Apache\* Hadoop on |CL-ATTR|. The Apache Hadoop software library is a
|
||||
framework for distributed processing of large data sets across clusters of
|
||||
computers using simple programming models. It is designed to scale up from
|
||||
single servers to thousands of machines, with each machine offering local
|
||||
computation and storage.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
Before following this tutorial, you should follow the
|
||||
:ref:`bare-metal-install-desktop` to ensure you have installed |CL|.
|
||||
|
||||
Before you install any new packages, update |CL| with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
For the purposes of this tutorial, we will install Hadoop in a single machine
|
||||
running both the master and slave daemons.
|
||||
|
||||
Install Apache Hadoop
|
||||
*********************
|
||||
|
||||
Apache Hadoop is included in the `big-data-basic` bundle. To install the
|
||||
framework, enter the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add big-data-basic
|
||||
|
||||
Configure Apache Hadoop
|
||||
***********************
|
||||
|
||||
#. To create the configuration directory, enter the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo mkdir /etc/hadoop
|
||||
|
||||
#. Copy the defaults from :file:`/usr/share/defaults/hadoop` to
|
||||
:file:`/etc/hadoop` with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ sudo cp /usr/share/defaults/hadoop/* /etc/hadoop
|
||||
|
||||
.. note:: Since |CL| is a stateless system, you should never modify the
|
||||
files under the :file:`/usr/share/defaults` directory. The software
|
||||
updater will overwrite those files.
|
||||
|
||||
Once all the configuration files are in :file:`/etc/hadoop`, we must edit
|
||||
them to fit our needs. The `NameNode` server is the master server. It manages
|
||||
the namespace of the files system and regulates the clients' access to files.
|
||||
The first file we edit, :file:`/etc/hadoop/core-site.xml`, informs the Hadoop
|
||||
daemon where `NameNode` is running.
|
||||
|
||||
In this tutorial, our `NameNode` runs in our `localhost`. Follow these steps
|
||||
to set it up correctly:
|
||||
|
||||
#. Open the :file:`/etc/hadoop/core-site.xml` file using the editor of your
|
||||
choice and modify the file as follows:
|
||||
|
||||
.. code-block:: xml
|
||||
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>fs.default.name</name>
|
||||
<value>hdfs://localhost:9000</value>
|
||||
</property>
|
||||
</configuration>
|
||||
|
||||
#. Edit the :file:`/etc/hadoop/hdfs-site.xml` file. This file configures the
|
||||
:abbr:`HDFS (Hadoop Distributed File System)` daemons. This configuration
|
||||
includes the list of permitted and excluded data nodes and the size of
|
||||
said blocks. In this example, we are setting the number of block
|
||||
replication to 1 from the default of 3 as follows:
|
||||
|
||||
.. code-block:: xml
|
||||
:emphasize-lines: 6
|
||||
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>dfs.replication</name>
|
||||
<value>1</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>dfs.permission</name>
|
||||
<value>false</value>
|
||||
</property>
|
||||
</configuration>
|
||||
|
||||
#. Edit the :file:`/etc/hadoop/mapred-site.xml` file. This file configures
|
||||
all daemons related to `MapReduce`: `JobTracker` and `TaskTrackers`. With
|
||||
`MapReduce`, Hadoop can process big amounts of data in multiple systems. In
|
||||
our example, we set :abbr:`YARN (Yet Another Resource Manager)` as our
|
||||
runtime framework for executing `MapReduce` jobs as follows:
|
||||
|
||||
.. code-block:: xml
|
||||
:emphasize-lines: 5,6
|
||||
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>mapreduce.framework.name</name>
|
||||
<value>yarn</value>
|
||||
</property>
|
||||
</configuration>
|
||||
|
||||
#. Edit the :file:`/etc/hadoop/yarn-site.xml` file. This file configures all
|
||||
daemons related to `YARN`: `ResourceManager` and `NodeManager`. In our
|
||||
example, we implement the `mapreduce_shuffle` service, which is the
|
||||
default as follows:
|
||||
|
||||
.. code-block:: xml
|
||||
:emphasize-lines: 4,5,8,9
|
||||
|
||||
<?xml version="1.0"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>yarn.nodemanager.aux-services</name>
|
||||
<value>mapreduce_shuffle</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
|
||||
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
|
||||
</property>
|
||||
</configuration>
|
||||
|
||||
Configure your SSH key
|
||||
**********************
|
||||
|
||||
#. Create a SSH key. If you already have one, skip this step.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo ssh-keygen -t rsa
|
||||
|
||||
|
||||
#. Copy the key to your authorized keys.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo cat /root/.ssh/id_rsa.pub | sudo tee -a /root/.ssh/authorized_keys
|
||||
|
||||
#. Log into the localhost. If no password prompt appears, you are ready to
|
||||
run the Hadoop daemons.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo ssh localhost
|
||||
|
||||
Run the Hadoop daemons
|
||||
**********************
|
||||
|
||||
With all the configuration files properly edited, we are ready to start the
|
||||
daemons.
|
||||
|
||||
When we format the `NameNode` server, it formats the meta-data related to
|
||||
data nodes. Thus, all the information on the data nodes is lost and the nodes
|
||||
can be reused for new data.
|
||||
|
||||
#. Format the `NameNode` server with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo hdfs namenode -format
|
||||
|
||||
#. Start the DFS in `NameNode` and `DataNodes` with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo start-dfs.sh
|
||||
|
||||
#. The console output should be similar to:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
Starting namenodes on [localhost]
|
||||
The authenticity of host 'localhost (::1)' can't be established.
|
||||
ECDSA key fingerprint is
|
||||
SHA256:97e+7TnomsS9W7GjFPjzY75HGBp+f1y6sA+ZFcOPIPU.
|
||||
Are you sure you want to continue connecting (yes/no)?
|
||||
|
||||
Enter `yes` to continue.
|
||||
|
||||
#. Start the `YARN` daemons `ResourceManager` and `NodeManager` with the
|
||||
following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo start-yarn.sh
|
||||
|
||||
#. Ensure everything is running as expected with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo jps
|
||||
|
||||
#. The console output should be similar to:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
22674 DataNode
|
||||
26228 Jps
|
||||
22533 NameNode
|
||||
23046 ResourceManager
|
||||
22854 SecondaryNameNode
|
||||
23150 NodeManager
|
||||
|
||||
Run the MapReduce wordcount example
|
||||
***********************************
|
||||
|
||||
#. Create the input directory.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo hdfs dfs -mkdir -p /user/root/input
|
||||
|
||||
#. Copy a file from the local file system to the HDFS.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo hdfs dfs -copyFromLocal local-file /user/root/input
|
||||
|
||||
#. Run the `wordcount` example.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo hadoop jar /usr/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount input output
|
||||
|
||||
#. Read output file "part-r-00000". This file contains the number of times
|
||||
each word appears in the file.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo hdfs dfs -cat /user/root/output/part-r-00000
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You successfully installed and setup a single node Hadoop cluster.
|
||||
Additionally, you ran a simple wordcount example.
|
||||
|
||||
Your single node Hadoop cluster is up and running!
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
.. _kata:
|
||||
|
||||
Install Kata Containers\*
|
||||
#########################
|
||||
|
||||
This tutorial describes how to install, configure, and run `Kata Containers`_
|
||||
on |CL-ATTR|. Kata Containers is an open source project developing a
|
||||
lightweight implementation of :abbr:`VMs (Virtual Machines)` that offer the
|
||||
speed of containers and the security of VMs.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
This tutorial assumes you have installed |CL| on your host system.
|
||||
For detailed instructions on installing |CL| on a bare metal system, follow
|
||||
the :ref:`bare metal installation tutorial<bare-metal-install-desktop>`.
|
||||
|
||||
If you have Clear Containers installed on your |CL| system, then follow the
|
||||
:ref:`migrate Clear Containers to Kata Containers tutorial<kata_migration>`.
|
||||
|
||||
Before you install any new packages, update |CL| with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
Install Kata Containers
|
||||
***********************
|
||||
|
||||
Kata Containers is included in the :file:`containers-virt` bundle.
|
||||
To install the framework, enter the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add containers-virt
|
||||
|
||||
Restart the Docker\* and Kata Containers systemd services.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart docker
|
||||
|
||||
Run Kata Containers
|
||||
*******************
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker run -ti busybox sh
|
||||
|
||||
.. note::
|
||||
|
||||
If you use a proxy server and your proxy environment variables are already
|
||||
set, run the following commands as a shell script to configure Docker:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker_service_dir="/etc/systemd/system/docker.service.d/"
|
||||
sudo mkdir -p "$docker_service_dir"
|
||||
cat <<EOF | sudo tee "$docker_service_dir/proxy.conf"
|
||||
[Service]
|
||||
Environment="HTTP_PROXY=$http_proxy"
|
||||
Environment="HTTPS_PROXY=$https_proxy"
|
||||
EOF
|
||||
echo "Reloading unit files and starting docker service"
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart docker
|
||||
sudo docker info
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You've successfully installed and set up Kata Containers on |CL|.
|
||||
|
||||
More information about Docker
|
||||
*****************************
|
||||
|
||||
Docker on |CL| provides a :file:`docker.service` file to start the Docker
|
||||
daemon. The daemon will use runc or kata-runtime depending on the
|
||||
environment:
|
||||
|
||||
* If you are running |CL| on bare metal or on a VM with Nested
|
||||
Virtualization activated, Docker uses kata-runtime as the
|
||||
default runtime.
|
||||
* If you are running |CL| on a VM without Nested Virtualization,
|
||||
Docker uses runc as the default runtime.
|
||||
|
||||
You do not need to manually configure the runtime for Docker, because
|
||||
it automatically uses the runtime supported by the system.
|
||||
|
||||
Troubleshooting
|
||||
===============
|
||||
|
||||
- To change the Docker storage driver, see
|
||||
:ref:`additional-docker-configuration`.
|
||||
|
||||
- For some |CL| versions before 27000, you may need to manually
|
||||
configure Docker\* to use Kata Containers by default.
|
||||
|
||||
To do so, enter:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo mkdir -p /etc/systemd/system/docker.service.d/
|
||||
cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/50-runtime.conf
|
||||
[Service]
|
||||
Environment="DOCKER_DEFAULT_RUNTIME=--default-runtime kata-runtime"
|
||||
EOF
|
||||
|
||||
- To check the version of |CL| on your system, enter:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd verify
|
||||
|
||||
|
||||
.. _Kata Containers: https://katacontainers.io/
|
||||
@@ -0,0 +1,91 @@
|
||||
.. _kata_migration:
|
||||
|
||||
Migrate Clear Containers to Kata Containers\*
|
||||
#############################################
|
||||
|
||||
`Clear Containers`_ and `Kata Containers`_ can co-exist in the same system.
|
||||
Both can be installed through the :file:`containers-virt bundle`. However, we
|
||||
recommend that you migrate to Kata Containers because Clear Containers is no
|
||||
longer maintained_ and will soon be deprecated on |CL-ATTR|.
|
||||
|
||||
This guide describes how to migrate to Kata Containers and assumes that:
|
||||
|
||||
* Clear Containers is on a Docker\* system.
|
||||
* Kata Containers is installed. See :ref:`kata`.
|
||||
|
||||
|
||||
Stop Clear Containers instances
|
||||
*******************************
|
||||
|
||||
As an unprivileged user, stop all running instances of Clear Containers:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
for container in $(sudo docker ps -q); do sudo docker stop $container; done
|
||||
|
||||
|
||||
Manually migrate customized configuration files
|
||||
***********************************************
|
||||
|
||||
If you have made changes to your `Clear Containers configuration`_, review
|
||||
those changes and decide whether to manually apply those changes to your
|
||||
`Kata Containers configuration`_.
|
||||
|
||||
Make any required changes before continuing this process.
|
||||
|
||||
.. note::
|
||||
|
||||
You do not need to manually remove any Clear Containers packages.
|
||||
|
||||
|
||||
Enable Kata Containers as default
|
||||
*********************************
|
||||
|
||||
#. Configure Docker to use the Kata Containers runtime by default.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo mkdir -p /etc/systemd/system/docker.service.d/
|
||||
cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/51-runtime.conf
|
||||
[Service]
|
||||
Environment="DOCKER_DEFAULT_RUNTIME=--default-runtime kata-runtime"
|
||||
EOF
|
||||
|
||||
#. Restart the Docker systemd services.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart docker
|
||||
|
||||
#. Verify Docker is using Kata Containers.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker info | grep -i 'default runtime'
|
||||
Default Runtime: kata-runtime
|
||||
|
||||
Run Kata Containers
|
||||
*******************
|
||||
|
||||
Use the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo docker run -ti busybox sh
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You've successfully migrated from Clear Containers to Kata Containers.
|
||||
|
||||
|
||||
.. _Clear Containers: https://github.com/clearcontainers
|
||||
|
||||
.. _Kata Containers: https://github.com/kata-containers
|
||||
|
||||
.. _maintained: https://github.com/kata-containers/documentation/blob/master/Upgrading.md#maintenance-warning
|
||||
|
||||
.. _Clear Containers configuration: https://github.com/clearcontainers/runtime#configuration
|
||||
|
||||
.. _Kata Containers configuration: https://github.com/kata-containers/runtime#configuration
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
.. _kubernetes-bp:
|
||||
|
||||
Kubernetes Best Practices on |CL|
|
||||
#################################
|
||||
|
||||
Use swupd to update clusters
|
||||
****************************
|
||||
|
||||
This tutorial shows you how to manage your Kubernetes cluster while using
|
||||
:command:`swupd` to update |CL-ATTR|.
|
||||
|
||||
In our tutorial :ref:`kubernetes`, we explain how to set up a Kubernetes
|
||||
cluster on |CL| using `kubeadm`. `Kubeadm documentation`_ often builds on the
|
||||
assumption that the distribution uses a traditional package manager (e.g.,
|
||||
RPM/DEB).
|
||||
|
||||
In contrast, |CL| uses `swupd` to update the OS, which in this case updates
|
||||
all of the kubernetes node and client binaries simultaneously, as part of
|
||||
the `cloud-native-basic` bundle (e.g., kubectl, kubeadm, kubelet). Running
|
||||
:command:`sudo swupd update` requires special care to ensure the OS
|
||||
incorporates the latest Kubernetes upgrades.
|
||||
|
||||
This document describes best practices to manage cluster upgrades with
|
||||
`kubeadm` on a |CL|-based cluster.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
Assure that you:
|
||||
|
||||
* Completed :ref:`kubernetes`
|
||||
* Installed the bundle `cloud-native-basic`
|
||||
|
||||
.. note::
|
||||
|
||||
Other Linux\* distros shown in Kubernetes upgrade documentation reflect
|
||||
`apt-get update`, `apt-mark hold kubeadm`, and similar commands; however, such commands **are not valid** on |CL|.
|
||||
|
||||
Update the control plane
|
||||
************************
|
||||
|
||||
#. Read kubernetes documentation `before you begin`_.
|
||||
|
||||
#. On your master node, run the command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
.. note::
|
||||
|
||||
If the minor version of Kubernetes changes, |CL| shows a message-of-the-day, or `motd`. When the `motd` appears, you **must postpone** a kubelet restart on master and nodes until the control plane is properly updated. :command:`swupd update` does not restart services automatically unless explicitly configured to do so.
|
||||
|
||||
#. Now follow these instructions in kubernetes documentation.
|
||||
|
||||
* `Upgrade control plane`_
|
||||
* `Drain control plane node`_
|
||||
* `Restart Kubelet and undrain node`_
|
||||
|
||||
Update worker nodes
|
||||
*******************
|
||||
|
||||
#. On each worker node, run the command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
#. Now follow these instructions in kubernetes documentation:
|
||||
|
||||
* `Drain node`_
|
||||
* `Update kubelet configuration`_
|
||||
* `Restart Kubelet and undrain node`_
|
||||
|
||||
.. _Kubeadm documentation: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/
|
||||
|
||||
.. _Restart Kubelet and undrain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#restart-the-kubelet-for-all-nodes
|
||||
|
||||
.. _Update kubelet configuration: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#upgrade-the-kubelet-config-on-worker-nodes
|
||||
|
||||
.. _Drain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes
|
||||
|
||||
.. _Restart kubelet and undrain node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#restart-the-kubelet-for-all-nodes
|
||||
|
||||
.. _Upgrade control plane: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#upgrade-the-control-plane-node
|
||||
|
||||
.. _Drain control plane node: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#drain-control-plane-and-worker-nodes
|
||||
|
||||
.. _Kubeadmn documentation: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
|
||||
|
||||
.. _before you begin: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/#before-you-begin
|
||||
|
||||
@@ -0,0 +1,414 @@
|
||||
.. _kubernetes:
|
||||
|
||||
Run Kubernetes\*
|
||||
################
|
||||
|
||||
This tutorial describes how to install, configure, and run the
|
||||
`Kubernetes container orchestration system`_ on |CL-ATTR| using CRI+O and
|
||||
kata-runtime.
|
||||
|
||||
Kubernetes\* is an open source system for automating deployment, scaling, and
|
||||
management of containerized applications. It groups containers that make up
|
||||
an application into logical units for easy management and discovery.
|
||||
|
||||
Kata Containers\* kata-runtime adheres to
|
||||
:abbr:`OCI (Open Container Initiative*)` guidelines and work seamlessly with
|
||||
Kubernetes. `Kata Containers`_ provide strong isolation for untrusted
|
||||
workloads or multi-tenant scenarios. Kata Containers can be
|
||||
allocated on a per-pod basis so you can mix and match both on the same host
|
||||
to suit your needs.
|
||||
|
||||
.. contents:: :local:
|
||||
:depth: 1
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
This tutorial assumes you have already installed |CL|. For detailed
|
||||
instructions on installing |CL| on a bare metal system, follow the
|
||||
:ref:`bare metal installation tutorial<bare-metal-install-desktop>`.
|
||||
Learn about the benefits of having an up-to-date system for cloud
|
||||
orchestration on the :ref:`swupd-about` page.
|
||||
|
||||
Before you install any new packages, update |CL| with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
Install Kubernetes and CRI runtimes
|
||||
***********************************
|
||||
|
||||
Kubernetes, a set of supported :abbr:`CRI (Container Runtime Interface)`
|
||||
runtimes, and networking plugins, are included in the `cloud-native-basic`_
|
||||
bundle.
|
||||
|
||||
To install this framework, enter the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add cloud-native-basic
|
||||
|
||||
.. note::
|
||||
|
||||
For more on networking plugins, see `Install pod network add-on`_.
|
||||
|
||||
Configure Kubernetes
|
||||
********************
|
||||
|
||||
This tutorial uses the basic default Kubernetes configuration for simplicity.
|
||||
You must define your Kubernetes configuration according to your specific
|
||||
deployment and your security needs.
|
||||
|
||||
#. Enable IP forwarding to avoid kubeadm `preflight check`_ errors:
|
||||
|
||||
Create (or edit if it exists) the file :file:`/etc/sysctl.d/60-k8s.conf`
|
||||
and include the following line:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
net.ipv4.ip_forward = 1
|
||||
|
||||
Apply the change:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl restart systemd-sysctl
|
||||
|
||||
#. Enable the kubelet service:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl enable kubelet.service
|
||||
|
||||
#. Disable swap using one of the following methods, either:
|
||||
|
||||
a) Temporarily:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swapoff -a
|
||||
|
||||
.. note::
|
||||
|
||||
Swap will be enabled at next reboot, causing failures in
|
||||
your cluster.
|
||||
|
||||
or:
|
||||
|
||||
b) Permanently:
|
||||
|
||||
Mask the swap partition:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl mask $(sed -n -e 's#^/dev/\([0-9a-z]*\).*#dev-\1.swap#p' /proc/swaps) 2>/dev/null
|
||||
sudo swapoff -a
|
||||
|
||||
.. note::
|
||||
|
||||
On systems with limited resources, some performance degradation may
|
||||
be observed while swap is disabled.
|
||||
|
||||
#. Switch to root to modify the `hosts` file:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo -s
|
||||
|
||||
#. Create (or edit if it exists) the hosts file that Kubernetes will read to
|
||||
locate the master's host:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
echo "127.0.0.1 localhost `hostname`" >> /etc/hosts
|
||||
|
||||
#. Exit root:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
exit
|
||||
|
||||
Configure and run Kubernetes
|
||||
****************************
|
||||
|
||||
This section describes how to configure and run Kubernetes with CRI-O and kata-runtime.
|
||||
|
||||
Configure and run CRI-O + kata-runtime
|
||||
======================================
|
||||
|
||||
#. Enable the CRI-O service:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl enable crio.service
|
||||
|
||||
#. Enter the commands:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart crio
|
||||
|
||||
#. Initialize the master control plane with the command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo kubeadm init --cri-socket=/run/crio/crio.sock
|
||||
|
||||
Install pod network add-on
|
||||
**************************
|
||||
|
||||
You must choose and install a `pod network add-on`_ to allow your pods to
|
||||
communicate. Check whether or not your add-on requires special flags when you
|
||||
initialize the master control plane.
|
||||
|
||||
The CRI-O default plugin_dir is :file:`/opt/cni/bin`. This must be a
|
||||
writable directory because third-party networking add-ons will install
|
||||
themselves there.
|
||||
|
||||
.. note::
|
||||
|
||||
CNI plugins provided by |CL| are installed as part of *cloud-native-basic*
|
||||
in :file:`/usr/libexec/cni/` and are currently *not* found by CRI-O by
|
||||
default. These separate directories are required because `swupd` controls
|
||||
the content of :file:`/usr` and leaves :file:`/opt` unchanged.
|
||||
|
||||
When using third-party network add-ons that rely on those plugins, such as
|
||||
Weave or Flannel do, make them available by creating symlinks:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo mkdir -p /opt/cni/bin
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
for i in /usr/libexec/cni/*; do sudo ln -sf $i /opt/cni/bin; done
|
||||
|
||||
**Notes about Weave Net add-on**
|
||||
|
||||
The Weave Net add-on works by default when the above configuration is done.
|
||||
|
||||
**Notes about flannel add-on**
|
||||
|
||||
If you choose the `flannel` add-on, then you must add the following to the
|
||||
`kubeadm init` command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
--pod-network-cidr 10.244.0.0/16
|
||||
|
||||
If you are using CRI-O and `flannel` and you want to use Kata Containers,
|
||||
edit the :file:`/etc/crio/crio.conf` file to add:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[crio.runtime]
|
||||
manage_network_ns_lifecycle = true
|
||||
|
||||
Use your cluster
|
||||
****************
|
||||
|
||||
Once your master control plane is successfully initialized, instructions on
|
||||
how to use your cluster and its *IP*, *token*, and *hash* values are
|
||||
displayed. It is important that you record the cluster values because they
|
||||
are needed when joining worker nodes to the cluster. Some values have a valid
|
||||
period. The values are presented in a format similar to:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert-hash <hash>
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You've successfully installed and set up Kubernetes in |CL| using CRI-O and
|
||||
kata-runtime. You are now ready to follow on-screen instructions to deploy a
|
||||
pod network to the cluster and join worker nodes with the displayed token
|
||||
and IP information.
|
||||
|
||||
Related topics
|
||||
**************
|
||||
|
||||
Read the Kubernetes documentation to learn more about:
|
||||
|
||||
* `Understanding basic Kubernetes architecture`_
|
||||
|
||||
* `Deploying an application to your cluster`_
|
||||
|
||||
* Installing a `pod network add-on`_
|
||||
|
||||
* `Joining your nodes`_
|
||||
|
||||
Cloud native setup automation (optional)
|
||||
****************************************
|
||||
|
||||
Clone the `cloud-native-setup`_ repository on your system and follow the instructions. This repository includes helper scripts to automate configuration.
|
||||
|
||||
Package configuration customization (optional)
|
||||
**********************************************
|
||||
|
||||
|CL| is a stateless system that looks for user-defined package configuration
|
||||
files in the :file:`/etc/<package-name>` directory to be used as default. If
|
||||
user-defined files are not found, |CL| uses the distribution-provided
|
||||
configuration files for each package.
|
||||
|
||||
If you customize any of the default package configuration files, you **must**
|
||||
store the customized files in the :file:`/etc/` directory. If you edit any of
|
||||
the distribution-provided default files, your changes will be lost in the
|
||||
next system update.
|
||||
|
||||
For example, to customize CRI-O configuration in your system, run the
|
||||
following commands:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo mkdir /etc/crio
|
||||
sudo cp /usr/share/defaults/crio/crio.conf /etc/crio/
|
||||
sudo $EDITOR /etc/crio/crio.conf
|
||||
|
||||
Learn more about `Stateless`_ in |CL| and view the |CL| `documentation`_.
|
||||
|
||||
Proxy configuration (optional)
|
||||
******************************
|
||||
|
||||
If you use a proxy server, you must set your proxy environment variables and
|
||||
create an appropriate proxy configuration file for both CRI-O services. Consult your IT department if you are behind a corporate proxy for
|
||||
the appropriate values. Ensure that your local IP is **explicitly included**
|
||||
in the environment variable *NO_PROXY*. (Setting *localhost* is not enough.)
|
||||
|
||||
If you have already set your proxy environment variables, run the following
|
||||
commands as a shell script to configure all of these services in one step:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
services=('crio')
|
||||
for s in "${services[@]}"; do
|
||||
sudo mkdir -p "/etc/systemd/system/${s}.service.d/"
|
||||
cat << EOF | sudo tee "/etc/systemd/system/${s}.service.d/proxy.conf"
|
||||
[Service]
|
||||
Environment="HTTP_PROXY=${http_proxy}"
|
||||
Environment="HTTPS_PROXY=${https_proxy}"
|
||||
Environment="SOCKS_PROXY=${socks_proxy}"
|
||||
Environment="NO_PROXY=${no_proxy}"
|
||||
EOF
|
||||
done
|
||||
|
||||
Next steps
|
||||
**********
|
||||
|
||||
:ref:`kubernetes-bp`
|
||||
|
||||
|
||||
Troubleshooting
|
||||
***************
|
||||
|
||||
* <HOSTNAME> not found in <IP> message.
|
||||
|
||||
Your DNS server may not be appropriately configured. Try adding an
|
||||
entry to the :file:`/etc/hosts` file with your host's IP and Name.
|
||||
|
||||
For example: 100.200.50.20 myhost
|
||||
|
||||
Use the commands :command:`hostname` and :command:`hostname -I` to retrieve them.
|
||||
|
||||
* Images cannot be pulled.
|
||||
|
||||
You may be behind a proxy server. Try configuring your proxy settings,
|
||||
using the environment variables *HTTP_PROXY*, *HTTPS_PROXY*, and *NO_PROXY*
|
||||
as required in your environment.
|
||||
|
||||
* Connection refused error.
|
||||
|
||||
If you are behind a proxy server, you may need to add the master's IP to
|
||||
the environment variable *NO_PROXY*.
|
||||
|
||||
* Connection timed-out or Access Refused errors.
|
||||
|
||||
You must ensure that the appropriate proxy settings are available from the
|
||||
same terminal where you will initialize the control plane. To verify the
|
||||
proxy settings that Kubernetes will actually use, run the commands:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
echo $HTTP_PROXY
|
||||
echo $HTTPS_PROXY
|
||||
echo $NO_PROXY
|
||||
|
||||
If the displayed proxy values are different from your assigned values, the
|
||||
cluster initialization will fail. Contact your IT support team to learn how
|
||||
to set the proxy variables permanently, and how to make them available for
|
||||
all the types of access that you will use, such as remote SSH access.
|
||||
|
||||
If the result of the above commands is blank, you may need to add a
|
||||
``profile`` to the :file:`/etc` directory. To do so, follow these steps.
|
||||
|
||||
#. Create a `profile` in :file:`/etc`
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo touch profile
|
||||
|
||||
#. With a preferred editor, open `profile`, and enter your proxy settings.
|
||||
Example shown below.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export "HTTP_PROXY=http://proxy.example.com:443"
|
||||
export "HTTPS_PROXY=http://proxy.example.com:445"
|
||||
export "SOCKS_PROXY=http://proxy.example.com:1080"
|
||||
export "NO_PROXY= site.com,.site.com,localhost,127.0.0.1,<master IP>
|
||||
|
||||
.. note::
|
||||
|
||||
<master IP> can be obtained by running :command:`ifconfig`.
|
||||
|
||||
#. Save and exit the `profile`.
|
||||
|
||||
#. Run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo source profile
|
||||
|
||||
#. To ensure your system isn't running previous session variables, run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo kubeadm reset --cri-socket=/run/crio/crio.sock
|
||||
|
||||
#. Continue below while passing `-E` in the command as shown.
|
||||
|
||||
* Missing environment variables.
|
||||
|
||||
If you are behind a proxy server, pass environment variables by adding *-E*
|
||||
to the command that initializes the master control plane.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
/* Kubernetes with CRI-O + kata-runtime */
|
||||
sudo -E kubeadm init --cri-socket=/run/crio/crio.sock
|
||||
|
||||
.. _Kubernetes container orchestration system: https://kubernetes.io/
|
||||
|
||||
.. _Kata Containers: https://katacontainers.io/
|
||||
|
||||
.. _Software Update documentation: https://clearlinux.org/documentation/clear-linux/concepts/swupd-about#updating
|
||||
|
||||
.. _cloud-native-basic: https://github.com/clearlinux/clr-bundles/blob/master/bundles/cloud-native-basic
|
||||
|
||||
.. _preflight check: https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#preflight-checks
|
||||
|
||||
.. _Understanding basic Kubernetes architecture: https://kubernetes.io/docs/user-journeys/users/application-developer/foundational/#section-3
|
||||
|
||||
.. _Deploying an application to your cluster: https://kubernetes.io/docs/user-journeys/users/application-developer/foundational/#section-2
|
||||
|
||||
.. _pod network add-on: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
|
||||
|
||||
.. _Joining your nodes: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#join-nodes
|
||||
|
||||
.. _Stateless: https://clearlinux.org/features/stateless
|
||||
|
||||
.. _documentation: https://clearlinux.org/documentation/clear-linux
|
||||
|
||||
.. _cloud-native-setup: https://github.com/clearlinux/cloud-native-setup
|
||||
|
||||
|
After Width: | Height: | Size: 58 KiB |
|
After Width: | Height: | Size: 63 KiB |
|
After Width: | Height: | Size: 136 KiB |
|
After Width: | Height: | Size: 141 KiB |
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 139 KiB |
|
After Width: | Height: | Size: 96 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 15 KiB |
@@ -0,0 +1,242 @@
|
||||
.. _machine-learning:
|
||||
|
||||
TensorFlow\* machine learning on |CL-ATTR|
|
||||
##########################################
|
||||
|
||||
This tutorial will demonstrate the installation and execusion of a TensorFlow\*
|
||||
machine learning example on |CL-ATTR|. It uses a Jupyter\* Notebook and MNIST
|
||||
data for handwriting recognition.
|
||||
|
||||
The initial steps will have you set up a Jupyter kernel and run a Notebook
|
||||
on a bare-metal |CL| system.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
This tutorial assumes you have installed |CL| on your host system. For
|
||||
detailed instructions on installing |CL| on a bare metal system, follow the
|
||||
:ref:`bare metal installation tutorial<bare-metal-install-desktop>`.
|
||||
|
||||
Before you install any new packages, update |CL| with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
Once your system is updated, add the following bundles to your system:
|
||||
|
||||
* `machine-learning-web-ui`: This bundle contains the Jupyter application.
|
||||
|
||||
* `machine-learning-basic`: This bundle contains TensorFlow and other useful
|
||||
tools.
|
||||
|
||||
To install the bundles, run the following commands in your :file:`$HOME`
|
||||
directory:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd bundle-add machine-learning-web-ui
|
||||
|
||||
sudo swupd bundle-add machine-learning-basic
|
||||
|
||||
Set up a Jupyter notebook
|
||||
*************************
|
||||
|
||||
With all required packages and libraries installed, set up the file structure
|
||||
for the Jupyter Notebook.
|
||||
|
||||
#. In the :file:`$HOME` directory, create a directory for the Jupyter
|
||||
Notebooks named :file:`Notebooks`.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mkdir Notebooks
|
||||
|
||||
#. Within :file:`Notebooks`, create a directory named :file:`Handwriting`.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mkdir Notebooks/Handwriting
|
||||
|
||||
#. Change to the new directory.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd Notebooks/Handwriting
|
||||
|
||||
#. Copy the :file:`MNIST_example.ipynb` file into the :file:`Handwriting`
|
||||
directory.
|
||||
|
||||
.. note::
|
||||
After installing the `machine-learning basic` bundle, you can find the
|
||||
example code under
|
||||
:file:`/usr/share/doc/tensorflow/MNIST_example.ipynb`.
|
||||
|
||||
|
||||
The example code downloads and decompresses the MNIST data directly into the
|
||||
:file:`./mnist` directory. Alternatively, download the four files directly
|
||||
from the Yann LeCun’s `MNIST Database website`_ and save them into a
|
||||
:file:`mnist` directory within the :file:`Handwriting` directory.
|
||||
|
||||
The files needed are:
|
||||
|
||||
* `train-images-idx3-ubyte.gz`_: Training set images (9912422 bytes)
|
||||
|
||||
* `train-labels-idx1-ubyte.gz`_: Training set labels (28881 bytes)
|
||||
|
||||
* `t10k-images-idx3-ubyte.gz`_: Test set images (1648877 bytes)
|
||||
|
||||
* `t10k-labels-idx1-ubyte.gz`_: Test set labels (4542 bytes)
|
||||
|
||||
Run the Jupyter machine learning example code
|
||||
*********************************************
|
||||
|
||||
With |CL|, Jupyter, and TensorFlow installed and configured, we can
|
||||
run the example code.
|
||||
|
||||
#. Go to the :file:`($HOME)/Notebooks` directory and start Jupyter with the
|
||||
following commands:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd ~/Notebooks
|
||||
|
||||
jupyter notebook
|
||||
|
||||
The Jupyter server starts and opens a web browser showing the Jupyter file
|
||||
manager with a list of files in the current directory, see figure 1.
|
||||
|
||||
.. figure:: figures/machine-learning-1.png
|
||||
:alt: Jupyter file manager
|
||||
|
||||
Figure 1: The Jupyter file manager shows the list of available files.
|
||||
|
||||
#. Click on the :file:`Handwriting` directory. The :file:`MNIST_example.ipynb`
|
||||
file created earlier should be listed there, see figure 2.
|
||||
|
||||
.. figure:: figures/machine-learning-2.png
|
||||
:alt: Example file within the Jupyter file manager
|
||||
|
||||
Figure 2: The example file within the Jupyter file manager.
|
||||
|
||||
#. To run the hand writing example, click on the :file:`MNIST_example.ipynb`
|
||||
file to load the notebook, see figure 3.
|
||||
|
||||
.. figure:: figures/machine-learning-3.png
|
||||
:alt: The loaded MNIST_example notebook
|
||||
|
||||
Figure 3: The loaded MNIST_example notebook within the Jupyter file
|
||||
manager.
|
||||
|
||||
#. Click the |run-cell| button to execute the code in the current cell and
|
||||
move to the next.
|
||||
|
||||
#. Select the :guilabel:`In [2]` cell and click the |run-cell| button to load
|
||||
the MNIST data. The successful output is shown on figure 4.
|
||||
|
||||
.. figure:: figures/machine-learning-4.png
|
||||
:alt: Successful import of MNIST data
|
||||
|
||||
Figure 4: Output after successfully importing the MNIST data.
|
||||
|
||||
|
||||
After the MNIST data was successfully downloaded and extracted into the
|
||||
:file:`mnist` directory within the :file:`($HOME)/Notebooks/Handwriting`
|
||||
directory, four .gz files are present and the four data sets were created:
|
||||
`trainX`, `trainY`, `testX` and `testY`.
|
||||
|
||||
#. To inspect the imported data, the function in :guilabel:`In [3]` first
|
||||
instructs Jupyter to reshape the data into an array of 28 x 28 images and to
|
||||
plot the area in a 28 x 28 grid. Click the |run-cell| button twice to show
|
||||
the first two digits in the `trainX` dataset, see figure 5.
|
||||
|
||||
.. figure:: figures/machine-learning-5.png
|
||||
:alt: Function to reshape data.
|
||||
|
||||
Figure 5: A function reshapes the data and displays the first two
|
||||
digits in the `trainX` dataset.
|
||||
|
||||
#. The :guilabel:`In [4]` cell defines the neural network. It provides the
|
||||
inputs, defines the hidden layers, runs the training model, and sets up
|
||||
the output layer, see figure 6. Click the |run-cell| button four times to
|
||||
perform these operations.
|
||||
|
||||
.. figure:: figures/machine-learning-6.png
|
||||
:alt: Defining, building and training the neural network model
|
||||
|
||||
Figure 6: Defining, building and training the neural network model.
|
||||
|
||||
#. To test the accuracy of the prediction the system makes, select the
|
||||
:guilabel:`In [8]` cell and click the |run-cell| button. In this example,
|
||||
the number 6 was predicted with a 99% accuracy, see figure 7.
|
||||
|
||||
.. figure:: figures/machine-learning-7.png
|
||||
:alt: Prediction example
|
||||
|
||||
Figure 7: The system predicts a number providing the accuracy of the
|
||||
prediction.
|
||||
|
||||
.. note::
|
||||
|
||||
To retest the accuracy of a random data point's prediction, run the
|
||||
cell :guilabel:`In [8]` again. It will take another random data point
|
||||
and predict its value.
|
||||
|
||||
#. To check the accuracy for the whole dataset, select the :guilabel:`In [10]`
|
||||
cell and click the |run-cell| button. Our example's accuracy is
|
||||
calculated as 97.17%, see figure 8.
|
||||
|
||||
.. figure:: figures/machine-learning-8.png
|
||||
:alt: System's accuracy
|
||||
|
||||
Figure 8: The system's accuracy for the entire data set.
|
||||
|
||||
For more in-depth information on the model used and the mathematics it entails,
|
||||
visit the TensorFlow tutorials
|
||||
`TensorFlow MNIST beginners demo`_ and `TensorFlow MNIST pros demo`_.
|
||||
|
||||
**Congratulations!**
|
||||
|
||||
You have successfully installed a Jupyter kernel on |CL|. In addition, you
|
||||
trained a neural network to successfully predict the values contained in a
|
||||
data set of hand-written number images.
|
||||
|
||||
Related topics
|
||||
**************
|
||||
|
||||
* `MNIST Database website`_
|
||||
* `TensorFlow MNIST beginners demo`_
|
||||
* `TensorFlow MNIST pros demo`_
|
||||
* `Jupyter main website`_
|
||||
* `Jupyter documentation`_
|
||||
* `MNIST at Wikipedia`_
|
||||
|
||||
.. _MNIST Database website:
|
||||
http://yann.lecun.com/exdb/mnist/
|
||||
|
||||
.. _train-images-idx3-ubyte.gz:
|
||||
http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
|
||||
|
||||
.. _train-labels-idx1-ubyte.gz:
|
||||
http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
|
||||
|
||||
.. _t10k-images-idx3-ubyte.gz:
|
||||
http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
|
||||
|
||||
.. _t10k-labels-idx1-ubyte.gz:
|
||||
http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
|
||||
|
||||
.. _Jupyter main website: http://jupyter.org/
|
||||
|
||||
.. _Jupyter documentation: https://jupyter.readthedocs.io/en/latest/index.html
|
||||
|
||||
.. _TensorFlow MNIST beginners demo:
|
||||
https://www.tensorflow.org/get_started/mnist/beginners
|
||||
|
||||
.. _TensorFlow MNIST pros demo:
|
||||
https://www.tensorflow.org/get_started/mnist/pros
|
||||
|
||||
.. _MNIST at Wikipedia:
|
||||
https://en.wikipedia.org/wiki/MNIST_database
|
||||
|
||||
.. |run-cell| image:: ./figures/run-cell-button.png
|
||||
|
After Width: | Height: | Size: 40 KiB |
|
After Width: | Height: | Size: 70 KiB |
|
After Width: | Height: | Size: 54 KiB |
|
After Width: | Height: | Size: 18 KiB |
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 27 KiB |
|
After Width: | Height: | Size: 83 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 77 KiB |
|
After Width: | Height: | Size: 96 KiB |
|
After Width: | Height: | Size: 126 KiB |
|
After Width: | Height: | Size: 123 KiB |
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 62 KiB |
|
After Width: | Height: | Size: 204 KiB |
|
After Width: | Height: | Size: 250 KiB |
|
After Width: | Height: | Size: 152 KiB |
|
After Width: | Height: | Size: 122 KiB |
|
After Width: | Height: | Size: 100 KiB |
|
After Width: | Height: | Size: 104 KiB |
|
After Width: | Height: | Size: 111 KiB |
|
After Width: | Height: | Size: 108 KiB |
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 197 KiB |
|
After Width: | Height: | Size: 173 KiB |
|
After Width: | Height: | Size: 173 KiB |
|
After Width: | Height: | Size: 82 KiB |
|
After Width: | Height: | Size: 117 KiB |
|
After Width: | Height: | Size: 142 KiB |
|
After Width: | Height: | Size: 128 KiB |
|
After Width: | Height: | Size: 131 KiB |
|
After Width: | Height: | Size: 79 KiB |
|
After Width: | Height: | Size: 159 KiB |
|
After Width: | Height: | Size: 122 KiB |
|
After Width: | Height: | Size: 128 KiB |
@@ -0,0 +1,140 @@
|
||||
.. _multi-boot-mint:
|
||||
|
||||
Install Linux Mint\* 18.1 *Serena* MATE
|
||||
#######################################
|
||||
|
||||
This guide describes Linux Mint-specific details of the :ref:`multi-boot`
|
||||
tutorial.
|
||||
|
||||
#. Start the Mint installer and follow the prompts.
|
||||
|
||||
#. At the :guilabel:`Installation type` screen, choose
|
||||
:guilabel:`Something else`. See Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-mint-1.png
|
||||
|
||||
Figure 1: Mint: Installation type.
|
||||
|
||||
#. Create a new root partition.
|
||||
|
||||
#. Under the :guilabel:`Device` column, select :guilabel:`free space`. See
|
||||
Figure 2.
|
||||
|
||||
.. figure:: figures/multi-boot-mint-2.png
|
||||
|
||||
Figure 2: Mint: Add partition.
|
||||
|
||||
#. Click the :guilabel:`+` button.
|
||||
|
||||
#. In the :guilabel:`Size` field, enter a value for the new partition
|
||||
size. For this example, we used *40000 MB*, as shown in Figure 3.
|
||||
|
||||
.. figure:: figures/multi-boot-mint-3.png
|
||||
|
||||
Figure 3: Mint: Configure new partition settings.
|
||||
|
||||
#. Set :guilabel:`Use as` to :guilabel:`Ext4 journaling file system`.
|
||||
|
||||
#. Set the :guilabel:`Mount point` to :guilabel:`/`.
|
||||
|
||||
#. Click :guilabel:`OK`.
|
||||
|
||||
#. Share the swap partition created by |CL|.
|
||||
|
||||
#. Under the :guilabel:`Device` column, select :file:`/dev/sda2`.
|
||||
|
||||
#. Click :guilabel:`Change`.
|
||||
|
||||
#. Confirm :guilabel:`Use as` is set to :guilabel:`Swap area`. See Figure 4.
|
||||
|
||||
.. figure:: figures/multi-boot-mint-4.png
|
||||
|
||||
Figure 4: Mint: Set swap partition.
|
||||
|
||||
#. Follow the remaining prompts to complete the Mint installation.
|
||||
|
||||
#. At this point, you cannot boot |CL| because `Grub`
|
||||
is the default boot loader. Follow these steps to make the |CL|
|
||||
Systemd-Boot the default boot loader and add Mint as a boot option.
|
||||
|
||||
#. Boot into Mint.
|
||||
|
||||
#. Log in.
|
||||
|
||||
#. Locate the Mint :file:`grub.cfg` file in the :file:`/boot/grub/`
|
||||
directory and look for the :guilabel:`menuentry` section. In Figure 5, the
|
||||
highlighted lines identify the kernel, the :file:`initrd` files, the root
|
||||
partition UUID, and the additional parameters used. Use this information
|
||||
to create a new Systemd-Boot entry for Mint.
|
||||
|
||||
.. figure:: figures/multi-boot-mint-5.png
|
||||
|
||||
Figure 5: Mint: grub.cfg file.
|
||||
|
||||
#. Copy the kernel and :file:`initrd` file to the EFI partition.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo cp /boot/vmlinuz-4.4.0-53-generic /boot/efi
|
||||
|
||||
sudo cp /boot/initrd.img-4.4.0-53-generic /boot/efi
|
||||
|
||||
#. Create a boot entry for Mint. At a minimum, the file must contain
|
||||
these settings:
|
||||
|
||||
+---------+------------------------------------+
|
||||
| Setting | Description |
|
||||
+=========+====================================+
|
||||
| title | Text to show in the boot menu |
|
||||
+---------+------------------------------------+
|
||||
| linux | Linux kernel image |
|
||||
+---------+------------------------------------+
|
||||
| initrd | initramfs image |
|
||||
+---------+------------------------------------+
|
||||
| options | Options to pass to the EFI program |
|
||||
| | or kernel boot parameters |
|
||||
+---------+------------------------------------+
|
||||
|
||||
See the `systemd boot loader documentation`_ for additional
|
||||
details.
|
||||
|
||||
The *options* parameters must specify the root partition UUID and
|
||||
any additional parameters that Mint requires.
|
||||
|
||||
.. note:: The root partition UUID used below is unique to this example.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudoedit /boot/efi/loader/entries/mint.conf
|
||||
|
||||
Add the following lines to the :file:`mint.conf` file:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
title Mint 18.1 Serena MATE
|
||||
|
||||
linux /vmlinuz-4.4.0-53-generic
|
||||
|
||||
initrd /initrd.img-4.4.0-53-generic
|
||||
|
||||
options root=UUID=af4901e1-6238-470a-8c14-bc0f0f7715ec ro
|
||||
|
||||
#. Re-install Systemd-Boot to make it the default boot loader.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo bootctl install --path /boot/efi
|
||||
|
||||
.. note::
|
||||
If an older version of Mint does not have the `bootctl` command,
|
||||
skip this step and see :ref:`multi-boot-restore-bl` to restore
|
||||
Systemd-Boot.
|
||||
|
||||
#. Reboot.
|
||||
|
||||
If you want to install other :abbr:`OSes (operating systems)`, refer to
|
||||
:ref:`multi-boot` for details.
|
||||
|
||||
|
||||
.. _systemd boot loader documentation:
|
||||
https://wiki.archlinux.org/index.php/Systemd-boot
|
||||
@@ -0,0 +1,58 @@
|
||||
.. _multi-boot-restore-bl:
|
||||
|
||||
Restore the |CL-ATTR| boot loader
|
||||
###################################
|
||||
|
||||
This guide is part of the :ref:`multi-boot` tutorial. If you install a new
|
||||
:abbr:`OS (operating system)` or upgrade an existing OS, the default boot
|
||||
loader may change from |CL| Systemd-Boot. This guide describes how to restore
|
||||
Systemd-Boot.
|
||||
|
||||
#. Boot the |CL| installer from a USB thumb drive. See :ref:`bootable-usb`.
|
||||
|
||||
#. At the introduction screen, press :kbd:`Control+Alt+F2` to bring up the
|
||||
|CL| console. See Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-restore-bl-1.png
|
||||
|
||||
Figure 1: |CL|: Console.
|
||||
|
||||
#. Log in as *root*.
|
||||
|
||||
.. note::
|
||||
When you log in for the first time as *root* through the console, you must
|
||||
set a new password.
|
||||
|
||||
#. Find the location of the |CL| EFI partition. In this example, it is
|
||||
:file:`/dev/sda3`. See Figure 2.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
fdisk -l
|
||||
|
||||
.. figure:: figures/multi-boot-restore-bl-2.png
|
||||
|
||||
Figure 2: |CL|: fdisk -l command.
|
||||
|
||||
#. Mount the EFI partition.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mount /dev/sda3 /mnt
|
||||
|
||||
#. Re-install Systemd-Boot to make it the default boot loader.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
bootctl install --path /mnt
|
||||
|
||||
#. Unmount the EFI partition.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
umount /mnt
|
||||
|
||||
#. Reboot.
|
||||
|
||||
If you want to install other :abbr:`OSes (operating systems)`, refer to
|
||||
:ref:`multi-boot` for details.
|
||||
@@ -0,0 +1,165 @@
|
||||
.. _multi-boot-rhel:
|
||||
|
||||
Install Red Hat\* Enterprise Linux 7.4 Beta
|
||||
###########################################
|
||||
|
||||
This guide describes Red Hat-specific details of the :ref:`multi-boot`
|
||||
tutorial.
|
||||
|
||||
#. Start the Red Hat installer and follow the prompts.
|
||||
|
||||
#. At the :guilabel:`INSTALLATION SUMMARY` screen, choose
|
||||
:guilabel:`INSTALLATION DESTINATION`. See Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-1.png
|
||||
|
||||
Figure 1: Red Hat: Installation summary.
|
||||
|
||||
#. In the :guilabel:`Device Selection` section, select a drive on which to
|
||||
install the OS. See Figure 2.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-2.png
|
||||
|
||||
Figure 2: Red Hat: Installation destination.
|
||||
|
||||
#. Under the :guilabel:`Other Storage Options` section, choose
|
||||
:guilabel:`I will configure partitioning`. See Figure 2.
|
||||
|
||||
#. Click :guilabel:`Done`.
|
||||
|
||||
#. Under the :menuselection:`New Red Hat Enterprise Linux 7.4 Installation
|
||||
--> New mount points will use the following partitioning scheme` section,
|
||||
select :menuselection:`Standard Partition` from the drop down list. See
|
||||
Figure 3.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-3.png
|
||||
|
||||
Figure 3: Red Hat: New partition scheme.
|
||||
|
||||
#. Create a new root partition.
|
||||
|
||||
#. Click the :menuselection:`+` button on the lower left corner.
|
||||
|
||||
#. Enter `/` and the new partition size. For this example, we specified 45
|
||||
GB. See Figure 4.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-4.png
|
||||
|
||||
Figure 4: Red Hat: Create new root partition.
|
||||
|
||||
#. Click :guilabel:`Add mount point`.
|
||||
|
||||
#. Share the swap partition that was created by |CL|. See Figure 5.
|
||||
|
||||
#. Expand :guilabel:`Unknown`.
|
||||
|
||||
#. Select :guilabel:`swap / sda2`.
|
||||
|
||||
#. Select :guilabel:`Reformat`.
|
||||
|
||||
#. Click :guilabel:`Update Settings`.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-5.png
|
||||
|
||||
Figure 5: Red Hat: Configure swap partition.
|
||||
|
||||
#. Share the EFI partition that was created by |CL|. See Figure 6.
|
||||
|
||||
#. Expand :guilabel:`Unknown.`
|
||||
|
||||
#. Select :guilabel:`EFI System Partition / sda3`.
|
||||
|
||||
#. Under :guilabel:`Mount Point`, enter `/boot/efi`.
|
||||
|
||||
#. Click :guilabel:`Update Settings`.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-6.png
|
||||
|
||||
Figure 6: Red Hat: Configure EFI partition.
|
||||
|
||||
#. Click :guilabel:`Done`.
|
||||
|
||||
#. Follow the remaining prompts to complete the Red Hat installation.
|
||||
|
||||
#. At this point, you cannot boot |CL| because `Grub` is the default boot
|
||||
loader. Follow these steps to make the |CL| Systemd-Boot the default boot
|
||||
loader and add Red Hat as a boot option:
|
||||
|
||||
#. Boot into Red Hat.
|
||||
|
||||
#. Log in.
|
||||
|
||||
#. Locate the Red Hat :file:`grub.cfg` file in the
|
||||
:file:`/boot/efi/EFI/redhat/` directory and look for the primary Red
|
||||
Hat :guilabel:`menuentry` section. In Figure 7, the highlighted lines
|
||||
identify the kernel and `initrd` filenames, root partition UUID, and
|
||||
additional parameters used. Use this information to create a
|
||||
new Systemd-Boot entry for Red Hat.
|
||||
|
||||
.. figure:: figures/multi-boot-rhel-7.png
|
||||
|
||||
Figure 7: Red Hat: grub.cfg file.
|
||||
|
||||
#. Copy the kernel and :file:`initrd` file to the EFI partition.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo cp /boot/vmlinuz-3.10.0-663.el7.x86_64 /boot/efi
|
||||
|
||||
sudo cp /boot/initramfs-3.10.0-663.el7.x86_64.img /boot/efi
|
||||
|
||||
#. Create a boot entry for Red Hat. At a minimum, the file must contain
|
||||
these settings:
|
||||
|
||||
+---------+---------------------------------------------------+
|
||||
| Setting | Description |
|
||||
+=========+===================================================+
|
||||
| title | Text to show in the boot menu |
|
||||
+---------+---------------------------------------------------+
|
||||
| linux | Linux kernel image |
|
||||
+---------+---------------------------------------------------+
|
||||
| initrd | initramfs image |
|
||||
+---------+---------------------------------------------------+
|
||||
| options | Options to pass to the EFI program or kernel boot |
|
||||
| | parameters |
|
||||
+---------+---------------------------------------------------+
|
||||
|
||||
See the `systemd boot loader documentation`_ for additional
|
||||
details.
|
||||
|
||||
The *options* parameters must specify the root partition UUID and any
|
||||
additional parameters that Red Hat requires.
|
||||
|
||||
.. note:: The root partition UUID used below is unique to this example.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudoedit /boot/efi/loader/entries/redhat.conf
|
||||
|
||||
Add the following lines to the :file:`redhat.conf` file:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
title Red Hat Enterprise Linux 7.4 Beta
|
||||
|
||||
linux /vmlinuz-3.10.0-663.el7.x86_64
|
||||
|
||||
initrd /initramfs-3.10.0-663.el7.x86_64.img
|
||||
|
||||
options root=UUID=30655c74-6cc1-4c55-8fcc-ac8bddcea4db ro
|
||||
crashkernel=auto rhgb LANG=en_US.UTF-8
|
||||
|
||||
#. Re-install Systemd-Boot to make it the default boot loader.
|
||||
|
||||
.. note::
|
||||
This version of Red Hat does not support `bootctl install`. Perform
|
||||
the steps in :ref:`multi-boot-restore-bl` instead.
|
||||
|
||||
#. Reboot.
|
||||
|
||||
If you want to install other :abbr:`OSes (operating systems)`, refer to
|
||||
:ref:`multi-boot` for details.
|
||||
|
||||
|
||||
.. _systemd boot loader documentation:
|
||||
https://wiki.archlinux.org/index.php/Systemd-boot
|
||||
@@ -0,0 +1,124 @@
|
||||
.. _multi-boot-sles:
|
||||
|
||||
Install SUSE\* Linux Enterprise Server 12 SP2
|
||||
#############################################
|
||||
|
||||
This guide describes SUSE-specific details of the :ref:`multi-boot`
|
||||
tutorial.
|
||||
|
||||
#. Start the SUSE installer and follow the prompts.
|
||||
|
||||
#. At the :guilabel:`Suggested Partitioning` screen, choose
|
||||
:guilabel:`Expert Partitioner`. See Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-sles-1.png
|
||||
|
||||
Figure 1: SUSE: Suggested partitioning.
|
||||
|
||||
**Optional:** Under :guilabel:`Available Storage on Linux` section,
|
||||
right-click the SUSE :file:`/home` partition and delete it. In this example, it is :file:`/dev/sda8`. See Figure 2.
|
||||
|
||||
.. figure:: figures/multi-boot-sles-2.png
|
||||
|
||||
Figure 2: SUSE: Delete /home partition.
|
||||
|
||||
#. Under :guilabel:`Available Storage on Linux` section, right-click the SUSE
|
||||
root partition and resize it. In this example, :file:`/dev/sda7` is
|
||||
resized to 45 GB. See Figure 3.
|
||||
|
||||
.. figure:: figures/multi-boot-sles-3.png
|
||||
|
||||
Figure 3: SUSE: Resize root partition.
|
||||
|
||||
#. Click :guilabel:`Accept`.
|
||||
|
||||
#. Follow the remaining prompts to complete the SUSE installation.
|
||||
|
||||
#. At this point, you cannot boot |CL| because `Grub`
|
||||
is the default boot loader. Follow these steps to make the |CL|
|
||||
Systemd-Boot the default boot loader and add SUSE as a boot option:
|
||||
|
||||
#. Boot into SUSE.
|
||||
|
||||
#. Log in.
|
||||
|
||||
#. Locate the SUSE :file:`grub.cfg` file in the :file:`/boot/grub2/` directory
|
||||
and look for the primary SUSE :guilabel:`menuentry` section. In Figure 4, the
|
||||
highlighted lines identify the kernel, the :file:`initrd` filenames, the
|
||||
root partition UUID, and the additional parameters used. Use this information
|
||||
to create a new Systemd-Boot entry for SUSE.
|
||||
|
||||
.. figure:: figures/multi-boot-sles-4.png
|
||||
|
||||
Figure 4: SUSE: grub.cfg file.
|
||||
|
||||
#. Copy the kernel and the :file:`initrd` file to the EFI partition.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo cp /boot/vmlinuz-4.4.21-69-default /boot/efi
|
||||
|
||||
sudo cp /boot/initrd-4.4.21-69-default /boot/efi
|
||||
|
||||
#. Create a boot entry for SUSE. At a minimum, the file must contain
|
||||
these settings:
|
||||
|
||||
+---------+---------------------------------------+
|
||||
| Setting | Description |
|
||||
+=========+=======================================+
|
||||
| title | Text to show in the boot menu |
|
||||
+---------+---------------------------------------+
|
||||
| linux | Linux kernel image |
|
||||
+---------+---------------------------------------+
|
||||
| initrd | initramfs image |
|
||||
+---------+---------------------------------------+
|
||||
| options | Options to pass to the EFI program or |
|
||||
| | kernel boot parameters |
|
||||
+---------+---------------------------------------+
|
||||
|
||||
See the `systemd boot loader documentation`_ for additional
|
||||
details.
|
||||
|
||||
The *options* parameter must specify the root partition UUID and
|
||||
any additional parameters SUSE requires.
|
||||
|
||||
.. note:: The root partition UUID used below is unique to this example.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudoedit /boot/efi/loader/entries/suse.conf
|
||||
|
||||
Add the following lines to the :file:`suse.conf` file:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
title SUSE Linux Enterprise 12 SP2
|
||||
|
||||
linux /vmlinuz-4.4.21-69-default
|
||||
|
||||
initrd /initrd-4.4.21-69-default
|
||||
|
||||
options root=UUID=b9e25e98-a644-4ac3-b955-ae32800ee350 ro
|
||||
resume=/dev/disk/by-uuid/6a50c032-1c1e-4b4a-b799-ca365bb10dc7
|
||||
splash=silent showopts crashkernel=109M,high
|
||||
crashkernel=72M,low
|
||||
|
||||
#. Re-install Systemd-Boot to make it the default boot loader.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo bootctl install --path /boot/efi
|
||||
|
||||
.. note::
|
||||
If an older version of SUSE does not have the `bootctl` command,
|
||||
skip this step and see :ref:`multi-boot-restore-bl` to restore
|
||||
Systemd-Boot.
|
||||
|
||||
#. Reboot.
|
||||
|
||||
If you want to install other :abbr:`OSes (operating systems)`, refer to
|
||||
:ref:`multi-boot` for details.
|
||||
|
||||
|
||||
.. _systemd boot loader documentation:
|
||||
https://wiki.archlinux.org/index.php/Systemd-boot
|
||||
@@ -0,0 +1,142 @@
|
||||
.. _multi-boot-ubuntu:
|
||||
|
||||
Install Ubuntu\* 16.04 LTS Desktop
|
||||
##################################
|
||||
|
||||
This guide describes Ubuntu-specific details of the :ref:`multi-boot`
|
||||
tutorial.
|
||||
|
||||
#. Start the Ubuntu installer and follow the prompts.
|
||||
|
||||
#. At the :guilabel:`Installation type` screen, choose
|
||||
:guilabel:`Something else`. See Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-ubuntu-1.png
|
||||
|
||||
Figure 1: Ubuntu: Installation type.
|
||||
|
||||
#. Create a new root partition.
|
||||
|
||||
#. Under the :guilabel:`Device` column, select :guilabel:`free space`. See
|
||||
Figure 2.
|
||||
|
||||
.. figure:: figures/multi-boot-ubuntu-2.png
|
||||
|
||||
Figure 2: Ubuntu: Add partition.
|
||||
|
||||
#. Click the :guilabel:`+` button on the lower left corner.
|
||||
|
||||
#. Enter the new partition size. For this example, we used *40000 MB*, as
|
||||
shown in Figure 3.
|
||||
|
||||
.. figure:: figures/multi-boot-ubuntu-3.png
|
||||
|
||||
Figure 3: Ubuntu: Configure new root partition.
|
||||
|
||||
#. Set :guilabel:`Use as` to :guilabel:`Ext4 journaling file system`.
|
||||
|
||||
#. Set the :guilabel:`Mount point` to `/`.
|
||||
|
||||
#. Click :guilabel:`OK`.
|
||||
|
||||
#. Under the :guilabel:`Format?` column, select the new partition to be
|
||||
formatted, in this example :file:`/dev/sda8`.
|
||||
|
||||
#. Share the swap partition that was created by |CL|.
|
||||
|
||||
#. Under the :guilabel:`Device` column, select :file:`/dev/sda2`.
|
||||
|
||||
#. Click :guilabel:`Change`.
|
||||
|
||||
#. Confirm :guilabel:`Use as` is set to :guilabel:`swap area`. See Figure 4.
|
||||
|
||||
.. figure:: figures/multi-boot-ubuntu-4.png
|
||||
|
||||
Figure 4: Ubuntu: Set swap partition.
|
||||
|
||||
#. Follow the remaining prompts to complete the Ubuntu installation.
|
||||
|
||||
#. At this point, you cannot boot |CL| because `Grub`
|
||||
is the default boot loader. Follow these steps to make the |CL|
|
||||
Systemd-Boot the default boot loader and add Ubuntu as a boot option:
|
||||
|
||||
#. Boot into Ubuntu.
|
||||
|
||||
#. Log in.
|
||||
|
||||
#. Locate the Ubuntu :file:`grub.cfg` file in the :file:`/boot/grub/`
|
||||
directory and look for the :guilabel:`menuentry` section. In Figure 5, the
|
||||
highlighted lines identify the kernel, the :file:`initrd` files, the
|
||||
root partition UUID, and the additional parameters used. Use this
|
||||
information to create a new Systemd-Boot entry for Ubuntu.
|
||||
|
||||
.. figure:: figures/multi-boot-ubuntu-5.png
|
||||
|
||||
Figure 5: Ubuntu: grub.cfg file.
|
||||
|
||||
#. Copy the kernel and the :file:`initrd` file to the EFI partition.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo cp /boot/vmlinuz-4.8.0-36-generic.efi.signed /boot/efi
|
||||
|
||||
sudo cp /boot/initrd.img-4.8.0-36-generic /boot/efi
|
||||
|
||||
#. Create a boot entry for Ubuntu. At a minimum, the file must contain
|
||||
these settings:
|
||||
|
||||
+---------+------------------------------------+
|
||||
| Setting | Description |
|
||||
+=========+====================================+
|
||||
| title | Text to show in the boot menu |
|
||||
+---------+------------------------------------+
|
||||
| linux | Linux kernel image |
|
||||
+---------+------------------------------------+
|
||||
| initrd | initramfs image |
|
||||
+---------+------------------------------------+
|
||||
| options | Options to pass to the EFI program |
|
||||
| | or kernel boot parameters |
|
||||
+---------+------------------------------------+
|
||||
|
||||
See the `systemd boot loader documentation`_ for additional
|
||||
details.
|
||||
|
||||
The *options* parameters must specify the root partition UUID and
|
||||
any additional parameters that Ubuntu requires.
|
||||
|
||||
.. note:: The root partition UUID used below is unique to this example.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudoedit /boot/efi/loader/entries/ubuntu.conf
|
||||
|
||||
Add the following lines to the :file:`ubuntu.conf` file:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
title Ubuntu 16.04 LTS Desktop
|
||||
|
||||
linux /vmlinuz-4.8.0-36-generic.efi.signed
|
||||
|
||||
initrd /initrd.img-4.8.0-36-generic
|
||||
|
||||
options root=UUID=17f0aa66-3467-4f99-b92c-8b2cea1045aa ro
|
||||
|
||||
#. Re-install Systemd-Boot to make it the default boot loader.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo bootctl install --path /boot/efi
|
||||
|
||||
.. note::
|
||||
If an older version of Ubuntu does not have the `bootctl` command,
|
||||
skip this step and see :ref:`multi-boot-restore-bl` to restore
|
||||
Systemd-Boot.
|
||||
|
||||
#. Reboot.
|
||||
|
||||
If you want to install other :abbr:`OSes (operating systems)`, refer to
|
||||
:ref:`multi-boot` for details.
|
||||
|
||||
.. _systemd boot loader documentation:
|
||||
https://wiki.archlinux.org/index.php/Systemd-boot
|
||||
@@ -0,0 +1,46 @@
|
||||
.. _multi-boot-win:
|
||||
|
||||
Install Windows\* Server 2016
|
||||
#############################
|
||||
|
||||
This guide describes Windows-specific details of the :ref:`multi-boot`
|
||||
tutorial.
|
||||
|
||||
#. Start the Windows installer and follow the prompts.
|
||||
|
||||
#. At the :guilabel:`Type of installation` screen, choose
|
||||
:guilabel:`Custom: Install Windows only (advanced)`. See Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-win-1.png
|
||||
|
||||
Figure 1: Windows: Choose installation type.
|
||||
|
||||
#. Select :guilabel:`Unallocated Space` and create a new partition of the
|
||||
desired size. In this example, we specified 50000 MB. See Figure 2.
|
||||
|
||||
.. figure:: figures/multi-boot-win-2.png
|
||||
|
||||
Figure 2: Windows: Create new partition.
|
||||
|
||||
.. note::
|
||||
Windows creates its own 100 MB EFI partition if none exists.
|
||||
In this example, Windows sees the EFI partition created during the
|
||||
|CL| installation and does not create one.
|
||||
|
||||
#. Select the newly created partition and follow the remaining prompts
|
||||
to complete the Windows installation. See Figure 3.
|
||||
|
||||
.. figure:: figures/multi-boot-win-3.png
|
||||
|
||||
Figure 3: Windows: Install on newly created partition.
|
||||
|
||||
#. Finish the Windows out-of-box-experience process.
|
||||
|
||||
#. At this point, you cannot boot |CL| because Windows is the
|
||||
default boot loader. See :ref:`multi-boot-restore-bl` to restore
|
||||
Systemd-Boot and add Windows to its boot menu.
|
||||
|
||||
|
||||
If you want to install other :abbr:`OSes (operating systems)`, refer to
|
||||
:ref:`multi-boot` for details.
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
.. _multi-boot:
|
||||
|
||||
Multi-boot |CL-ATTR| with other operating systems
|
||||
#################################################
|
||||
|
||||
|CL-ATTR| uses the Systemd-Boot boot loader, which does not support multi-
|
||||
booting without manual manipulation. This tutorial shows how to configure the
|
||||
|CL| boot loader to work with other :abbr:`OSes (operating systems)`.
|
||||
|
||||
Process overview
|
||||
****************
|
||||
|
||||
The process to install other operating systems for a multi-booting computer is
|
||||
described below. Install |CL| first, then install other operating systems in
|
||||
any order.
|
||||
|
||||
#. Install |CL| first with a EFI partition large enough to store the kernels
|
||||
of other operating systems and their initrds, in the case of Linux
|
||||
distributions.
|
||||
|
||||
#. Install the next operating system without creating its own EFI
|
||||
partition.
|
||||
|
||||
#. Boot into the newly installed operating system.
|
||||
|
||||
#. For Linux distributions, copy its kernel and `initrd` to the |CL| EFI
|
||||
partition. This step is not needed for Windows\*.
|
||||
|
||||
#. Add an entry for the newly installed operating system in the
|
||||
Systemd-Boot menu.
|
||||
|
||||
#. Make Systemd-Boot the default boot loader.
|
||||
|
||||
#. Repeat the previous steps to install each additional operating system.
|
||||
|
||||
If you update any installed operating systems, be aware that:
|
||||
|
||||
* The default boot loader may change from |CL| Systemd-Boot. Perform the
|
||||
steps in :ref:`multi-boot-restore-bl`.
|
||||
|
||||
* Linux kernels or `initrd` images may change. Keep their corresponding Systemd-Boot
|
||||
:file:`/boot/efi/loader/entries/*.conf` files up-to-date.
|
||||
|
||||
This process is not guaranteed to work with all Linux distributions and all
|
||||
their versions. The next section lists the OSes that we tested.
|
||||
|
||||
|
||||
Tested operating systems
|
||||
************************
|
||||
|
||||
The following operating systems were tested on an Intel® NUC6i7KYK with 32GB
|
||||
RAM and a 360GB SSD. Table 1 lists the information specific to the
|
||||
installation of the tested operating systems.
|
||||
|
||||
.. csv-table:: Table 1: OS specific installation information
|
||||
:header: # , OS, Version, Partition Size [#]_, Swap Size [#]_, EFI Partition Size [#]_, Download Link
|
||||
|
||||
1,Clear Linux,16140,50 GB,8 GB,1 GB,https://cdn.download.clearlinux.org/releases/16140/clear/
|
||||
2,Windows,Server 2016,50 GB,N/A,Shared with #1,https://www.microsoft.com/en-us/cloud-platform/windows-server
|
||||
3,Red Hat\*,Server 7.4 Beta,45 GB,Shared with #1,Shared with #1,https://access.redhat.com/downloads/
|
||||
4,SUSE\*,Server 12 SP2,45 GB,Shared with #1,Shared with #1,https://www.suse.com/download-linux/
|
||||
5,Ubuntu\*,16.04.02 LTS Desktop,40 GB,Shared with #1,Shared with #1,https://www.ubuntu.com/download/desktop
|
||||
6,Linux Mint\*,18.1 *Serena* MATE,40 GB,Shared with #1,Shared with #1,https://linuxmint.com/edition.php?id=228
|
||||
|
||||
Table notes:
|
||||
|
||||
.. [#] Configure the partition size as desired.
|
||||
|
||||
|
||||
.. [#] To save disk space, share a single swap partition between
|
||||
multiple Linux installations. Swap size was determined using these
|
||||
`recommended swap partition sizes`_.
|
||||
|
||||
|
||||
.. [#] The EFI partition holds the kernel and boot information for |CL| and
|
||||
other operating systems. The partition size is dependent on the number
|
||||
of operating systems to be installed. In general, allocate about 100 MB
|
||||
per operating system. For this tutorial, we used 1 GB.
|
||||
|
||||
|
||||
|
||||
.. _multi-boot-detail-proc:
|
||||
|
||||
Detailed procedures
|
||||
*******************
|
||||
|
||||
* :ref:`multi-boot-cl` (below)
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
multi-boot-win
|
||||
multi-boot-rhel
|
||||
multi-boot-sles
|
||||
multi-boot-ubuntu
|
||||
multi-boot-mint
|
||||
multi-boot-restore-bl
|
||||
|
||||
|
||||
.. _multi-boot-cl:
|
||||
|
||||
Install |CL|
|
||||
************
|
||||
|
||||
Navigation tips for text-based installation interfaces:
|
||||
|
||||
* Use the :kbd:`Up Arrow` and :kbd:`Down Arrow` keys to move between
|
||||
the options on the screen.
|
||||
|
||||
* Use the :kbd:`Space` to select or highlight an option.
|
||||
|
||||
* Press :kbd:`Enter` to activate the selected option and to move ahead.
|
||||
|
||||
Installation details
|
||||
====================
|
||||
|
||||
#. Create a bootable USB drive of the |CL| installer using one of the methods
|
||||
below.
|
||||
|
||||
* :ref:`bootable-usb-linux`
|
||||
* :ref:`bootable-usb-mac`
|
||||
* :ref:`bootable-usb-windows`
|
||||
|
||||
#. Start the |CL| installer and follow the prompts.
|
||||
|
||||
#. On the :guilabel:`Choose Installation Type` screen, choose
|
||||
:guilabel:`Manual (Advanced)`, as shown in Figure 1.
|
||||
|
||||
.. figure:: figures/multi-boot-01.png
|
||||
|
||||
Figure 1: |CL| installer: Choose installation type screen.
|
||||
|
||||
#. On the :guilabel:`Choose partitioning method` screen, choose
|
||||
:guilabel:`Manually configure mounts and partitions`, as shown in
|
||||
Figure 2.
|
||||
|
||||
.. figure:: figures/multi-boot-02.png
|
||||
|
||||
Figure 2: |CL|: Choose partitioning method.
|
||||
|
||||
#. Select the drive, in this case :file:`/dev/sda`, and press :kbd:`Enter` to
|
||||
go into the `cgdisk` partitioning tool. See Figure 3.
|
||||
|
||||
.. figure:: figures/multi-boot-03.png
|
||||
|
||||
Figure 3: |CL|: Choose drive to partition.
|
||||
|
||||
#. Create a new root partition.
|
||||
|
||||
#. Select :guilabel:`New`, as shown in Figure 4.
|
||||
|
||||
.. _multi-boot-04:
|
||||
|
||||
.. figure:: figures/multi-boot-04.png
|
||||
|
||||
Figure 4: |CL|: Create new partition.
|
||||
|
||||
#. Accept the default first sector.
|
||||
|
||||
#. Specify the desired size of the partition. For this example, we
|
||||
specified *50 GB*. See Figure 5.
|
||||
|
||||
.. figure:: figures/multi-boot-05.png
|
||||
|
||||
Figure 5: |CL|: New partition size.
|
||||
|
||||
#. Set the partition type to :guilabel:`8300 (Linux filesystem)`, as shown
|
||||
in Figure 6.
|
||||
|
||||
.. figure:: figures/multi-boot-06.png
|
||||
|
||||
Figure 6: |CL|: Set partition type.
|
||||
|
||||
#. Name the partition :file:`CL-root`. This name makes it easier to
|
||||
identify later. See Figure 7.
|
||||
|
||||
.. figure:: figures/multi-boot-07.png
|
||||
|
||||
Figure 7: |CL|: Name partition.
|
||||
|
||||
#. Create a new swap partition as shown in Figure 8.
|
||||
|
||||
.. figure:: figures/multi-boot-08.png
|
||||
|
||||
Figure 8: |CL|: Create swap partition.
|
||||
|
||||
#. Select the *free space* partition located at the bottom of the column.
|
||||
|
||||
#. Select :guilabel:`New`. See :ref:`Figure 4<multi-boot-04>`.
|
||||
|
||||
#. Accept the default first sector.
|
||||
|
||||
#. Specify the desired size of the swap partition. For this example, we
|
||||
used 8 GB. See the `recommended swap partition sizes`_ for guidance.
|
||||
|
||||
#. Set the partition type to :guilabel:`8200 (Linux swap)`.
|
||||
|
||||
#. Name the partition :file:`CL-swap`.
|
||||
|
||||
#. Create a new EFI partition as shown in Figure 9.
|
||||
|
||||
.. figure:: figures/multi-boot-09.png
|
||||
|
||||
Figure 9: |CL|: Create EFI partition.
|
||||
|
||||
#. In the :guilabel:`Partition Type` column, select :guilabel:`free space`
|
||||
located at the bottom of the column.
|
||||
|
||||
#. Select :guilabel:`New`. See :ref:`Figure 4<multi-boot-04>`.
|
||||
|
||||
#. Accept the default first sector.
|
||||
|
||||
#. Specify the desired size of the partition. For this example, we used
|
||||
1024 MB. This partition will hold |CL|, the kernels of the other
|
||||
operating systems, and their boot information. Its size depends on the
|
||||
number of installed operating systems. In general, allocate about 100 MB
|
||||
per operating system. For this example, we used 1024 MB.
|
||||
|
||||
#. Set the partition type to :guilabel:`ef00 (EFI partition)`.
|
||||
|
||||
#. Name the partition :file:`CL-EFI`.
|
||||
|
||||
#. Select :guilabel:`Write` to apply the new partition table.
|
||||
|
||||
#. Select :guilabel:`Quit` to exit the `cgdisk` tool.
|
||||
|
||||
#. On the :guilabel:`Set mount points` screen, specify the mount points and
|
||||
format settings as shown in Figure 10.
|
||||
|
||||
.. figure:: figures/multi-boot-10.png
|
||||
|
||||
Figure 10: |CL|: Set mount points.
|
||||
|
||||
#. On the :guilabel:`User configuration` screen, select
|
||||
:guilabel:`Create an administrative user`, as shown in Figure 11.
|
||||
|
||||
.. figure:: figures/multi-boot-11.png
|
||||
|
||||
Figure 11: |CL|: User configuration.
|
||||
|
||||
#. Select :guilabel:`Add user to sudoers?`, as shown in Figure 12.
|
||||
|
||||
.. figure:: figures/multi-boot-12.png
|
||||
|
||||
Figure 12: |CL|: Add user as sudoer.
|
||||
|
||||
#. Follow the remaining prompts to complete the installation and finish
|
||||
the out-of-box-experience for |CL|.
|
||||
|
||||
#. Log in.
|
||||
|
||||
#. Add a Systemd-Boot timeout period or Systemd-Boot will not present the
|
||||
boot menu of available OSes to choose from and will always boot |CL|.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo clr-boot-manager set-timeout 20
|
||||
|
||||
sudo clr-boot-manager update
|
||||
|
||||
#. Reboot.
|
||||
|
||||
If you want to install other OSes, refer to :ref:`multi-boot-detail-proc`.
|
||||
|
||||
.. _recommended swap partition sizes:
|
||||
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/ch-swapspace.html
|
||||
@@ -0,0 +1,287 @@
|
||||
.. _nvidia:
|
||||
|
||||
Install NVIDIA Drivers
|
||||
######################
|
||||
|
||||
NVIDIA is a manufacture of graphics processing units (GPU), also known as
|
||||
graphics cards.
|
||||
|
||||
NVIDIA devices on Linux have two popular device driver options: the opensource
|
||||
drivers from the `nouveau project`_ or the proprietary drivers published by
|
||||
NVIDIA. The nouveau drivers are built into the |CL-ATTR| kernel and are loaded
|
||||
automatically at system boot if a compatible card is detected.
|
||||
|
||||
These instructions show how to use the proprietary NVIDIA drivers which
|
||||
require a manual installation.
|
||||
|
||||
.. note::
|
||||
|
||||
Software installed outside of :ref:`swupd <swupd-about>` is not updated with |CL|
|
||||
updates and must be updated and maintained manually.
|
||||
|
||||
|
||||
|
||||
.. contents:: :local:
|
||||
:depth: 2
|
||||
|
||||
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
* A |CL| system with a desktop installed
|
||||
* A NVIDIA device installed
|
||||
|
||||
|
||||
Install DKMS
|
||||
************
|
||||
|
||||
The :ref:`Dynamic Kernel Module System (DKMS)
|
||||
<kernel-modules-dkms>` allows the NVIDIA kernel modules to be automatically
|
||||
integrated when kernel updates occur in |CL|.
|
||||
|
||||
Install the appropriate DKMS bundle using the instructions below:
|
||||
|
||||
.. note::
|
||||
The Long Term Support (LTS) kernel variant is more likely to remain
|
||||
compatible between updates with NVIDIA drivers.
|
||||
|
||||
|
||||
.. include:: ../guides/maintenance/kernel-modules-dkms.rst
|
||||
:start-after: kernel-modules-dkms-install-begin:
|
||||
:end-before: kernel-modules-dkms-install-end:
|
||||
|
||||
|
||||
Download and install the NVIDIA Drivers
|
||||
***************************************
|
||||
|
||||
|
||||
Download the NVIDIA Drivers for Linux
|
||||
=====================================
|
||||
|
||||
#. Identify the model of NVIDIA GPU that is installed.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo lshw -C display
|
||||
|
||||
|
||||
#. Go to the `NVIDIA Driver Downloads website`_ . Search for and download the
|
||||
appropriate driver based on the model of NVIDIA GPU you have with *Linux
|
||||
64-bit* selected as the Operating System .
|
||||
|
||||
|
||||
#. Open a terminal and navigate to where the
|
||||
:file:`NVIDIA-Linux-x86_64-<VERSION>.run` file was saved. In this
|
||||
example, it was saved in the Downloads folder.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd ~/Downloads/
|
||||
|
||||
#. Make the :file:`NVIDIA-Linux-x86_64-<VERSION>.run` file executable.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
chmod +x :file:`NVIDIA-Linux-x86_64-<VERSION>.run`
|
||||
|
||||
|
||||
Disable the nouveau Driver
|
||||
==========================
|
||||
|
||||
The proprietary NVIDIA driver is incompatible with the nouveau driver and
|
||||
needs to be disabled before installation can continue.
|
||||
|
||||
#. Disable the nouveau driver by creating a blacklist file under
|
||||
:file:`/etc/modprobe.d` and reboot.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo mkdir /etc/modprobe.d
|
||||
|
||||
printf "blacklist nouveau \noptions nouveau modeset=0 \n" | sudo tee --append /etc/modprobe.d/disable-nouveau.conf
|
||||
|
||||
|
||||
#. Reboot the system and log back in. It is normal for the graphical
|
||||
environment to not start with no NVIDIA driver loaded.
|
||||
|
||||
|
||||
|
||||
Configure the Dynamic Linker
|
||||
============================
|
||||
|
||||
The NVIDIA installer will be directed to install files under
|
||||
:file:`/opt/nvidia` as much as possible to keep its contents isolated from the
|
||||
rest of the |CL| system files under :file:`/usr`. The dynamic linker will
|
||||
need to be configured to use the NVIDIA-provided libraries.
|
||||
|
||||
|
||||
#. Configure the dynamic linker to look for and cache shared libraries under
|
||||
:file:`/opt/nvidia/lib` and :file:`/opt/nvidia/lib32`.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
echo "include /etc/ld.so.conf.d/*.conf" | sudo tee --append /etc/ld.so.conf
|
||||
|
||||
sudo mkdir /etc/ld.so.conf.d
|
||||
printf "/opt/nvidia/lib \n/opt/nvidia/lib32 \n" | sudo tee --append /etc/ld.so.conf.d/nvidia.conf
|
||||
|
||||
|
||||
|
||||
Install the NVIDIA Drivers
|
||||
==========================
|
||||
|
||||
|
||||
#. A terminal not running on */dev/tty1* is useful to view uninterrupted
|
||||
installation progress. Switch to a secondary virtual terminal by pushing
|
||||
:command:`CTRL + ALT + F2` or remotely login over SSH.
|
||||
|
||||
|
||||
#. Navigate into the directory where the NVIDIA installer was downloaded.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd ~/Downloads/
|
||||
|
||||
|
||||
#. Run the installer with the advanced options below.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo ./NVIDIA-Linux-x86_64-<VERSION>.run \
|
||||
--utility-prefix=/opt/nvidia \
|
||||
--opengl-prefix=/opt/nvidia \
|
||||
--compat32-prefix=/opt/nvidia \
|
||||
--compat32-libdir=lib32 \
|
||||
--x-prefix=/opt/nvidia \
|
||||
--documentation-prefix=/opt/nvidia \
|
||||
--no-precompiled-interface \
|
||||
--no-nvidia-modprobe \
|
||||
--no-distro-scripts \
|
||||
--force-libglx-indirect \
|
||||
--dkms \
|
||||
--silent
|
||||
|
||||
|
||||
#. The graphical interface may automatically start after the NVIDIA driver
|
||||
is loaded. Return to the working terminal and log back in if necessary.
|
||||
|
||||
|
||||
#. Validate the nvidia kernel modules are loaded.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lsmod | grep ^nvidia
|
||||
|
||||
|
||||
#. Run a |CL| system verification to restore files that the NVIDIA installer
|
||||
likely deleted.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd verify --quick --fix --bundles=lib-opengl
|
||||
|
||||
.. note::
|
||||
|
||||
The NVIDIA software places some files under the :file:`/usr` subdirectory
|
||||
which are not managed by |CL| and conflict with the |CL| stateless design.
|
||||
Although a limited version of :command:`swupd verify --fix` is ran above,
|
||||
other uses of the :command:`swupd verify --fix` command should be avoided
|
||||
with the proprietary NVIDIA drivers installed.
|
||||
|
||||
|
||||
|
||||
|
||||
Updating the NVIDIA Drivers
|
||||
***************************
|
||||
|
||||
The proprietary NVIDIA drivers are installed manually outside of :ref:`swupd
|
||||
<swupd-about>` and must be updated manually when needed.
|
||||
|
||||
Updating the NVIDIA drivers follows the same steps as initial installation,
|
||||
however the desktop environment must first be stopped so that the drivers are
|
||||
not in use.
|
||||
|
||||
#. Follow the steps in `Download the NVIDIA Drivers for Linux`_ section to get
|
||||
the latest NVIDIA drivers.
|
||||
|
||||
#. Temporarily set the default boot target to the *multi-user* which is
|
||||
a non-graphical runtime.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl set-default multi-user.target
|
||||
|
||||
|
||||
#. Reboot the system and log back in. It is normal for the graphical
|
||||
environment to not start.
|
||||
|
||||
#. Follow the steps in `Install the NVIDIA Drivers`_ section to update
|
||||
the NVIDIA drivers. This installation will overwrite the previous NVIDIA
|
||||
drivers and files.
|
||||
|
||||
#. Set the default boot target back to the *graphical* target.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl set-default graphical.target
|
||||
|
||||
|
||||
#. Reboot the system and log back in.
|
||||
|
||||
#. Trigger a flatpak update which will download the runtime corresponding
|
||||
with the new NVIDIA drivers for flatpak apps requiring it.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
flatpak update
|
||||
|
||||
|
||||
Uninstalling the NVIDIA Drivers
|
||||
*******************************
|
||||
|
||||
The NVIDIA drivers and associated software can be uninstalled and nouveau
|
||||
driver restored by:
|
||||
|
||||
#. Remove the previously created file :file:`/etc/modprobe.d` that
|
||||
prevents nouveau from loading.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo rm /etc/modprobe.d/disable-nouveau.conf
|
||||
|
||||
|
||||
#. Run the :command:`sudo /opt/nvidia/bin/nvidia-uninstall`
|
||||
|
||||
#. Follow the prompts on the screen and reboot the system.
|
||||
|
||||
|
||||
Debugging Installation of NVIDIA Drivers
|
||||
****************************************
|
||||
|
||||
* The NVIDIA driver places installer and uninstaller logs under
|
||||
:file:`/var/log/nvidia-install` and :file:`/var/log/nvidia-uninstall`.
|
||||
|
||||
* :file:`NVIDIA-Linux-x86_64-<VERSION>.run --advanced-options` shows many
|
||||
parameters to control installation behavior.
|
||||
|
||||
* :file:`NVIDIA-Linux-x86_64-<VERSION>.run --extract-only` extracts
|
||||
installation files into a directory named
|
||||
:file:`NVIDIA-Linux-x86_64-<VERSION>`.
|
||||
|
||||
|
||||
|
||||
Additional resources
|
||||
********************
|
||||
|
||||
* `Why aren't the NVIDIA Linux drivers open source? <https://nvidia.custhelp.com/app/answers/detail/a_id/1849/kw/Linux>`_
|
||||
|
||||
* `Where can I get support for NVIDIA Linux drivers? <https://nvidia.custhelp.com/app/answers/detail/a_id/44/kw/linux>`_
|
||||
|
||||
* `NVIDIA Accelerated Linux Graphics Driver Installation Guides <https://download.nvidia.com/XFree86/Linux-x86_64/>`_
|
||||
|
||||
.. _`nouveau project`: https://nouveau.freedesktop.org/wiki/
|
||||
|
||||
.. _`NVIDIA Driver Downloads website`: https://www.nvidia.com/download/index.aspx
|
||||
|
||||
|
||||
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 105 KiB |
|
After Width: | Height: | Size: 42 KiB |
|
After Width: | Height: | Size: 28 KiB |
@@ -0,0 +1,80 @@
|
||||
.. _smb-desktop:
|
||||
|
||||
Connect to Windows\* shared location from |CL-ATTR| desktop
|
||||
###########################################################
|
||||
|
||||
This tutorial explains how to access an existing shared drive on
|
||||
Windows\* via Server Message Block (SMB) / Common Internet File System (CIFS)
|
||||
from the |CL| GNOME desktop. CIFS filesystem is generally used to access
|
||||
shared storage locations, or share files.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
* You have already `set up a shared location on Windows`_
|
||||
|
||||
Connect to Windows shared location with Nautilus
|
||||
************************************************
|
||||
|
||||
#. From the desktop, select :guilabel:`Files` from the application menu.
|
||||
|
||||
.. note::
|
||||
|
||||
GNOME Files is also known as `Nautilus`.
|
||||
|
||||
#. In :guilabel:`Files`, select :guilabel:`Other Locations`.
|
||||
|
||||
.. figure:: figures/smb-desktop-1.png
|
||||
:scale: 100%
|
||||
:alt: Files, Other Locations
|
||||
|
||||
Figure 1: Files, Other Locations
|
||||
|
||||
#. In the lower taskbar, beside :guilabel:`Connect to Server`,
|
||||
enter the file-sharing address using the Windows sharing schema:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
smb://servername/Share
|
||||
|
||||
.. figure:: figures/smb-desktop-2.png
|
||||
:scale: 100%
|
||||
:alt: Connect to Server
|
||||
|
||||
Figure 2: Connect to Server
|
||||
|
||||
#. Optional: If there are issues with DNS, you can use an IP address in
|
||||
place of the `servername` above. You must still specify the share.
|
||||
|
||||
#. Optional: On the Windows machine, in a CLI, retrieve the IP address by
|
||||
entering the command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ifconfig
|
||||
|
||||
.. note::
|
||||
|
||||
If using the IP address, assure that it is accessible and secure.
|
||||
|
||||
#. Select :guilabel:`Connect`.
|
||||
|
||||
#. The server will request authentication, as shown in Figure 3.
|
||||
|
||||
.. figure:: figures/smb-desktop-3.png
|
||||
:scale: 100%
|
||||
:alt: Authentication
|
||||
|
||||
Figure 3: Authentication
|
||||
|
||||
#. Log in with the same Windows system credentials for which you granted
|
||||
access to the share.
|
||||
|
||||
#. Select the appropriate checkbox to save your credentials. Consider
|
||||
carefully the security risks as a result of your selection.
|
||||
|
||||
|
||||
.. _set up a shared location on Windows: https://www.howtogeek.com/176471/how-to-share-files-between-windows-and-linux/
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
.. _clear-samba-share-to-Windows:
|
||||
|
||||
Enable simple file sharing with a Windows\* machine using Samba\*
|
||||
#################################################################
|
||||
|
||||
This tutorial describes how to enable simple file sharing from a system
|
||||
running |CL-ATTR| to a Windows machine using Samba. For more advanced sharing,
|
||||
refer to the `Samba guide`_.
|
||||
|
||||
Prerequisites
|
||||
*************
|
||||
|
||||
This tutorial assumes you have installed |CL| on your host system. For
|
||||
detailed instructions, follow the steps in :ref:`bare-metal-install-desktop`.
|
||||
|
||||
Before you install any new packages, update |CL| with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo swupd update
|
||||
|
||||
|
||||
Set up file sharing
|
||||
*******************
|
||||
|
||||
#. Log in and get root privileges.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo -s
|
||||
|
||||
#. Add the storage-utils bundle, which includes the Samba binaries.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
swupd bundle-add storage-utils
|
||||
|
||||
.. note::
|
||||
|
||||
The os-clr-on-clr bundle also includes the Samba binaries.
|
||||
|
||||
#. Create a configuration file called :file:`/etc/samba/smb.conf`. In this
|
||||
example, `[Downloads]` enables a folder share with a specific user.
|
||||
`[Documents]` enables a folder share with any user. The example assumes that a
|
||||
user account `clearlinuxuser` already exists.
|
||||
|
||||
If `valid users` is not specified, then anyone with a user account on the
|
||||
machine and with their Samba password already set can access the folder.
|
||||
However, the account is only able to access files and folders for which
|
||||
they have appropriate permissions.
|
||||
|
||||
Use :command:`chown` or :command:`chmod` to change either the owner of the
|
||||
file or the permissions to allow other users to access the file.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
[Global]
|
||||
map to guest = bad user
|
||||
|
||||
[Downloads]
|
||||
path=/home/clearlinuxuser/Downloads
|
||||
read only = no
|
||||
guest ok = no
|
||||
valid users = clearlinuxuser
|
||||
|
||||
[Documents]
|
||||
path=/home/clearlinuxuser/Documents
|
||||
read only = no
|
||||
browsable = yes
|
||||
guest ok = yes
|
||||
|
||||
#. Enable the Samba daemon to start every time.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
systemctl enable smb
|
||||
systemctl start smb
|
||||
|
||||
#. Use :command:`smbpasswd` to add the initial password for the user
|
||||
account to access the share. Be aware that Samba maintains its own list of
|
||||
passwords for user accounts. The Samba password list can be different than
|
||||
the password used to log in.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
smbpasswd -a clearlinuxuser
|
||||
|
||||
Setup is complete and a Windows machine on the same network can access the
|
||||
shares. Windows uses the format :file:`\\\\[server IP or hostname]\\folder` to
|
||||
access shares. Access the shares directly with Windows Explorer or by
|
||||
mapping a network drive.
|
||||
|
||||
Use the IP address of the |CL| machine for an easy access method. If the
|
||||
|CL| machine is behind an Active Directory domain controller or a DNS server,
|
||||
use the hostname of the |CL| machine. For other ways to access shares using a
|
||||
hostname instead of an IP address, see `Chapter 7 of the Samba guide`_.
|
||||
|
||||
|
||||
Map |CL| drive in Windows
|
||||
*************************
|
||||
|
||||
#. Open Windows Explorer and click on the left sidebar on :guilabel:`This PC`
|
||||
to change the options available at the top.
|
||||
|
||||
#. Click the :guilabel:`Map Network Drive` icon and enter the path in the
|
||||
format: :file:`\\\\[server IP or hostname]\\[shared folder]`
|
||||
|
||||
#. Check the box :guilabel:`Connect using different credentials`. Enter
|
||||
the Samba user `clearlinuxuser` and the password created with
|
||||
:command:`smbpasswd`. See Figure 1 for details.
|
||||
|
||||
.. figure:: figures/smb-1.png
|
||||
:scale: 70%
|
||||
:alt: Map a network drive in Windows Explorer
|
||||
|
||||
Figure 1: Map a network drive in Windows Explorer.
|
||||
|
||||
When complete, Windows Explorer displays the share drive as shown in Figure 2.
|
||||
|
||||
.. figure:: figures/smb-2.png
|
||||
:scale: 70%
|
||||
:alt: View a share drive in Windows Explorer
|
||||
|
||||
Figure 2: View a share drive in Windows Explorer.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
.. _Samba guide: https://www.samba.org/samba/docs/using_samba/ch00.html
|
||||
.. _Chapter 7 of the Samba guide: https://www.samba.org/samba/docs/using_samba/ch07.html
|
||||