From bb5aa9d49a90f4ec33b7f0fce7a7a7a127ff121c Mon Sep 17 00:00:00 2001 From: MCamp859 Date: Mon, 12 Mar 2018 13:49:58 -0400 Subject: [PATCH] Edited for grammar, formatting, and text flow. Modified topic title, heading text, and heading levels. Deleted prompt character. Replaced # with sudo. Corrected links to external webpages. Edited text for active voice and consistency between steps. Signed-off-by: MCamp859 --- source/clear-linux/guides/network/dpdk.rst | 320 +++++++++++---------- 1 file changed, 165 insertions(+), 155 deletions(-) diff --git a/source/clear-linux/guides/network/dpdk.rst b/source/clear-linux/guides/network/dpdk.rst index 9c339ced..2f8d4127 100644 --- a/source/clear-linux/guides/network/dpdk.rst +++ b/source/clear-linux/guides/network/dpdk.rst @@ -1,337 +1,347 @@ .. _dpdk: -Send packages between platforms -############################### +Use DPDK to send packets between platforms +########################################## -:abbr:`Data Plane Development Kit (DPDK)` is a set of libraries and drivers -for fast packet processing. This document describes how to run a basic use -case for **l3fwd DPDK example**. The objective is to *send packages between -two platforms* using a traffic generator called :ref:`pktgen `, -where the l3fwd example application will forward those packages, see -:ref:`figure 1 `. +This document describes how to send packets between two platforms in the +simple configuration shown in :ref:`Figure 1 `. The example uses the +:abbr:`Data Plane Development Kit (DPDK)`, which is a set of libraries, +drivers, sample applications, and tools for fast packet processing. .. _f1: .. figure:: ./figures/pktgen_lw3fd.png :align: center - :alt: platform A and B + :alt: Platform A and B - Figure 1: Environment for l3fwd DPDK application. + Figure 1: Environment for l3fwd DPDK application + This example uses the following DPDK components: -**Requirements:** +* pktgen: Traffic generator. See `pktgen documentation`_ for details. +* l3fwd: Layer 3 forwarding example application. See + `l3fwd documentation`_ for details. -* Two platforms using Clear Linux* for Intel® Architecture (recommended - release `13330`_ or higher). -* Both images have the **kernel-native bundle** added. -* Install of **network-basic-dev** bundle: +Prerequisites +************* - .. code-block:: bash +* Two platforms using |CLOSIA| release `13330`_ or higher. +* Both images must include the :file:`kernel-native bundle`. +* Install the :file:`network-basic-dev` bundle with the command: - # swupd bundle-add network-basic-dev + .. code-block:: bash + + sudo swupd bundle-add network-basic-dev + +* Each platform must have at least one :abbr:`NIC (Network Interface Card)`. + Check the `DPDK project`_ for the list of supported `dpdk.org NICs`_. -* The platforms must have two NICs, at least one each. It's very important to - check network card compatibility with the `DPDK project`_. You can do this - on the `dpdk.org NICS`_ site. * Two network cables. -Installing dpdk and build l3fwd example (Platform B) -==================================================== +Install dpdk and build l3fwd example (Platform B) +************************************************* -#. Move to ``l3fwd`` example. +#. Change to the :file:`l3fwd` example directory. .. code-block:: bash - # cd /usr/share/dpdk/examples/l3fwd + sudo cd /usr/share/dpdk/examples/l3fwd -#. Assign ``RTE_SDK var`` to the makefiles path. +#. Assign :envvar:`RTE_SDK` variable to the makefiles path. .. code-block:: bash - # export RTE_SDK=/usr/share/dpdk/ + sudo export RTE_SDK=/usr/share/dpdk/ -#. Assign ``RTE_TARGET var`` the value where the gcc config file is located. +#. Assign :envvar:`RTE_TARGET` variable to the location of the gcc\* config + file. .. code-block:: bash - # export RTE_TARGET=x86_64-native-linuxapp-gcc + sudo export RTE_TARGET=x86_64-native-linuxapp-gcc -#. Build the ``l3fwd`` application, then add the configuration header to - the ``CFLAGS`` var. +#. Build the `l3fwd` application and add the configuration header to + the :makevar:`CFLAGS` variable. .. code-block:: bash - # make CFLAGS+="-include /usr/include/rte_config.h" + sudo make CFLAGS+="-include /usr/include/rte_config.h" +Build pktgen (Platform A) +************************* -.. _sec_pktgen: - -Building Pktgen (Platform A) -============================ - -Since the **pktgen** project is currently not included in Clear Linux OS for -Intel Architecture, you must download it from upstream and build it: - -#. Download the `pktgen tar package`_ 3.1.2 or newer. +#. Download the `pktgen tar package`_ v3.1.2 or newer. #. Decompress packages and move to uncompressed source directory. -#. Assign ``RTE_SDK var`` the path where makefiles are located. +#. Assign :envvar:`RTE_SDK` variable to the path where makefiles are located. .. code-block:: bash - # export RTE_SDK=/usr/share/dpdk/ + sudo export RTE_SDK=/usr/share/dpdk/ -#. Assign ``RTE_TARGET var`` the value where the gcc config file is located. +#. Assign :envvar:`RTE_TARGET` to the location of the gcc config file. .. code-block:: bash - # export RTE_TARGET=x86_64-native-linuxapp-gcc + sudo export RTE_TARGET=x86_64-native-linuxapp-gcc -#. Build pktgen project, and set the ``CONFIG_RTE_BUILD_SHARED_LIB`` variable - with "n". +#. Build the `pktgen` project and set the :makevar:`CONFIG_RTE_BUILD_SHARED_LIB` variable + to "n". .. code-block:: bash - # make CONFIG_RTE_BUILD_SHARED_LIB=n + sudo make CONFIG_RTE_BUILD_SHARED_LIB=n -Binding NICs to DPDK kernel drivers (Platforms A and B) -======================================================= +Bind NICs to DPDK kernel drivers (Platforms A and B) +**************************************************** -The ``l3fwd`` application uses two NICs. DPDK has useful tools for binding +The `l3fwd` application uses two NICs. The DPDK includes tools for binding NICs to DPDK modules to run DPDK applications. -#. Load the dpdk I/O kernel module +#. Load the DPDK I/O kernel module. .. code-block:: bash - # modprobe vfio-pci + sudo modprobe vfio-pci -#. Check the status of your NICs; this will show which network cards are not - busy. When another application is using them, the status shows ``Active``, +#. Check the NIC status to determine which network cards are not + busy. When another application is using them, the status shows `Active`, and those NICs cannot be bound. .. code-block:: bash - # dpdk-devbind --status + sudo dpdk-devbind --status -#. Bind two available NICs. The general syntax for binding is - **dpdk-devbind --bind=vfio-pci **, - and the following is a working example: +#. Bind two available NICs. The general syntax for binding is: + :command:`dpdk-devbind --bind=vfio-pci `. + A working example is shown below: .. code-block:: bash - # dpdk-devbind --bind=vfio-pci 01:00.0 + sudo dpdk-devbind --bind=vfio-pci 01:00.0 -#. Check that your NICs binded correctly by checking the status; ``drv`` should - have ``igb_uio`` value; at this point, the NICs are using the DPDK modules. +#. Check the NIC status to verify that the NICs are bound correctly. If + successful, `drv` displays the value `igb_uio`, which confirms + that the NICs are using the DPDK modules. -Setting hugepages (platforms A and B) -===================================== +Set hugepages (Platforms A and B) +********************************* -Clear Linux OS for Intel Architecture supports ``hugepages`` for the large -memory pool allocation used for packet buffers. +|CLOSIA| supports `hugepages` for the large memory pool allocation used for +packet buffers. -#. Set number of hugepages. +#. Set the number of hugepages. .. code-block:: bash - # echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + sudo echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages #. Allocate pages on NUMA machines. .. code-block:: bash - # echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - # echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages + sudo echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages + sudo echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages #. Make memory available for DPDK. .. code-block:: bash - # mkdir -p /mnt/huge $ mount -t hugetlbfs nodev /mnt/huge + sudo mkdir -p /mnt/huge $ mount -t hugetlbfs nodev /mnt/huge - If you would like to know more about this, refer to the `DPDK guide`_. + For more information, refer to the `DPDK guide`_ System Requirements + section. -Setting a physical environment (Platforms A and B) -================================================== +Set up the physical environment (Platforms A and B) +*************************************************** -To achieve the model proposed in the introduction of this topic, (:ref:`f1`), -we need to connect the first Grantley’s NICs to the second Grantley’s NICs -using the network cables, see :ref:`figure 2`. +Connect the NICs on Platform A to the NICs on Platform B using the network +cables as shown in :ref:`Figure 2`. .. _f2: .. figure:: ./figures/pyshical_net.png - Figure 2: Physical network environment. + Figure 2: Physical network environment -Running l3fwd application (Platform B) -====================================== +Run l3fwd application (Platform B) +********************************** -The ``l3fwd`` application is one of the DPDK examples available when you -install the ``dpdk-dev`` bundle; this application forwards packages from one -NIC to another. +The `l3fwd` application is one of the DPDK examples available when you +install the :file:`dpdk-dev` bundle. `l3fwd` forwards packets from one +NIC to another. For details, refer to the `l3fwd documentation`_. #. Open the l3fwd example directory. .. code-block:: bash - # cd /usr/share/dpdk/examples/l3fwd + sudo cd /usr/share/dpdk/examples/l3fwd -#. **This step is very important.** DPDK needs poll drivers for work; these - poll drivers are shared objects in :file:`/usr/lib64`. DPDK supports some - NICs. The full list available at the `dpdk.org NICS`_ docs. You should know - which kernel module the NIC is using and choose a poll driver according to - your NICs. +#. **This step is very important.** -#. At this point the system must have ``hugepages`` requirements. The NICs - bound and the configuration for running ``pktgen`` depends upon network use - cases and available system resources. Use the ``-d`` flag for setting the - pull driver. For example, if the NICs are using ``e1000`` network driver, - they are going to use ``e1000`` poll driver (``librte_pmd_e1000.so``); it - should be in :file:`/usr/lib64` in Clear Linux OS for Intel Architecture, - and it should be enough to add the name. For example + #. DPDK needs poll mode drivers to operate. + #. Poll mode drivers are shared objects in :file:`/usr/lib64`. + #. See the full list of supported NICs at `dpdk.org NICs`_. + #. You must know which kernel module each NIC is using and choose a poll + mode driver that corresponds to your NICs. + +#. NIC binding and `pktgen` configuration depends upon network use cases and + available system resources. Use the :command:`-d` flag to set the poll mode + driver. + + The following example assumes that the NICs use the `e1000` network driver + and the `e1000` poll mode driver. The :file:`librte_pmd_e1000.so` is + located in :file:`/usr/lib64` in |CL|. .. code-block:: bash - # ./build/l3fwd -c 0x3 -n 2 -d librte_pmd_e1000.so -- -p 0x3 --config="(0,0,0),(1,0,1)" + sudo ./build/l3fwd -c 0x3 -n 2 -d librte_pmd_e1000.so -- -p 0x3 --config="(0,0,0),(1,0,1)" -#. When the application starts to run, it will show information about the - ``l3fwd`` running, so pay attention when the application is Initializing - ports. After port 0 initialization, you'll see a MAC address and the same - for port 1. Save this information for setting configuration to `Pktgen` - project. +#. The `l3fwd` application shows port initialization details at startup. + After port 0 initialization completes, `l3fwd` shows a MAC address and + information for port 1. -Running Pktgen application (Platform A) -=========================================== + Save the MAC address for configuring the `pktgen` project. -The `Pktgen` is network traffic generator. It measures the network packaging -performance in a forwarding use case. +Run pktgen application (Platform A) +*********************************** -#. At this point the system must have ``hugepages`` requirements and the NICs - bound. The configuration for running ``pktgen`` depends upon the network use - case and the available system resources. The following is a basic +`pktgen` is a network traffic generator included in the DPDK. + +#. `pktgen` configuration depends upon the network setup and the + available system resources. The following example shows a basic configuration. .. code-block:: bash - # ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 0xf -n 4 -- -p 0xf -P -m "1.0, 2.1" + sudo ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 0xf -n 4 -- -p 0xf -P -m "1.0, 2.1" #. Enable active colorful output (optional). - .. code-block:: console + .. code-block:: bash Pktgen> theme enable -#. The ``l3fwd`` application showed a MAC address per-port initialized; this - MAC addresses should have been set in the pktgen environment:: +#. Use the MAC addresses shown by the `l3fwd` application during initialization. + The command to set the MAC addresses in `pktgen` has the format: - > set mac + .. code-block:: bash - And a working example: + set mac - .. code-block:: console + Here is a working example: + + .. code-block:: bash Pktgen> set mac 0 00:1E:67:CB:E8:C9 Pktgen> set mac 1 00:1E:67:CB:E8:C9 -#. Start to send packages using the next command: +#. Send packets. - .. code-block:: console + .. code-block:: bash Pktgen> start 0-1 -#. If you followed these steps correctly, you'll see that ``pktgen`` is sending - and receiving packages. For more information, see the `Pktgen - documentation`_. +For more details, see the `pktgen documentation`_. - -Annex A: Using pass-through for running on virtual machines -=========================================================== +Appendix A: Use pass-through for virtual machines +************************************************* This section explains how to set up a virtual environment where virtual -machines control the host's NICs. +machines control the NICs on the host. #. Create a new directory and move to it. -#. Download or create a ``start_qemu.sh`` script for running a kvm virtual +#. Download or create a :file:`start_qemu.sh` script for running a kvm virtual machine: .. code-block:: bash - $ curl -O https://download.clearlinux.org/image/start_qemu.sh + sudo curl -O https://download.clearlinux.org/image/start_qemu.sh -#. Download a bare-metal image of Clear Linux OS for Intel Architecture and - rename it as ``clear.img``. +#. Download a bare-metal image of |CLOSIA| and rename it as :file:`clear.img`. -#. Look for an entry for device and vendor & device ID: +#. Look for an Ethernet\* device entry that contains vendor and device ID: .. code-block:: bash - $ lspci -nn | grep Ethernet + sudo lspci -nn | grep Ethernet - An output example from the last step:: + An example output: + + .. code-block:: console 03:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] - where ``8086:1521`` is ``vendor:device ID`` and ``03:00.0`` is the entry for - device. Make note of this information; it is necessary for unbinding a - host's NICs. + where `03:00.0` is the device entry and `8086:1521` is the `vendor:device + ID`. Record this information, because you need it to unbind the NICs from a + host. -#. Unbind NICs from host to do passthrough with virtual machines. Clear Linux - OS for Intel Architecture currently supports this action. You can use the - following commands:: + +#. Unbind the NICs from the host to do pass-through with virtual machines. |CLOSIA| + supports this action. The commands take the format: + + .. code-block:: bash echo "vendor device_ID" > /sys/bus/pci/drivers/pci-stub/new_id echo "entry for device" > /sys/bus/pci/drivers/igb/unbind echo "entry for device" > /sys/bus/pci/drivers/pci-stub/bind echo "vendor device_ID" > /sys/bus/pci/drivers/pci-stub/remove_id + Here is a working example: + .. code-block:: bash - $ echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/new_id - $ echo "0000:03:00.0" > /sys/bus/pci/drivers/igb/unbind - $ echo "0000:03:00.0" > /sys/bus/pci/drivers/pci-stub/bind - $ echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/remove_id + sudo echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/new_id + sudo echo "0000:03:00.0" > /sys/bus/pci/drivers/igb/unbind + sudo echo "0000:03:00.0" > /sys/bus/pci/drivers/pci-stub/bind + sudo echo "8086 1521" > /sys/bus/pci/drivers/pci-stub/remove_id -#. Assign to the KVM virtual machine (guest) the unbound NICs previously noted. - Modify the ``start_qemu.sh`` script in ``qemu-system-x86_64`` arguments, and - add the lines with the host's NICs information:: +#. Assign the unbound NICs to the KVM virtual machine (guest). + Modify the :file:`start_qemu.sh` script in `qemu-system-x86_64` arguments, and + add the lines with the host's NICs information in the format: - -device pci-assign,host="",id=passnic0,addr=03.0 - -device pci-assign,host="",id=passnic1,addr=04.0 + .. code-block:: bash - A working example: + -device pci-assign,host="",id=passnic0,addr=03.0 + -device pci-assign,host="",id=passnic1,addr=04.0 + + Here is a working example: .. code-block:: bash -device pci-assign,host=03:00.0,id=passnic0,addr=03.0 \ -device pci-assign,host=03:00.3,id=passnic1,addr=04.0 \ -#. If you would like to add more NUMA machines to the virtual machine, you can - add the next line in the Makefile boot target:: +#. Add more NUMA machines to the virtual machine by adding lines to the + Makefile boot target in the format: + + .. code-block:: bash -numa node,mem=,cpus= - As a working example for a virtual machine with 4096 of memory and four CPUs, the configuration - would look like this:: + Here is a working example for a virtual machine with 4096 memory and four + CPUs: + + .. code-block:: bash -numa node,mem=2048,cpus=0-1 \ -numa node,mem=2048,cpus=2-3 \ - This means that each NUMA machine has to use the same quantity of memory. + .. note:: Each NUMA machine must use the same quantity of memory. -#. Finally, run the ``start_qemu.sh`` script. +#. Run the :file:`start_qemu.sh` script. .. _13330: https://download.clearlinux.org/releases/13330/ .. _DPDK project: http://dpdk.org -.. _dpdk.org NICS: http://dpdk.org/doc/nics +.. _dpdk.org NICs: http://dpdk.org/doc/nics .. _pktgen tar package: http://dpdk.org/browse/apps/pktgen-dpdk/refs .. _DPDK guide: http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html -.. _Pktgen documentation: `Pktgen documentation`_ https://media.readthedocs.org/pdf/pktgen/latest/pktgen.pdf +.. _l3fwd documentation: http://dpdk.org/doc/guides/sample_app_ug/l3_forward.html +.. _pktgen documentation: http://pktgen-dpdk.readthedocs.io/en/latest/index.html