From fbfe5f43c6e2aedb506d014aa40761c701d94d8e Mon Sep 17 00:00:00 2001 From: Mary Camp Date: Fri, 4 Oct 2019 13:46:53 -0400 Subject: [PATCH] New doc: create custom application container image (#775) - Add CLI steps to contextualize diff output. - Add Sphinx roles for clarity of command flags. - Apply edits for clarity of sequence and prior to output diffs. --- .../maintenance/container-image-modify.rst | 477 ++++++++++++++++++ ...-container.rst => container-image-new.rst} | 12 +- source/tutorials/docker.rst | 2 +- 3 files changed, 484 insertions(+), 7 deletions(-) create mode 100644 source/guides/maintenance/container-image-modify.rst rename source/guides/maintenance/{custom-clear-container.rst => container-image-new.rst} (96%) diff --git a/source/guides/maintenance/container-image-modify.rst b/source/guides/maintenance/container-image-modify.rst new file mode 100644 index 00000000..eb06301b --- /dev/null +++ b/source/guides/maintenance/container-image-modify.rst @@ -0,0 +1,477 @@ +.. _container-image-modify: + +Modify a |CL|-based container image +################################### + +This guide describes how to customize |CL-ATTR|-based container +`images on Docker Hub`_, which include popular applications and runtimes. + +.. contents:: + :local: + :depth: 2 + +Overview +******** + +Most of these images utilize a Docker build feature called a +`multi-stage build to reduce image size`_ while some use single-stage build Dockerfiles. An official base `clearlinux image on Docker Hub`_ is also available. To create a generic |CL| container image, see +:ref:`our guide `. + +Prerequisites +************* + +* Set up a functional Docker environment as described in :ref:`docker`. + +* Download the |CL| microservice Dockerfile repo with the following + command: + + .. code-block:: bash + + git clone https://github.com/clearlinux/dockerfiles.git + +Examples +******** + +Example 1: Add a bundle +======================= + +In this example, we add :command:`wget` to the **clearlinux/redis** +Dockerfile. + +#. Open a Terminal on |CL| and enter the :file:`dockerfiles/redis` directory. + +#. Enter :command:`swupd search wget` to discover which |CL| bundle includes + the software. + +#. Open a an editor to modify the Dockerfile with + :command:`$EDITOR Dockerfile`. + +#. Append the :command:`wget` bundle to the :command:`--bundles=` parameter + of the :command:`swupd os-install` command. + +#. Run :command:`git diff`. + + The output shows the edits made after adding :command:`wget` in the clearlinux/redis Dockerfile. + + .. code-block:: diff + + diff --git a/redis/Dockerfile b/redis/Dockerfile + index af977cb..b1effab 100644 + --- a/redis/Dockerfile + +++ b/redis/Dockerfile + @@ -15,7 +15,7 @@ RUN source /os-release && \ + mkdir /install_root \ + && swupd os-install -V ${VERSION_ID} \ + --path /install_root --statedir /swupd-state \ + - --bundles=redis-native,findutils,su-exec --no-boot-update + + --bundles=redis-native,findutils,su-exec,wget --no-boot-update + +#. Build the Dockerfile and apply a unique tag name. In this this example, + we use :command:`wget_added` and add proxies. + + .. code-block:: bash + + docker build \ + --no-cache \ + --build-arg http_proxy=$http_proxy \ + --build-arg https_proxy=$https_proxy \ + -t clearlinux/redis:wget_added . + +#. Run the Dockerfile with the `wget --version` command to verify that + :command:`wget` has been added to the image. + + .. code-block:: bash + + docker run clearlinux/redis:wget_added wget --version + +#. The output shows: + + .. code-block:: console + + GNU Wget 1.20.3 built on linux-gnu. + + -cares +digest -gpgme +https +ipv6 -iri +large-file -metalink +nls + -ntlm +opie -psl +ssl/openssl + +Example 2: Change |CL| version (single-stage Dockerfiles) +========================================================= + +This example shows how to rebuild single-stage containers against a specific +OS version, :file:``, by adding a new argument to the Docker build command line. + +#. Open a Terminal on |CL| and enter the + :file:`dockerfiles/machine-learning-ui` directory. + +#. Rebuild the :file:`clearlinux/machine-learning-ui`. Add an extra build + argument :command:`swupd_args="-m "`; in this case, the build version is 31090. + + .. code-block:: bash + :linenos: + :emphasize-lines: 6 + + docker build \ + --no-cache \ + --build-arg http_proxy=$http_proxy \ + --build-arg https_proxy=$https_proxy \ + -t clearlinux/machine-learning-ui:31090 \ + --build-arg swupd_args="-m 31090" . + +#. Run the docker container image: + + .. code-block:: bash + + docker run clearlinux/machine-learning-ui:31090 swupd info + +#. Sample output shows: + + .. code-block:: console + + Distribution: Clear Linux OS + Installed version: 31090 + Version URL: https://cdn.download.clearlinux.org/update + Content URL: https://cdn.download.clearlinux.org/update + +Example 3: Change Clear Linux OS version (multi-stage Dockerfiles) +=================================================================== + +This example shows how to rebuild multi-stage Dockerfiles to use a specific +OS version. + +.. important:: + + All upper layers of multi-stage Dockerfiles inherit the + |CL| version from the base layer. Rebuild the all underlying base layers against the desired OS version. In this example, four base layers must be rebuilt. + +#. Change directory into :file:`dockerfiles`. + +First layer +----------- + +#. Rebuild the first layer, *os-core*. Add an extra build argument + :command:`swupd_args="-m "`; in this case, the build + version is 31090. + + .. code-block:: bash + :linenos: + :emphasize-lines: 6 + + docker build \ + --no-cache \ + --build-arg http_proxy=$http_proxy \ + --build-arg https_proxy=$https_proxy \ + -t clearlinux/os-core:31090 \ + --build-arg swupd_args="-m 31090" os-core/ + +Second layer +------------ + +The next layer is :file:`clearlinux/redis`. + +#. Change the :file:`redis/Dockerfile` to use the desired OS + version; in this case, the build version is 31090. + +#. Run :command:`git diff`. + + The output shows a diff of a modified :file:`clearlinux/redis` Dockerfile that uses the previously built clearlinux/os-core:31090 container image. + + .. code-block:: diff + + diff --git a/redis/Dockerfile b/redis/Dockerfile + index af977cb..d666e6e 100644 + --- a/redis/Dockerfile + +++ b/redis/Dockerfile + @@ -7,7 +7,7 @@ RUN swupd update --no-boot-update $swupd_args + + # Grab os-release info from the minimal base image so + # that the new content matches the exact OS version + -COPY --from=clearlinux/os-core:latest /usr/lib/os-release / + +COPY --from=clearlinux/os-core:31090 /usr/lib/os-release / + + # Install additional content in a target directory + # using the os version from the minimal base + @@ -26,7 +26,7 @@ COPY --from=clearlinux/os-core:latest / / + os_core_install/ + RUN cd / && \ + find os_core_install | sed -e 's/os_core_install/install_root/' | xargs rm -d &> /dev/null || true + + -FROM clearlinux/os-core:latest + +FROM clearlinux/os-core:31090 + +#. Build the Dockerfile. + + .. code-block:: bash + + docker build \ + --no-cache \ + --build-arg http_proxy=$http_proxy \ + --build-arg https_proxy=$https_proxy \ + -t clearlinux/redis:31090 redis/ + +Third layer +----------- + +The next layer is :file:`clearlinux/httpd`. + +#. Change the :file:`httpd/Dockerfile` to use the desired OS + version; in this case, the build version is 31090. + +#. Run :command:`git diff`. + + The output shows a diff of a modified :file:`clearlinux/httpd` Dockerfile that uses the previously built clearlinux/os-core:31090. + + .. code-block:: diff + + diff --git a/httpd/Dockerfile b/httpd/Dockerfile + index 6b2a6bf..9df89e4 100644 + --- a/httpd/Dockerfile + +++ b/httpd/Dockerfile + @@ -7,7 +7,7 @@ RUN swupd update --no-boot-update $swupd_args + + # Grab os-release info from the minimal base image so + # that the new content matches the exact OS version + -COPY --from=clearlinux/os-core:latest /usr/lib/os-release / + +COPY --from=clearlinux/os-core:31090 /usr/lib/os-release / + + # Install additional content in a target directory + # using the os version from the minimal base + @@ -26,7 +26,7 @@ COPY --from=clearlinux/os-core:latest / / + os_core_install/ + RUN cd / && \ + find os_core_install | sed -e 's/os_core_install/install_root/' | xargs rm -d &> /dev/null || true + + -FROM clearlinux/os-core:latest + +FROM clearlinux/os-core:31090 + +#. Build Dockerfile. + + .. code-block:: bash + + docker build \ + --no-cache \ + --build-arg http_proxy=$http_proxy \ + --build-arg https_proxy=$https_proxy \ + -t clearlinux/httpd:31090 httpd/ + +Fourth layer +------------ + +The next layer is :file:`clearlinux/cgit`. + +#. Change the :file:`cgit/Dockerfile` to use the desired OS + version; in this case, the build version is 31090. + +#. Run :command:`git diff`. + + The output shows: + + .. code-block:: diff + + diff --git a/cgit/Dockerfile b/cgit/Dockerfile + index 9a3796d..59260fe 100644 + --- a/cgit/Dockerfile + +++ b/cgit/Dockerfile + @@ -7,7 +7,7 @@ RUN swupd update --no-boot-update $swupd_args + + # Grab os-release info from the minimal base image so + # that the new content matches the exact OS version + -COPY --from=clearlinux/httpd:latest /usr/lib/os-release / + +COPY --from=clearlinux/httpd:31090 /usr/lib/os-release / + + # Install additional content in a target directory + # using the os version from the minimal base + @@ -22,11 +22,11 @@ RUN source /os-release && \ + # file exists on different layers. To minimize docker + # image size, remove the overlapped files before copy. + RUN mkdir /os_core_install + -COPY --from=clearlinux/httpd:latest / /os_core_install/ + +COPY --from=clearlinux/httpd:31090 / /os_core_install/ + RUN cd / && \ + find os_core_install | sed -e 's/os_core_install/install_root/' | xargs rm -d &> /dev/null || true + + -FROM clearlinux/httpd:latest + +FROM clearlinux/httpd:31090 + +#. Build Dockerfile. + + .. code-block:: bash + + docker build \ + --no-cache \ + --build-arg http_proxy=$http_proxy \ + --build-arg https_proxy=$https_proxy \ + -t clearlinux/cgit:31090 cgit/ + +#. Verify the installed OS version by noting the :command:`VERSION_ID` value + in the :file:`/usr/lib/os-release` file in the container filesystem. + + .. code-block:: bash + :linenos: + :emphasize-lines: 6 + + docker run clearlinux/cgit:31090 cat /usr/lib/os-release + NAME="Clear Linux OS" + VERSION=1 + ID=clear-linux-os + ID_LIKE=clear-linux-os + VERSION_ID=31090 + PRETTY_NAME="Clear Linux OS" + ANSI_COLOR="1;35" + HOME_URL="https://clearlinux.org" + SUPPORT_URL="https://clearlinux.org" + BUG_REPORT_URL="mailto:dev@lists.clearlinux.org" + PRIVACY_POLICY_URL=http://www.intel.com/privacy + +Example 4: Customize an application image at runtime +==================================================== + +This section describes how to modify a published |CL| container at runtime. +In this example, we add Tensorflow\* into a :command:`clearlinux/python` +container. This approach can help accelerate the feature development process. + +First console +------------- + +#. Launch the clearlinux/python container. + + .. code-block:: bash + + docker run -it --rm clearlinux/python + Python 3.7.3 (default, Jun 17 2019, 00:47:04) + [GCC 9.1.1 20190616 gcc-9-branch@272336] on linux + Type "help", "copyright", "credits" or "license" for more information. + +#. Try to import Tensorflow inside the container using the command: + :command:`import tensorflow as tf`. The example below shows the expected + error message because the Docker image does not yet include the Tensorflow module. + + .. code-block:: bash + + >>> import tensorflow as tf + Traceback (most recent call last): + File "", line 1, in + ModuleNotFoundError: No module named 'tensorflow' + >>> + +Second console +-------------- + +#. In another console, find the :command:`` of + clearlinux/python launched. This example Container ID is d4ce9d526fa6. + + .. code-block:: bash + + docker ps + +#. The output shows: + + .. code-block:: console + + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + d4ce9d526fa6 clearlinux/python python3 About a minute ago Up About a minute amazing_villani + +#. Connect to the running clearlinux/python container. + + .. code-block:: bash + + docker exec -it d4ce9d526fa6 /usr/bin/bash + root@d4ce9d526fa6/ # + + +#. Use :command:`swupd` to install the machine-learning-tensorflow bundle. + + .. code-block:: bash + + root@d4ce9d526fa6/ # swupd bundle-add machine-learning-tensorflow + Loading required manifests... + Downloading packs (692.32 Mb) for: + - machine-learning-tensorflow + … … + ...100% + Finishing packs extraction... + No extra files need to be downloaded + Installing bundle(s) files... + ...100% + Calling post-update helper scripts. + Successfully installed 1 bundle + +#. After the machine-learning-tensorflow bundle is installed in the + container, in the first console, import Tensorflow, which will be + successful now. You could also save the updated container using the + command :command:`docker commit `. + + .. code-block:: bash + + >>> import tensorflow as tf + >>> tf.__version__ + '1.13.1' + +Third console +------------- + +#. In a third console, save the container with a new tag. Our example uses + the tag `tensorflow_added` to identify our modified container. + + .. code-block:: bash + + docker commit d4ce9d526fa6 clearlinux/python:tensorflow_added + +#. Launch the modified container, and then import Tensorflow with success. + + .. code-block:: bash + + docker run -it clearlinux/python:tensorflow_added + Python 3.7.3 (default, Jun 17 2019, 00:47:04) + [GCC 9.1.1 20190616 gcc-9-branch@272336] on linux + Type "help", "copyright", "credits" or "license" for more information. + + .. code-block:: bash + + >>> import tensorflow as tf + >>> tf.__version__ + '1.13.1' + >>> + +Background +********** + +Multi-stage Dockerfiles contain more than one :command:`FROM` directive. All +of the multi-stage Clear Linux OS Dockerfiles share a common base layer +called :command:`clearlinux/os-core:latest`. All of the higher level layers +inherit the Clear Linux OS version from this base layer. + +For details on how we leveraged multi-stage Docker builds, see the article +`Minimizing Clear Linux OS container sizes`_. + +#. :command:`clearlinux/os-core` is built once per day. It is a container + containing a minimal Linux userspace. + +#. The target container image uses either :command:`clearlinux/os-core` as a + base layer or another container image :command:`clearlinux/` as a base + layer. + +#. Bundle(s) containing the application are downloaded during the first stage + of the build process using :command:`swupd`. + +#. The final container image is a composition of its base layer and the + specific feature layer, via :command:`FROM clearlinux/:latest + , such as: os-core, httpd, and via :command:`COPY --from=builder / + install_root /`. Using this method, the target container images are kept + up to date without file duplication. For application-centric containers, + `os-core-update` is excluded to improve size optimization. + +Related topics +************** + +* :ref:`docker` +* :ref:`container-image-new` + +.. _images on Docker Hub: https://hub.docker.com/u/clearlinux +.. _GitHub\*: https://github.com/clearlinux/dockerfiles +.. _clearlinux image on Docker Hub: https://hub.docker.com/_/clearlinux +.. _clearlinux microservice dockerfile repo: https://github.com/clearlinux/dockerfiles + +.. _multi-stage build: https://docs.docker.com/develop/develop-images/multistage-build/ + +.. _Minimizing Clear Linux OS container sizes: https://clearlinux.org/blogs-news/minimizing-clear-linux-os-container-sizes + +.. _multi-stage build to reduce image size: https://clearlinux.org/blogs-news/minimizing-clear-linux-os-container-sizes \ No newline at end of file diff --git a/source/guides/maintenance/custom-clear-container.rst b/source/guides/maintenance/container-image-new.rst similarity index 96% rename from source/guides/maintenance/custom-clear-container.rst rename to source/guides/maintenance/container-image-new.rst index fa0e51e8..df81c53e 100644 --- a/source/guides/maintenance/custom-clear-container.rst +++ b/source/guides/maintenance/container-image-new.rst @@ -1,11 +1,11 @@ -.. _custom-clear-container: +.. _container-image-new: -Build a custom |CL-ATTR| based Docker container image -####################################################### +Build a new |CL|-based container image +###################################### -This guide contains the steps to build a custom container image. The official -base |CL-ATTR| container image is published on Docker\* Hub and is updated on -a regular basis. +This guide describes how to build a new |CL-ATTR|-based container image. The +official base |CL-ATTR| container image is published on Docker\* Hub and is +updated on a regular basis. .. contents:: :local: diff --git a/source/tutorials/docker.rst b/source/tutorials/docker.rst index d1868234..5dc10dfc 100644 --- a/source/tutorials/docker.rst +++ b/source/tutorials/docker.rst @@ -19,7 +19,7 @@ This tutorial covers: If you want to use |CL| as a Docker container image, refer to the official |CL| container image `published on Docker* Hub `_ - and our guide to :ref:`custom-clear-container`. + and our guide to :ref:`container-image-new`. Prerequisites *************