From 6c3e1964f2dae52b3ee2b5a7557b981cda67d37e Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Tue, 14 Aug 2018 16:24:40 -0700 Subject: [PATCH 01/11] Add document for deploying Clear at scale Inital add for deploying Clear Linux at scale. --- source/clear-linux/guides/deploy-at-scale.rst | 342 ++++++++++++++++++ 1 file changed, 342 insertions(+) create mode 100644 source/clear-linux/guides/deploy-at-scale.rst diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst new file mode 100644 index 00000000..4312a84f --- /dev/null +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -0,0 +1,342 @@ +.. _deploy-at-scale.rst: + +Deploying |CL| at Scale +############################## + +Once you are comfrotable with `Clear Linux concepts`_, your next step +may be needing to understand how to deploy |CL| at scale in your +enoviornment. + +Throughout this document the term *endpoint* will be used to generally refer +to a system targeted for |CL| installation, whether that is a datacenter +system or unit deployed in field. + +.. note:: + + This is not a replacement or blueprint for designing your IT operating + environment. These are simply recommendations that should be implemented + with consideration. + + Your |CL| deployment should complement the + existing environment and available tools. It is assumed foundational core IT + dependencies, such as network, in your environment are in a healthy and scaled + to suit the deployment. + + + +.. contents:: :local: + :depth: 2 + + + + + +Pick a |CL| usage and update strategy +========================================== + +Different business scenarios can call for different deployment methodologies. +|CLOSIA| offers the flexibility to continue consuming upstream |CL| +distribution or the option to fork away from the |CL| distribution and +act as your own OSV. + + +Below are overviews of both approaches and some considerations. + + + + +Option #1: Use the |CL| as the upstream origin (mixin) +------------------------------------------------------ + +This approach is the *easier to adopt* by relying on the |CL| upstream for +packaging updates for you to deploy. + + +Custom software or packages that are not available in a preformed bundles can +be added using the `mixin process`_ to form a custom bundle. +If custom bundles are needed, you will solely be responsible for maintaining +the custom bundle(s) and testing between |CL| releases in your environment, +while the rest of the operating system and preformed bundles come from the +|CL| upstream. + + + +#. Ensure |CL| systems are able to be inventoried, managed, and orchestrated + to coordinate software updates. + +#. The Clear upstream is updated extremely fast with autoupdate enabled by + default, however you may wish to act as an intermediary buffer between + the OS releases. If you do decide to act as a gate to |CL| versions, + define a desired release cadence for yourself which is realistic with the + operational expectations of your environment. + +#. Make use of a web caching proxy for |CL| updates for devices connected to + a local area network (LAN), such as a datacenter, to increase the speed + and resiliency of updates from the |CL| update servers. + + Your caching proxy server is just like any other web application; + |WEB-SERVER-SCALE| + + + +Option #2: Create your own Linux distribution (mix) +--------------------------------------------------- + +This approach forks away from the |CL| upstream and has you act as your own +Operating System Vendor (OSV) by leveraging the `mixer process`_ to create +customized images based on |CL|. This is a level of responsibility that +requires having more infrastructure and processes to adopt. In return, this +approach *offers you a high degree of control and customization*. + + +* Development systems which are generating bundles and updates should be + sufficiently performant for the task and seperate from the swupd update + webservers which are serving update content to production machines. + +* swupd update webservers which are serving update content to + production machines (see `mixer process`_ for more information) should be + appropriately scaled and + + Your swupd update server is just like any other web application; + |WEB-SERVER-SCALE| + + + +Adopt an agile methodology +-------------------------- +The cloud, and other scaled deployments, are all about flexibility and speed. +It only makes sense that any |CL| deployment strategy should follow suit. + +Manually rebuilding your own bundles or mix upon every release is not +sustainable at a large scale. A |CL| deployment pipeline should be agile +enough to validate and produce new versions with speed. Whether or not those +updates actually make their way to your production can be seperate +business decision. However this *ability to frequently roll new versions* of +software to your endpoints is a very important prerequisite. + +You own the validation and lifecycle of the OS and should treat it like any +other software development lifecycle. Below are some pointers on this subject: + +* Thoroughly understand the custom software packages, which are not + distributed with |CL|, that you will need to integrate with |CL| and + maintain along with their dependencies. + + +* Setup a path to production for building |CL| based images. At minimum this + should include: + + * A development clr-on-clr environment to test building packages and + bundles for |CL| systems. + + * A pre-production environment to deploy |CL| versions to before + production + + +* Employ a continuous integration and continuous deployment (CI/CD) philosophy + in order to: + + - Automatically pull custom packages as they are updated from their + upstream projects or vendors. + + - Generate |CL| bundles and potentially bootable images with your + customizations, if any. + + - Measure against metrics and indicators which are relevant to your + business (e.g. performance, power, etc) from release to release. + + - Integrate with your organization's governance processes, such as change + control. + + + + + + + +Versioning Infrastructure +------------------------- + +|CL| version numbers have a deep meaning as they version of the whole +infrastructure stack - from the OS components to libraries to applications. + +Good record keeping can be powerful here.You should keep a detailed registry +and history of previously deployed versions and their contents. + +With a simple glance at the |CL| version numbers deployed, you should be able +to determine with confidence if your Clear systems are patched against a +particular security vulnerability or incorporate a critical new feature. + +This practice opens the door to measured tracking and responses for software +fixes. + + + + +Pick an image distribution strategy +=================================== + +Once you have decided on a usage and update strategy, you should understand +*how* the |CL| will be deployed to your endpoints. In a large scale +deployment, interactive installers should be avoided in favor of automated +installations or prebuilt images. + +There are many well-known ways to install an operating system at scale. Each +have their own benefits, and one may lend itself easier in your environment +depending on the resources available to you. + +See the `reference of Clear Linux image types`_ + + +Below are some common ways to install |CL| to systems at scale: + + +Baremetal +---------- + +Preboot Execution Environments (PXE) or other +out-of-band booting options are one way to a |CL| image or installer +offers a way to distribute |CL| to physical baremetal systems on a LAN. + +This option works well if your your customizations are fairly small in size +and infrastructure can be stateless. + +The |CL| `downloads page`_ offers a Live Image and can be deployed as +a PXE boot server if one doesn't already exist in your environment. Also see +`documentation on installing Clear Linux on bare metal systems`_ + + + +Cloud Instances or Virtual Machines +----------------------------------- +Image templates in the form of cloneable disks are an effective way to +distribute |CL| for virtual machine environments, whether on-premise or +hosted by a Cloud Solution Provider (CSP). + +When used in concert with cloud VM migration features, +this can be a good option for allowing your applications a degree of high +availability and workload mobility; VMs can be restarted on a cluster of +hypervisor host or moved between datacenters transparently. + +The |CL| `downloads page`_ offers example prebuilt VM images and is +readily available on popular CSPs. Also see +`documentation on installing Clear Linux in VMs`_. + + + +Containers +---------- + +Containerization platforms allow images to pulled from a +repository and deployed repeatedly as isolated containers. + +Containers with a |CL| image can be a good option to blueprint and ship +your application, including all its dependencies, as an artifact while +allowing you or your customers to dynamically orchestrate and scale +applications. + +|CL| is capable of running a Docker host, has a container image which can +be pulled from DockerHub, or building a customized container. +For more information visit the `containers page`_. + + + + +Considerations with stateless systems +===================================== +An important |CL| concept is statelessness and partitioning of system data +from user data. This concept can change the way you think about an at scale +deployment. + + +Backup strategy +--------------- + +A |CL| system and its infrastructure should be considered commodity and +easily reproducible.Avoid focusing on backing up the operating system itself +or default values. + +Instead, focus on backing up what's important and unique - the application +and data. In other words, only focus on backing up critical areas like +`/home`, `/etc`, and `/var`. + + + + +Meaningful Logging & Telemetry +------------------------------ + +Offload logging and telemetry from endpoints to external servers so it is +persistent and can be accessed on another server when an issue occurs. + + +* Remote syslogging in |CL| is available through the + `systemd journal-remote service`_ + + +* |CL| offers a `native telemetry solution`_ which can be a powerful tool + in a large deployment to quickly crowdsource issues of interest. Take + advantage of this feature with care consideration of who the audience is + for telemetry events, what information is valuable to collect, and expose + events appropriately. + + Your telemetry server is just like any other web application; + |WEB-SERVER-SCALE| + + + + +Orchestration and Configuration Management +------------------------------------------------ + +In cloud environments, where systems can be ephemeral, being able to configure +and maintain generic instances is valuable. + + +|CL| offers an efficient cloud-init style solution, `micro-config-drive`_, +through the *os-cloudguest* bundles which allows you to configure many +common Day 1 operations such as setting hostname, creating users, or placing +SSH keys in an automated way at boot. + + +A configuration management tool is useful for maintaining consistent system +and application-level configuration. Ansible\* is offered through the +*sysadmin-hostmgmt* bundle as a configuration management and automation tool. + + + +Cloud-native applications +----------------------------------- + +An Infrastructure OS can design for good behavior, but it is ultimately up +applications to make agile design choices and flows. Applications deployed +upon |CL| should aim to be host-aware but not depend on any specific host to +run. References should be relative and dynamic when possible. + +The application architecture should incorporate an appropriate tolerance for +infrastructure outages. Don't just keep stateless design as a noted feature. +Continiously test its use; Automate its use by redeploying |CL| and +application on new hosts. This naturally minimizes configuration drift, +challenges your monitoring systems, and business continuity plans. + + + + + +.. _`Clear Linux concepts`: https://clearlinux.org/documentation/clear-linux/concepts +.. _`mixin process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixin +.. _`mixer process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixer +.. _`reference of Clear Linux image types`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/image-types +.. _`documentation on installing Clear Linux on bare metal systems`: https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install +.. _`downloads page`: https://download.clearlinux.org/image/ +.. _`documentation on installing Clear Linux in VMs`: https://clearlinux.org/documentation/clear-linux/get-started/virtual-machine-install +.. _`containers page`: https://clearlinux.org/containers +.. _`systemd journal-remote service`: https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html +.. _`native telemetry solution`: https://clearlinux.org/features/telemetry +.. _`micro-config-drive`: https://github.com/clearlinux/micro-config-drive + +.. |WEB-SERVER-SCALE| replace:: + There are many well-known ways to achieve a scaleable and resilient web + servers for this purpose however, implementation details out of scope from this + document. In general, they should be located close to your endpoints, + high available according to your business needs, and easy to scale with a + loadbalancer when necessary. From ddadce7857cfb0b69e1ad68e4a64276f5d2b2155 Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Tue, 14 Aug 2018 16:31:37 -0700 Subject: [PATCH 02/11] RST syntax fixes and minor intro edit --- source/clear-linux/guides/deploy-at-scale.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index 4312a84f..57868a25 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -1,11 +1,11 @@ .. _deploy-at-scale.rst: Deploying |CL| at Scale -############################## +####################### Once you are comfrotable with `Clear Linux concepts`_, your next step -may be needing to understand how to deploy |CL| at scale in your -enoviornment. +as a system administraotr may be needing to understand how to deploy +|CL| at scale in your enoviornment. Throughout this document the term *endpoint* will be used to generally refer to a system targeted for |CL| installation, whether that is a datacenter @@ -32,7 +32,7 @@ system or unit deployed in field. Pick a |CL| usage and update strategy -========================================== +===================================== Different business scenarios can call for different deployment methodologies. |CLOSIA| offers the flexibility to continue consuming upstream |CL| From f252dce315ad754cf80800c26169571df5f31ec6 Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Wed, 22 Aug 2018 15:46:36 -0700 Subject: [PATCH 03/11] Fix a few typos noticed by @tpleavitt --- source/clear-linux/guides/deploy-at-scale.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index 57868a25..cd6aed8a 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -3,8 +3,8 @@ Deploying |CL| at Scale ####################### -Once you are comfrotable with `Clear Linux concepts`_, your next step -as a system administraotr may be needing to understand how to deploy +Once you are comfortable with `Clear Linux concepts`_, your next step +as a system administrator may be needing to understand how to deploy |CL| at scale in your enoviornment. Throughout this document the term *endpoint* will be used to generally refer @@ -35,7 +35,7 @@ Pick a |CL| usage and update strategy ===================================== Different business scenarios can call for different deployment methodologies. -|CLOSIA| offers the flexibility to continue consuming upstream |CL| +|CLOSIA| offers the flexibility to continue consuming the upstream |CL| distribution or the option to fork away from the |CL| distribution and act as your own OSV. From 0a8310245965611c370688a18c1b63993d8dfb56 Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Wed, 22 Aug 2018 16:00:59 -0700 Subject: [PATCH 04/11] Fix more typos --- source/clear-linux/guides/deploy-at-scale.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index cd6aed8a..a4f7dcf1 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -5,7 +5,7 @@ Deploying |CL| at Scale Once you are comfortable with `Clear Linux concepts`_, your next step as a system administrator may be needing to understand how to deploy -|CL| at scale in your enoviornment. +|CL| at scale in your environment. Throughout this document the term *endpoint* will be used to generally refer to a system targeted for |CL| installation, whether that is a datacenter @@ -90,7 +90,7 @@ approach *offers you a high degree of control and customization*. * Development systems which are generating bundles and updates should be - sufficiently performant for the task and seperate from the swupd update + sufficiently performant for the task and separate from the swupd update webservers which are serving update content to production machines. * swupd update webservers which are serving update content to @@ -110,7 +110,7 @@ It only makes sense that any |CL| deployment strategy should follow suit. Manually rebuilding your own bundles or mix upon every release is not sustainable at a large scale. A |CL| deployment pipeline should be agile enough to validate and produce new versions with speed. Whether or not those -updates actually make their way to your production can be seperate +updates actually make their way to your production can be separate business decision. However this *ability to frequently roll new versions* of software to your endpoints is a very important prerequisite. @@ -197,7 +197,7 @@ Preboot Execution Environments (PXE) or other out-of-band booting options are one way to a |CL| image or installer offers a way to distribute |CL| to physical baremetal systems on a LAN. -This option works well if your your customizations are fairly small in size +This option works well if your customizations are fairly small in size and infrastructure can be stateless. The |CL| `downloads page`_ offers a Live Image and can be deployed as @@ -252,7 +252,7 @@ Backup strategy --------------- A |CL| system and its infrastructure should be considered commodity and -easily reproducible.Avoid focusing on backing up the operating system itself +easily reproducible. Avoid focusing on backing up the operating system itself or default values. Instead, focus on backing up what's important and unique - the application @@ -314,7 +314,7 @@ run. References should be relative and dynamic when possible. The application architecture should incorporate an appropriate tolerance for infrastructure outages. Don't just keep stateless design as a noted feature. -Continiously test its use; Automate its use by redeploying |CL| and +Continuously test its use; Automate its use by redeploying |CL| and application on new hosts. This naturally minimizes configuration drift, challenges your monitoring systems, and business continuity plans. @@ -335,7 +335,7 @@ challenges your monitoring systems, and business continuity plans. .. _`micro-config-drive`: https://github.com/clearlinux/micro-config-drive .. |WEB-SERVER-SCALE| replace:: - There are many well-known ways to achieve a scaleable and resilient web + There are many well-known ways to achieve a scalable and resilient web servers for this purpose however, implementation details out of scope from this document. In general, they should be located close to your endpoints, high available according to your business needs, and easy to scale with a From d4a07f44702b57629aeb010fdd5d64cc87f55761 Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Wed, 22 Aug 2018 16:09:50 -0700 Subject: [PATCH 05/11] Add OSV abbreviation --- source/clear-linux/guides/deploy-at-scale.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index a4f7dcf1..b8c2ecc4 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -37,7 +37,7 @@ Pick a |CL| usage and update strategy Different business scenarios can call for different deployment methodologies. |CLOSIA| offers the flexibility to continue consuming the upstream |CL| distribution or the option to fork away from the |CL| distribution and -act as your own OSV. +act as your own :abbr:`OSV (Operating System Vendor)`. Below are overviews of both approaches and some considerations. @@ -83,10 +83,10 @@ Option #2: Create your own Linux distribution (mix) --------------------------------------------------- This approach forks away from the |CL| upstream and has you act as your own -Operating System Vendor (OSV) by leveraging the `mixer process`_ to create -customized images based on |CL|. This is a level of responsibility that -requires having more infrastructure and processes to adopt. In return, this -approach *offers you a high degree of control and customization*. +:abbr:`OSV (Operating System Vendor)` by leveraging the `mixer process`_ to +create customized images based on |CL|. This is a level of responsibility +that requires having more infrastructure and processes to adopt. In return, this approach +*offers you a high degree of control and customization*. * Development systems which are generating bundles and updates should be From c47f74cac58878bf34beda1af4a5c41654c14613 Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Mon, 27 Aug 2018 13:36:32 -0700 Subject: [PATCH 06/11] Fix grammatical error --- source/clear-linux/guides/deploy-at-scale.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index b8c2ecc4..becd3cd9 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -48,7 +48,7 @@ Below are overviews of both approaches and some considerations. Option #1: Use the |CL| as the upstream origin (mixin) ------------------------------------------------------ -This approach is the *easier to adopt* by relying on the |CL| upstream for +This approach is *easier to adopt* by relying on the |CL| upstream for packaging updates for you to deploy. From e78e3eff1bd5eec02db59e1595fe6bb417d10b9f Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Tue, 28 Aug 2018 15:28:35 -0700 Subject: [PATCH 07/11] Fix some RST syntax --- source/clear-linux/guides/deploy-at-scale.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index becd3cd9..ffe4556b 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -85,8 +85,8 @@ Option #2: Create your own Linux distribution (mix) This approach forks away from the |CL| upstream and has you act as your own :abbr:`OSV (Operating System Vendor)` by leveraging the `mixer process`_ to create customized images based on |CL|. This is a level of responsibility -that requires having more infrastructure and processes to adopt. In return, this approach -*offers you a high degree of control and customization*. +that requires having more infrastructure and processes to adopt. In return, +this approach *offers you a high degree of control and customization*. * Development systems which are generating bundles and updates should be @@ -191,7 +191,7 @@ Below are some common ways to install |CL| to systems at scale: Baremetal ----------- +--------- Preboot Execution Environments (PXE) or other out-of-band booting options are one way to a |CL| image or installer @@ -286,7 +286,7 @@ persistent and can be accessed on another server when an issue occurs. Orchestration and Configuration Management ------------------------------------------------- +------------------------------------------ In cloud environments, where systems can be ephemeral, being able to configure and maintain generic instances is valuable. @@ -305,7 +305,7 @@ and application-level configuration. Ansible\* is offered through the Cloud-native applications ------------------------------------ +------------------------- An Infrastructure OS can design for good behavior, but it is ultimately up applications to make agile design choices and flows. Applications deployed From 58ca30f1bb81023be77aab17298494f248fc13bc Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Wed, 29 Aug 2018 13:21:57 -0700 Subject: [PATCH 08/11] Add doc to guides TOC --- source/clear-linux/guides/guides.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/source/clear-linux/guides/guides.rst b/source/clear-linux/guides/guides.rst index c45065fb..e8f20f8f 100644 --- a/source/clear-linux/guides/guides.rst +++ b/source/clear-linux/guides/guides.rst @@ -11,3 +11,4 @@ after completing the |CL| :ref:`installation `. maintenance/maintenance network/network + deploy-at-scale From febbb7af8791074cd7f43b2806e92f939df88ae9 Mon Sep 17 00:00:00 2001 From: puneetse <22071208+puneetse@users.noreply.github.com> Date: Wed, 29 Aug 2018 16:56:40 -0700 Subject: [PATCH 09/11] Add link to ister and cloud-init guide --- source/clear-linux/guides/deploy-at-scale.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index ffe4556b..498aefc0 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -293,9 +293,11 @@ and maintain generic instances is valuable. |CL| offers an efficient cloud-init style solution, `micro-config-drive`_, -through the *os-cloudguest* bundles which allows you to configure many +through the *os-cloudguest* bundles which allow you to configure many common Day 1 operations such as setting hostname, creating users, or placing -SSH keys in an automated way at boot. +SSH keys in an automated way at boot. For more information on +automating configuration during deployment of |CL| endpoints see +the `documentation on bulk provisioning`_ . A configuration management tool is useful for maintaining consistent system @@ -333,6 +335,7 @@ challenges your monitoring systems, and business continuity plans. .. _`systemd journal-remote service`: https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html .. _`native telemetry solution`: https://clearlinux.org/features/telemetry .. _`micro-config-drive`: https://github.com/clearlinux/micro-config-drive +.. _`documentation on bulk provisioning`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/bulk-provision .. |WEB-SERVER-SCALE| replace:: There are many well-known ways to achieve a scalable and resilient web From 878af25d9e340263f3aa45069a319b01adcb6fa1 Mon Sep 17 00:00:00 2001 From: Kevin Putnam Date: Thu, 30 Aug 2018 14:27:09 -0700 Subject: [PATCH 10/11] Corrected some typos and made some wording adjustments. --- source/clear-linux/guides/deploy-at-scale.rst | 192 ++++++------------ 1 file changed, 67 insertions(+), 125 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index 498aefc0..ae286d8e 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -4,46 +4,35 @@ Deploying |CL| at Scale ####################### Once you are comfortable with `Clear Linux concepts`_, your next step -as a system administrator may be needing to understand how to deploy -|CL| at scale in your environment. +as a system administrator is to understand how to deploy Clear Linux* +at scale in your environment. -Throughout this document the term *endpoint* will be used to generally refer -to a system targeted for |CL| installation, whether that is a datacenter -system or unit deployed in field. +In this document the term *endpoint* refers to a system targeted for +|CL| installation, whether that is a datacenter system or unit deployed in +field. .. note:: - This is not a replacement or blueprint for designing your IT operating - environment. These are simply recommendations that should be implemented - with consideration. + This is not a replacement or blueprint for designing your own IT + operating environment. - Your |CL| deployment should complement the - existing environment and available tools. It is assumed foundational core IT - dependencies, such as network, in your environment are in a healthy and scaled - to suit the deployment. - - + Your |CL| deployment should complement the existing environment and + available tools. It is assumed foundational core IT dependencies of your + environment, such as your network, are healthy and scaled to suit the + deployment. .. contents:: :local: :depth: 2 - - - - Pick a |CL| usage and update strategy ===================================== -Different business scenarios can call for different deployment methodologies. +Different business scenarios call for different deployment methodologies. |CLOSIA| offers the flexibility to continue consuming the upstream |CL| distribution or the option to fork away from the |CL| distribution and act as your own :abbr:`OSV (Operating System Vendor)`. - Below are overviews of both approaches and some considerations. - - - Option #1: Use the |CL| as the upstream origin (mixin) ------------------------------------------------------ @@ -51,44 +40,38 @@ Option #1: Use the |CL| as the upstream origin (mixin) This approach is *easier to adopt* by relying on the |CL| upstream for packaging updates for you to deploy. - -Custom software or packages that are not available in a preformed bundles can -be added using the `mixin process`_ to form a custom bundle. -If custom bundles are needed, you will solely be responsible for maintaining +Custom software or packages that are not available in a preformed bundle +can be added using the `mixin process`_ to form a custom bundle. +If custom bundles are needed, you will be responsible for maintaining the custom bundle(s) and testing between |CL| releases in your environment, while the rest of the operating system and preformed bundles come from the |CL| upstream. - - #. Ensure |CL| systems are able to be inventoried, managed, and orchestrated to coordinate software updates. -#. The Clear upstream is updated extremely fast with autoupdate enabled by - default, however you may wish to act as an intermediary buffer between - the OS releases. If you do decide to act as a gate to |CL| versions, - define a desired release cadence for yourself which is realistic with the - operational expectations of your environment. +#. With autoupdate enabled, |CL| is updated daily, however you may wish to + act as an intermediary buffer between the OS releases. If you do decide + to act as a gate to |CL| versions, define a desired release cadence for + yourself which is realistic with the operational expectations of your + environment. -#. Make use of a web caching proxy for |CL| updates for devices connected to - a local area network (LAN), such as a datacenter, to increase the speed +#. Use a web caching proxy for |CL| updates for devices connected to + a local area network (LAN), like a datacenter, to increase the speed and resiliency of updates from the |CL| update servers. - Your caching proxy server is just like any other web application; + Your caching proxy server is just like any other web application. |WEB-SERVER-SCALE| - - Option #2: Create your own Linux distribution (mix) --------------------------------------------------- This approach forks away from the |CL| upstream and has you act as your own :abbr:`OSV (Operating System Vendor)` by leveraging the `mixer process`_ to create customized images based on |CL|. This is a level of responsibility -that requires having more infrastructure and processes to adopt. In return, +that requires having more infrastructure and processes to adopt. In return, this approach *offers you a high degree of control and customization*. - * Development systems which are generating bundles and updates should be sufficiently performant for the task and separate from the swupd update webservers which are serving update content to production machines. @@ -97,30 +80,26 @@ this approach *offers you a high degree of control and customization*. production machines (see `mixer process`_ for more information) should be appropriately scaled and - Your swupd update server is just like any other web application; - |WEB-SERVER-SCALE| - - + Your swupd update server is just like any other web application. + |WEB-SERVER-SCALE| Adopt an agile methodology -------------------------- The cloud, and other scaled deployments, are all about flexibility and speed. It only makes sense that any |CL| deployment strategy should follow suit. -Manually rebuilding your own bundles or mix upon every release is not +Manually rebuilding your own bundles or mix for every release is not sustainable at a large scale. A |CL| deployment pipeline should be agile enough to validate and produce new versions with speed. Whether or not those updates actually make their way to your production can be separate business decision. However this *ability to frequently roll new versions* of -software to your endpoints is a very important prerequisite. +software to your endpoints is an important prerequisite. You own the validation and lifecycle of the OS and should treat it like any -other software development lifecycle. Below are some pointers on this subject: - -* Thoroughly understand the custom software packages, which are not - distributed with |CL|, that you will need to integrate with |CL| and - maintain along with their dependencies. +other software development lifecycle. Below are some pointers: +* Thoroughly understand the custom software packages that you will need to + integrate with |CL| and maintain along with their dependencies. * Setup a path to production for building |CL| based images. At minimum this should include: @@ -131,9 +110,8 @@ other software development lifecycle. Below are some pointers on this subject: * A pre-production environment to deploy |CL| versions to before production - -* Employ a continuous integration and continuous deployment (CI/CD) philosophy - in order to: +* Employ a continuous integration and continuous deployment (CI/CD) + philosophy in order to: - Automatically pull custom packages as they are updated from their upstream projects or vendors. @@ -144,39 +122,28 @@ other software development lifecycle. Below are some pointers on this subject: - Measure against metrics and indicators which are relevant to your business (e.g. performance, power, etc) from release to release. - - Integrate with your organization's governance processes, such as change + - Integrate with your organization's governance processes, such as change control. - - - - - - Versioning Infrastructure ------------------------- -|CL| version numbers have a deep meaning as they version of the whole -infrastructure stack - from the OS components to libraries to applications. +|CL| version numbers are very important as they apply to the whole +infrastructure stack from OS components to libraries and +applications. -Good record keeping can be powerful here.You should keep a detailed registry +Good record keeping is important, so you should keep a detailed registry and history of previously deployed versions and their contents. -With a simple glance at the |CL| version numbers deployed, you should be able -to determine with confidence if your Clear systems are patched against a +With a glance at the |CL| version numbers deployed, you should be +able to tell if your Clear systems are patched against a particular security vulnerability or incorporate a critical new feature. -This practice opens the door to measured tracking and responses for software -fixes. - - - - Pick an image distribution strategy =================================== Once you have decided on a usage and update strategy, you should understand -*how* the |CL| will be deployed to your endpoints. In a large scale +*how* |CL| will be deployed to your endpoints. In a large scale deployment, interactive installers should be avoided in favor of automated installations or prebuilt images. @@ -186,30 +153,26 @@ depending on the resources available to you. See the `reference of Clear Linux image types`_ - Below are some common ways to install |CL| to systems at scale: - Baremetal --------- Preboot Execution Environments (PXE) or other -out-of-band booting options are one way to a |CL| image or installer -offers a way to distribute |CL| to physical baremetal systems on a LAN. +out-of-band booting options are one way to distribute |CL| +to physical baremetal systems on a LAN. This option works well if your customizations are fairly small in size and infrastructure can be stateless. -The |CL| `downloads page`_ offers a Live Image and can be deployed as +The |CL| `downloads page`_ offers a Live Image that can be deployed as a PXE boot server if one doesn't already exist in your environment. Also see `documentation on installing Clear Linux on bare metal systems`_ - - Cloud Instances or Virtual Machines ----------------------------------- Image templates in the form of cloneable disks are an effective way to -distribute |CL| for virtual machine environments, whether on-premise or +distribute |CL| for virtual machine environments, whether on-premises or hosted by a Cloud Solution Provider (CSP). When used in concert with cloud VM migration features, @@ -221,12 +184,10 @@ The |CL| `downloads page`_ offers example prebuilt VM images and is readily available on popular CSPs. Also see `documentation on installing Clear Linux in VMs`_. - - Containers ---------- -Containerization platforms allow images to pulled from a +Containerization platforms allow images to be pulled from a repository and deployed repeatedly as isolated containers. Containers with a |CL| image can be a good option to blueprint and ship @@ -235,83 +196,68 @@ allowing you or your customers to dynamically orchestrate and scale applications. |CL| is capable of running a Docker host, has a container image which can -be pulled from DockerHub, or building a customized container. +be pulled from DockerHub, or can be built as a customized container. For more information visit the `containers page`_. - - - Considerations with stateless systems ===================================== An important |CL| concept is statelessness and partitioning of system data from user data. This concept can change the way you think about an at scale deployment. - Backup strategy --------------- -A |CL| system and its infrastructure should be considered commodity and -easily reproducible. Avoid focusing on backing up the operating system itself -or default values. +A |CL| system and its infrastructure should be considered a commodity and +be easily reproducible. Avoid focusing on backing up the operating system +itself or default values. Instead, focus on backing up what's important and unique - the application and data. In other words, only focus on backing up critical areas like `/home`, `/etc`, and `/var`. - - - Meaningful Logging & Telemetry ------------------------------ -Offload logging and telemetry from endpoints to external servers so it is +Offload logging and telemetry from endpoints to external servers, so it is persistent and can be accessed on another server when an issue occurs. - * Remote syslogging in |CL| is available through the `systemd journal-remote service`_ - * |CL| offers a `native telemetry solution`_ which can be a powerful tool - in a large deployment to quickly crowdsource issues of interest. Take - advantage of this feature with care consideration of who the audience is - for telemetry events, what information is valuable to collect, and expose - events appropriately. + for a large deployment to quickly crowdsource issues of interest. Take + advantage of this feature with careful consideration of the target audience + and the kind of data that would be valuable, and expose events + appropriately. Your telemetry server is just like any other web application; |WEB-SERVER-SCALE| - - - Orchestration and Configuration Management ------------------------------------------ -In cloud environments, where systems can be ephemeral, being able to configure -and maintain generic instances is valuable. - +In cloud environments, where systems can be ephemeral, being able to +configure and maintain generic instances is valuable. |CL| offers an efficient cloud-init style solution, `micro-config-drive`_, -through the *os-cloudguest* bundles which allow you to configure many -common Day 1 operations such as setting hostname, creating users, or placing +through the *os-cloudguest* bundles which allow you to configure many Day 1 +tasks such as setting hostname, creating users, or placing SSH keys in an automated way at boot. For more information on automating configuration during deployment of |CL| endpoints see the `documentation on bulk provisioning`_ . - A configuration management tool is useful for maintaining consistent system and application-level configuration. Ansible\* is offered through the -*sysadmin-hostmgmt* bundle as a configuration management and automation tool. - - +*sysadmin-hostmgmt* bundle as a configuration management and automation +tool. Cloud-native applications ------------------------- An Infrastructure OS can design for good behavior, but it is ultimately up -applications to make agile design choices and flows. Applications deployed -upon |CL| should aim to be host-aware but not depend on any specific host to +to applications to make agile design choices. Applications deployed +on |CL| should aim to be host-aware but not depend on any specific host to run. References should be relative and dynamic when possible. The application architecture should incorporate an appropriate tolerance for @@ -320,10 +266,6 @@ Continuously test its use; Automate its use by redeploying |CL| and application on new hosts. This naturally minimizes configuration drift, challenges your monitoring systems, and business continuity plans. - - - - .. _`Clear Linux concepts`: https://clearlinux.org/documentation/clear-linux/concepts .. _`mixin process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixin .. _`mixer process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixer @@ -339,7 +281,7 @@ challenges your monitoring systems, and business continuity plans. .. |WEB-SERVER-SCALE| replace:: There are many well-known ways to achieve a scalable and resilient web - servers for this purpose however, implementation details out of scope from this - document. In general, they should be located close to your endpoints, - high available according to your business needs, and easy to scale with a - loadbalancer when necessary. + server for this purpose, however implementation details are not in the + scope of this document. In general, they should be close to your + endpoints, highly available, and easy to scale with a load balancer when + necessary. \ No newline at end of file From fd209affcd54f34575d20aa94cfc856e8d1b78c8 Mon Sep 17 00:00:00 2001 From: Kevin Putnam Date: Thu, 6 Sep 2018 13:47:38 -0700 Subject: [PATCH 11/11] fixed 3rd reference to WEB-SERVER-SCALE and dangling and --- source/clear-linux/guides/deploy-at-scale.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst index ae286d8e..9b819057 100644 --- a/source/clear-linux/guides/deploy-at-scale.rst +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -78,9 +78,9 @@ this approach *offers you a high degree of control and customization*. * swupd update webservers which are serving update content to production machines (see `mixer process`_ for more information) should be - appropriately scaled and - - Your swupd update server is just like any other web application. + appropriately scaled. + + Your swupd update server is just like any other web application. |WEB-SERVER-SCALE| Adopt an agile methodology @@ -231,7 +231,7 @@ persistent and can be accessed on another server when an issue occurs. and the kind of data that would be valuable, and expose events appropriately. - Your telemetry server is just like any other web application; + Your telemetry server is just like any other web application. |WEB-SERVER-SCALE| Orchestration and Configuration Management