diff --git a/source/clear-linux/guides/deploy-at-scale.rst b/source/clear-linux/guides/deploy-at-scale.rst new file mode 100644 index 00000000..9b819057 --- /dev/null +++ b/source/clear-linux/guides/deploy-at-scale.rst @@ -0,0 +1,287 @@ +.. _deploy-at-scale.rst: + +Deploying |CL| at Scale +####################### + +Once you are comfortable with `Clear Linux concepts`_, your next step +as a system administrator is to understand how to deploy Clear Linux* +at scale in your environment. + +In this document the term *endpoint* refers to a system targeted for +|CL| installation, whether that is a datacenter system or unit deployed in +field. + +.. note:: + + This is not a replacement or blueprint for designing your own IT + operating environment. + + Your |CL| deployment should complement the existing environment and + available tools. It is assumed foundational core IT dependencies of your + environment, such as your network, are healthy and scaled to suit the + deployment. + +.. contents:: :local: + :depth: 2 + +Pick a |CL| usage and update strategy +===================================== + +Different business scenarios call for different deployment methodologies. +|CLOSIA| offers the flexibility to continue consuming the upstream |CL| +distribution or the option to fork away from the |CL| distribution and +act as your own :abbr:`OSV (Operating System Vendor)`. + +Below are overviews of both approaches and some considerations. + +Option #1: Use the |CL| as the upstream origin (mixin) +------------------------------------------------------ + +This approach is *easier to adopt* by relying on the |CL| upstream for +packaging updates for you to deploy. + +Custom software or packages that are not available in a preformed bundle +can be added using the `mixin process`_ to form a custom bundle. +If custom bundles are needed, you will be responsible for maintaining +the custom bundle(s) and testing between |CL| releases in your environment, +while the rest of the operating system and preformed bundles come from the +|CL| upstream. + +#. Ensure |CL| systems are able to be inventoried, managed, and orchestrated + to coordinate software updates. + +#. With autoupdate enabled, |CL| is updated daily, however you may wish to + act as an intermediary buffer between the OS releases. If you do decide + to act as a gate to |CL| versions, define a desired release cadence for + yourself which is realistic with the operational expectations of your + environment. + +#. Use a web caching proxy for |CL| updates for devices connected to + a local area network (LAN), like a datacenter, to increase the speed + and resiliency of updates from the |CL| update servers. + + Your caching proxy server is just like any other web application. + |WEB-SERVER-SCALE| + +Option #2: Create your own Linux distribution (mix) +--------------------------------------------------- + +This approach forks away from the |CL| upstream and has you act as your own +:abbr:`OSV (Operating System Vendor)` by leveraging the `mixer process`_ to +create customized images based on |CL|. This is a level of responsibility +that requires having more infrastructure and processes to adopt. In return, +this approach *offers you a high degree of control and customization*. + +* Development systems which are generating bundles and updates should be + sufficiently performant for the task and separate from the swupd update + webservers which are serving update content to production machines. + +* swupd update webservers which are serving update content to + production machines (see `mixer process`_ for more information) should be + appropriately scaled. + + Your swupd update server is just like any other web application. + |WEB-SERVER-SCALE| + +Adopt an agile methodology +-------------------------- +The cloud, and other scaled deployments, are all about flexibility and speed. +It only makes sense that any |CL| deployment strategy should follow suit. + +Manually rebuilding your own bundles or mix for every release is not +sustainable at a large scale. A |CL| deployment pipeline should be agile +enough to validate and produce new versions with speed. Whether or not those +updates actually make their way to your production can be separate +business decision. However this *ability to frequently roll new versions* of +software to your endpoints is an important prerequisite. + +You own the validation and lifecycle of the OS and should treat it like any +other software development lifecycle. Below are some pointers: + +* Thoroughly understand the custom software packages that you will need to + integrate with |CL| and maintain along with their dependencies. + +* Setup a path to production for building |CL| based images. At minimum this + should include: + + * A development clr-on-clr environment to test building packages and + bundles for |CL| systems. + + * A pre-production environment to deploy |CL| versions to before + production + +* Employ a continuous integration and continuous deployment (CI/CD) + philosophy in order to: + + - Automatically pull custom packages as they are updated from their + upstream projects or vendors. + + - Generate |CL| bundles and potentially bootable images with your + customizations, if any. + + - Measure against metrics and indicators which are relevant to your + business (e.g. performance, power, etc) from release to release. + + - Integrate with your organization's governance processes, such as change + control. + +Versioning Infrastructure +------------------------- + +|CL| version numbers are very important as they apply to the whole +infrastructure stack from OS components to libraries and +applications. + +Good record keeping is important, so you should keep a detailed registry +and history of previously deployed versions and their contents. + +With a glance at the |CL| version numbers deployed, you should be +able to tell if your Clear systems are patched against a +particular security vulnerability or incorporate a critical new feature. + +Pick an image distribution strategy +=================================== + +Once you have decided on a usage and update strategy, you should understand +*how* |CL| will be deployed to your endpoints. In a large scale +deployment, interactive installers should be avoided in favor of automated +installations or prebuilt images. + +There are many well-known ways to install an operating system at scale. Each +have their own benefits, and one may lend itself easier in your environment +depending on the resources available to you. + +See the `reference of Clear Linux image types`_ + +Below are some common ways to install |CL| to systems at scale: + +Baremetal +--------- + +Preboot Execution Environments (PXE) or other +out-of-band booting options are one way to distribute |CL| +to physical baremetal systems on a LAN. + +This option works well if your customizations are fairly small in size +and infrastructure can be stateless. + +The |CL| `downloads page`_ offers a Live Image that can be deployed as +a PXE boot server if one doesn't already exist in your environment. Also see +`documentation on installing Clear Linux on bare metal systems`_ + +Cloud Instances or Virtual Machines +----------------------------------- +Image templates in the form of cloneable disks are an effective way to +distribute |CL| for virtual machine environments, whether on-premises or +hosted by a Cloud Solution Provider (CSP). + +When used in concert with cloud VM migration features, +this can be a good option for allowing your applications a degree of high +availability and workload mobility; VMs can be restarted on a cluster of +hypervisor host or moved between datacenters transparently. + +The |CL| `downloads page`_ offers example prebuilt VM images and is +readily available on popular CSPs. Also see +`documentation on installing Clear Linux in VMs`_. + +Containers +---------- + +Containerization platforms allow images to be pulled from a +repository and deployed repeatedly as isolated containers. + +Containers with a |CL| image can be a good option to blueprint and ship +your application, including all its dependencies, as an artifact while +allowing you or your customers to dynamically orchestrate and scale +applications. + +|CL| is capable of running a Docker host, has a container image which can +be pulled from DockerHub, or can be built as a customized container. +For more information visit the `containers page`_. + +Considerations with stateless systems +===================================== +An important |CL| concept is statelessness and partitioning of system data +from user data. This concept can change the way you think about an at scale +deployment. + +Backup strategy +--------------- + +A |CL| system and its infrastructure should be considered a commodity and +be easily reproducible. Avoid focusing on backing up the operating system +itself or default values. + +Instead, focus on backing up what's important and unique - the application +and data. In other words, only focus on backing up critical areas like +`/home`, `/etc`, and `/var`. + +Meaningful Logging & Telemetry +------------------------------ + +Offload logging and telemetry from endpoints to external servers, so it is +persistent and can be accessed on another server when an issue occurs. + +* Remote syslogging in |CL| is available through the + `systemd journal-remote service`_ + +* |CL| offers a `native telemetry solution`_ which can be a powerful tool + for a large deployment to quickly crowdsource issues of interest. Take + advantage of this feature with careful consideration of the target audience + and the kind of data that would be valuable, and expose events + appropriately. + + Your telemetry server is just like any other web application. + |WEB-SERVER-SCALE| + +Orchestration and Configuration Management +------------------------------------------ + +In cloud environments, where systems can be ephemeral, being able to +configure and maintain generic instances is valuable. + +|CL| offers an efficient cloud-init style solution, `micro-config-drive`_, +through the *os-cloudguest* bundles which allow you to configure many Day 1 +tasks such as setting hostname, creating users, or placing +SSH keys in an automated way at boot. For more information on +automating configuration during deployment of |CL| endpoints see +the `documentation on bulk provisioning`_ . + +A configuration management tool is useful for maintaining consistent system +and application-level configuration. Ansible\* is offered through the +*sysadmin-hostmgmt* bundle as a configuration management and automation +tool. + +Cloud-native applications +------------------------- + +An Infrastructure OS can design for good behavior, but it is ultimately up +to applications to make agile design choices. Applications deployed +on |CL| should aim to be host-aware but not depend on any specific host to +run. References should be relative and dynamic when possible. + +The application architecture should incorporate an appropriate tolerance for +infrastructure outages. Don't just keep stateless design as a noted feature. +Continuously test its use; Automate its use by redeploying |CL| and +application on new hosts. This naturally minimizes configuration drift, +challenges your monitoring systems, and business continuity plans. + +.. _`Clear Linux concepts`: https://clearlinux.org/documentation/clear-linux/concepts +.. _`mixin process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixin +.. _`mixer process`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixer +.. _`reference of Clear Linux image types`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/image-types +.. _`documentation on installing Clear Linux on bare metal systems`: https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install +.. _`downloads page`: https://download.clearlinux.org/image/ +.. _`documentation on installing Clear Linux in VMs`: https://clearlinux.org/documentation/clear-linux/get-started/virtual-machine-install +.. _`containers page`: https://clearlinux.org/containers +.. _`systemd journal-remote service`: https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html +.. _`native telemetry solution`: https://clearlinux.org/features/telemetry +.. _`micro-config-drive`: https://github.com/clearlinux/micro-config-drive +.. _`documentation on bulk provisioning`: https://clearlinux.org/documentation/clear-linux/guides/maintenance/bulk-provision + +.. |WEB-SERVER-SCALE| replace:: + There are many well-known ways to achieve a scalable and resilient web + server for this purpose, however implementation details are not in the + scope of this document. In general, they should be close to your + endpoints, highly available, and easy to scale with a load balancer when + necessary. \ No newline at end of file diff --git a/source/clear-linux/guides/guides.rst b/source/clear-linux/guides/guides.rst index 2be109e1..f2c65575 100644 --- a/source/clear-linux/guides/guides.rst +++ b/source/clear-linux/guides/guides.rst @@ -18,3 +18,4 @@ after completing the |CL| :ref:`installation `. maintenance/maintenance network/network + deploy-at-scale