Sign in

12 Principles for Building and Managing Kubernetes and Cloud Infrastructure

Building a resilient and reliable Kubernetes cloud infrastructure requires more than getting your clusters up and running with a fancy provisioning tool. Solid infrastructure design is a sequence of architecture decisions and experienced implementation. Luckily, many organizations and experts went through this path and shared their experiences.

I believe that success does not have a single recipe, however, there are patterns and principles that we learned from failures and successes. I summarize a list of these core principles which cloud experts and decision-makers can refer to.

Simplify

For infrastructure creation and management, we do not need to add another layer of complexity as the infrastructure itself is meant to be seamless and transparent to the products. As the primary concern and focus should remain on the product, not the infrastructure.

Here comes the simplification principle; which does not mean applying trivial solutions but simplifying the complex ones. This leads us to decisions such as deploying fewer clusters or avoiding multi-region and multi-cloud architectures; as long as we do not have a solid use case to justify them.

The simplification principle also applies to the infrastructure features and services we deploy to a Kubernetes cluster. It could be very attractive to deploy extra services (gold-plating the cluster) hoping to make it a powerful and feature-rich cluster. On the contrary, this will end up complicating the operations and decreasing the platform reliability.

Cloud Managed

Standardization

Defining your set of standards covers processes for operations runbooks and playbooks, as well as technology standardization such as using containers, Kubernetes, and standard tools across teams.

These tools should have preferred characteristics, such as being open source but battle-tested in production, support and promote infrastructure as code, immutability, cloud-agnostic, and simple to use, and deploy with minimum infrastructure (think Ansible and Terraform).

The same principle applies to the technology stack and tools we choose, as unifying and standardizing the tools and technology stack across the teams is proven to be more efficient than having a set of inhomogeneous tools that end up hard to manage, and even if one of these tools is best for a use case, simplicity benefits will always overcome that.

Immutability

Immutability leads to adopting the mentality of operating Kubernetes clusters as cattle instead of individual pets.

Everything as Code

Automation

Source Of Truth

Design For Availability

Cloud-Agnostic

Business Continuity

However, coping with the increased scaling needs and making it real-time remains a challenge, and with introducing containers to deploy and run apps it becomes easy to deploy and scale them in seconds rather than minutes. This puts pressure on Kubernetes and the underlying infrastructure layers to support such massive scaling capabilities of the containers.

You need to plan the scaling requirements for the future to support business expansion and continuity. Capacity planning questions such as whether to use a single large cluster versus smaller multiple clusters, how to manage the infrastructure cost, what are the best nodes sizes, and what is the most efficient resource utilization. All of these questions are required to be answered before creating the cluster and to remain ongoing during its operations.

Plan For Failures

When designing a Kubernetes cluster, you have to design it to survive outages and failures, usually by adopting high-availability concepts. You also have to intentionally test and mitigate your infrastructure and systems failures. You can do this by using chaos engineering techniques, disaster recovery automation, infrastructure testing, and having complete infrastructure CD and IaC.

Operational Efficiency

Learn about designing and building production-ready infrastructure in my new book about Kubernetes: 292 pages full of best practices, insights, and hands-on to help you successfully build and manage your Kubernetes infrastructure. https://www.amazon.com/dp/1800202458/

Author of Kubernetes in Production Best Practices