5 Myths of Kubernetes

The December 2021 report from the CNCF showed that 5.6 million developers use Kubernetes today. We adopted Kubernetes a little over three years ago and felt this is a good time to reflect on what we thought we were getting into and what we have actually experienced. Here are five areas where the prevailing wisdom did not match our expectations.

1.  Kubernetes Is Hard to Learn

Alvin Toffler, the well-known futurist, made one of the more prescient observations: “The illiterate of the 21st century,” Toffler wrote, “will not be those who cannot read and write, but those who cannot learn, unlearn, and relearn.”

Kubernetes is a different paradigm than the AWS, Google, and Azure Cloud, therefore, getting started with Kubernetes is more about unlearning than learning. Unlike the other platform-as-a-Service (PaaS) offerings by the Big Three cloud providers, Kubernetes abstracts and controls the underlying infrastructure using a declarative model. Kubernetes then ensures that the infrastructure and containers are orchestrated to this declared “desired state.”

Once one grasps this, much like these other cloud providers, one tends to incrementally learn and use the capabilities as needed. For instance, AWS alone has over 250 services and tools and it is doubtful developers have deep expertise in all of those offerings. With Kubernetes, of the 63 resources available to us on our development cluster, our team uses and has deep knowledge of about a third of these. As with any new platform or framework, focus on the big picture and core concepts first and then dive deep depending on the problem you are trying to solve.

In the past, setting up and operating Kubernetes was difficult. Now, there are managed Kubernetes providers for as little as $20/mo. I would advise skipping self-managed Kubernetes altogether when exploring whether Kubernetes is right for you.

2. You Don't Need Kubernetes

Have you ever had the experience of building a product that has a few customers that then hits that magic moment when you achieve product-market fit? This is when you have to rapidly scale from almost nothing to hundreds of thousands or even millions of users.

From experience, the worst time to re-platform or re-architect for scale is when your servers are burning up and you are delivering a less than ideal customer experience. When this happens, the organization’s priorities shift from strategic to tactical and long-term thinking takes a back seat. After all, there may be no long-term if you are churning hard-earned customers and partners because your product is not reliable.

That said, rather than writing about the benefits here, this is an excellent article on the 6 Benefits of Kubernetes. Here, we focus on the when and not the why one should use Kubernetes.

Kubernetes is one of those technologies that is now relatively low-cost to operate when you are small and can grow as your customer base grows. So while it is true that you do not need Kubernetes when your application is simple and the customer load is low, it is far better to have it in place from the get-go than to re-platform when it becomes an absolute necessity.

3. You Need a Heroku for Kubernetes

Building cloud applications that scale requires infrastructure and plumbing that is often not strategic to your business. Examples include databases, queues, auto-scalers, and other architectural components. It’s perfectly reasonable that there are platforms that accelerate various parts of the development lifecycle.

While there are opinionated platforms can help get you up and running with Kubernetes quickly, they often do so by hiding the underlying system and abstracting away many of the core concepts. Every layer you add to your architecture between bare metal and your application eats into your COGs. While it is a good idea to leverage PaaS solutions that are non-strategic to your application, one has to be careful about runtime dependencies that are cheap to get started but difficult or impossible to tune for cost as your customer base grows.

Introducing an easy-to-use layer on top of Kubernetes may mean simply kicking the learning can into the future at which point not only are you facing a re-platforming challenge but an organizational training challenge.

When evaluating technologies that serve to make Kubernetes easy, it’s vital to understand if it is tooling that makes a tedious, repetitive, brittle task easier, faster, and cheaper versus introducing an abstraction that may prove to be a hidden liability in the long run.

4. Docker for Developers, Kubernetes for Ops

During development, the team is most comfortable with tools such as IDEs and debuggers which run locally. The challenge of a complex product is that developers work with a scaled-down or mocked local development environment which does not match the production environment. While local development gives the team a great development experience, it often results in features working in the local environment but not as intended in production. Local development tools that help you mock Kubernetes containers gloss over this issue.

While we understand that it is not possible to achieve 100% production parity, development teams should strive to minimize the difference between local, pre-production, and production environments. As these environments drift apart, there is a real impact on team productivity, software quality, and security.

Mocking the application dependencies does not scale well. According to the CNCF, 80% of companies have more than 50 services running in their Kubernetes cluster. You may have a few services today but once writing services become easy, they tend to multiply. It quickly becomes untenable to run all these locally especially if the application has high compute or data processing requirements.

5. You Have to Run Everything in Kubernetes

Containers are a powerful tool to deliver and scale a unit of work, and they can co-exist as part of a hybrid technology stack. One does not have to view Kubernetes adoption as an all-or-nothing proposition. Architects can use a phased or hybrid approach when transitioning to Kubernetes.

It is perfectly reasonable to have an architecture comprising of legacy monolithic services along with PaaS services and containerized services running inside or outside Kubernetes. You can use Kubernetes for your containerized services and keep everything else you have in place. This can be your steady-state or an intermediate step to migrating to a fully containerized application.

It took us some time to get the balance of in-cluster and out-of-cluster services right so do take some time to experiment with the architecture that is right for your needs.