GitOps the Hard Way, with Argo CD

Why Servers Should Be Disposable: Entropy, Kernighan's Law, and Immutable Infrastructure

The 2nd law of thermodynamics states that systems naturally tend toward increasing entropy, or disorder, over time. The world of software and infrastructure is no exception. System entropy is a real thing and it's a challenge to manage. So instead of tweaking an accumulation of changes or trying to manage inconsistencies and growing complexity cleverly, Terraform, Kubernetes, Argo CD, and other tools may yield to a disposable approach to ensure that the desired state is reached. This is why we use the term "immutable infrastructure" to describe this approach, where services and deployments are not changed but replaced (in most cases) when a change is needed.

Kernighan's law is a great analogy to this situation. It states that "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

In the same way, managing the accumulation of changes and inconsistencies cleverly is not as smart as it seems. It's the opposite of smartness - a trap that will lead to more challenges and problems in the future. Boring, simple but reliable, predictable, reproducible, and scalable deployments are the best way to manage the unpredictable and complex world of software and infrastructure.

In addition to this, cloud native is an all-inclusive approach and requires a different way of thinking and architecting your systems - a mindset shift is needed. You can't, for example, use Kubernetes without fully embracing the cloud native principles. It's all or nothing. Fully automated provisioning, health checks, circuit breakers, auto-scaling, self-healing, load balancing, progressive delivery, and other required practices should be part of your platform.

These mechanisms are not only for the sake of automation, but they are also for the sake of reliability, scalability, and security. Kubernetes is built on these principles and Argo CD is built for Kubernetes. When you use Argo CD, you are using a tool that is designed to work with a declarative/immutable infrastructure and a cloud native mindset on Kubernetes that enforce these principles. GitOps is the glue that holds everything together and is a perfect fit for Kubernetes.

Back to the questions that the participant asked me before.

Why would I delete a perfectly working server and create a new one just because I want to update a package?

The answer is as simple as this: a server you keep updating is a server nobody can fully describe anymore.

Every in-place change - a patched package, a hotfixed config, a manually installed library - is a change someone made at some point for some reason, and six months later nobody remembers the reason or even that the change happened. Two servers that started identical drift apart. The one that has been up for 400 days is not "battle-tested", it is unauditable. When it finally breaks, you are debugging a state that exists nowhere except on that one machine.