Holistic IaC

Description:

Am I really owner of my work ? How long can I exploit results of my work ? Such questions always appear in my mind when I do some kind of automation/IaC ... Recently I've finished refactoring my own infrastructure (was forming for many years), it was not only about soft/hard in general, but some exotic changes (several examples for better understanding of variety of such changes):

  1. Regular mouse was replaced with trackball.
  2. Moved to wireless workspace.
  3. SecureCRT replaced with kitty + sshs (zellij on its way too).
  4. Clipboard shared across different hosts with CopyQ.
  5. Wiki completely migrated to Obsidian.

Here some numbers about this refactoring before I go back to IaC meditation:

  1. There were three k8s clusters. Replaced with a single one.
  2. Single Openstack cluster, Rocky. Removed, virtual machines migrated to KubeVirt.
  3. Four physical k8s node (68 cores, 384GB RAM, 7T SSD, 55T HDD).
  4. 100+ k8s applications.
  5. 7k+ CI/CD runs during migration process.
  6. gitstats: 318 days, 262 active days (82.35%), files: 3858, lines: 608188 (876610 added, 268422 removed), commits: 6954 (average 26.5 commits per active day, 21.9 per all days).

Meditation #1: Seed

Deploying from the ground means you must have some basement, like: repo with deploying scenarios, docker images, continuous integration tool, encryption tools etc. I call them - seed objects. They provide a starting point for entire infrastructure. During refactoring I elaborated such seed objects (independent podman containers, they don't depend on each other and can work completely autonomous):

  1. Gitea - a git server that contains code for deploying/automation.
  2. Minio - a S3 server that contains: states, artifacts for building/provisioning other services.
  3. Jenkins - an automation server for CI/CD jobs.
  4. Citizen - self-hosted Terraform registry.
  5. Harbor - a Docker/OCI registry.
  6. EJBCA - a PKI server.
  7. FreeIPA - a directory server, DNS server.
  8. Keycloak - an identity provider.

They all created from tar archives and initialized with bootstrap scripts (remember, there is nothing yet, only you and your desire to start from scratch), which in turn means that you really own your infrastructure. You don't depend on external repos, registries, availability/deprecation etc. Even more, you can provision it further, because all these seed objects (except citizen) have corresponding terraform providers.

Am I happy with such seed objects approach ? No, definitely:

  1. It's just a set of different tools, they all dedicated to specific functionality. There is no such a class of seed software (or I don't know about) that has main purpose - deploy systems from the ground and ensure full ownership.
  2. Yes, I use sops for encrypting secrets inside initial repo, but I need encryption for entire seed workspace (artifacts too, not only secrets). Yes, I can use file/block based encryption, but such solutions are far from flexibility.
  3. I still need to do some bootstrapping (run scripts, see results, switch between nodes etc.). It's error-prone, quality and clearness of such initialization must be high, there should be some changes evolution/back propagation/staging mechanisms etc.

Meditation #2: Tool

There is no such an universal tool that allows to deploy/provision any existing peace of software. Nowadays it's always a set of tools like Ansible and Terraform. Just imagine a service (or a group of services) that can be partly provision with specific Terraform provider, partly with Ansible and the final part with a shell script. Now you have to provision additional service with newer version. To support a newer version you have to upgrade Terraform provider, make some adjustments to Ansible roles and refactor shell script. How to provide compatibility with previous service ?

  1. Upgrade first service to a new version. It might not be an option.
  2. Write compatible code. You might have no time for this and/or it's a very complex task.
  3. Pack new code into docker image and use it in separate CI/CD executors (one for old one, second for a new service). Management burden is heavy and quickly become a mess.

The answer to this problem could be having a versioning set of tools (OCI artifact/module) for specific version of a service. In comparison to dedicated executor - it's more granular and executor operates over steps/artifacts instead of containing the whole set of tools.

Meditation #3: Autowiring

Probably 100% of people who described their infrastructure in code wrote modules/roles. Those modules can be more or less universal, have or not have many options/variables, they can lay along other modules in a single repo or placed into special registries/hubs.
The main problem with modules is repetitive reusing/configuration (if modules are universal enough, of course). You might say "but wait, modules are intended exactly for this!" and I immediately respond.
Yes, modules are good for reusing. But if I have to set parameters for each module that I want to bind to each other again and again - the value of modules disappearing, especially with complex modules which contain enormous amount of variables.
Simple example: I don't want to set up cert-manager and vault again and again, I just want to provide CA cert/key for PKI engine and see how cert-manager creates certificates, this is what I really want.
Sure, I can combine modules into larger one and provide even a higher abstraction, but I lose flexibility of dedicated modules, bring even more parameters complexity and make modules very fragile to any change.

The solution to this might be an autowiring (first time I saw this term during Spring Framework learning) when some of autowiring capabilities can be declared in OCI module metadata.

Meditation #4: Relationship

Everything is in relationship with everything. There are many different objects like variables, scripts, modules etc. and they can be declared/have different forms like truly declared variable in code, string, environment variable, regular file etc.
This is a very serious problem, because changing some objects in hierarchy doesn't automatically trigger all dependant code (especially in right order). Yes, you can set up rules for running code on write to specific paths and use resource locking/allocation mechanism, but it's only tip of an iceberg and such approach doesn't cover all possible situations.

Some kind of annotation mechanism or even a plugin for popular IDE could be a solution for this problem.

Meditation #5: View

I think many people familiar with such tools like Dash for building custom UI for data analytics needs. Probably much more people understand the value of graphical representation in general.

Well, having a framework for building custom UI for automation needs - is a good thing - especially if it doesn't depend on additional services/platforms and can work autonomously alongside your seed objects. It nicely incorporates in owning paradigm - "I wrote this once - it must work always".