Evolving golden images over time

[cilki]

Last year I asked someone who I think has a notably customized desktop setup, what they thought about goldboot's central concept (define your system declaratively and build golden images that can be deployed to bare metal).

The reply was that they prefer to setup their system from scratch because it's a forced opportunity to revisit past decisions and therefore make improvements. If all of that process is automated, then you end up with basically the same system over and over again.

Surely others share this perspective too, so we should think about how exactly to facilitate an image evolving over time through this lens. My opinion is that it's sort of like refining a piece of code: you make improvements to a particular section and then actually run it to see if the changes are good (in this analogy, "code changes" are like changing your goldboot config, and "running it" is like building a new image and trying it out).

To distill it down into a potentially simpler question: imagine your system is already defined as a goldboot config and have a CI pipeline outputting golden images. How do you improve that config over time?

[theAkito]

it's sort of like refining a piece of code: you make improvements to a particular section and then actually run it to see if the changes are good (in this analogy, "code changes" are like changing your goldboot config, and "running it" is like building a new image and trying it out).

I think, that is a natural step on the path to crafting a golden image. Revisions, etc. are very helpful, especially if you, for example, changed to a different method, strategy or approach in solving the same problem. For example, there might be a software solution to your problem, that has become deprecated or unmaintained, so you switch to the replacement. It is then nice to be able to look back, to see, when, why & how you changed it.

The "trying out" aspect is way harder with operating system images. Usually, I like a rapid develop, then trial & error approach, when developing software, because it is a convenient & relatively easy way of achieving your goal, without overthinking (& -engineering) your solution, especially if you make tons of little apps for little problems, as I do in my free time.
With operating systems it is not that simple, because there are in most cases so many problems to solve. "Trying out" every feature you need, can take days, if you do not have a huge, pre-configured test suite, specifically configured for this particular use-case.

My personal experience with having a bit of pre-configured Linux systems started out with plebby Bash scripts, simply installing stuff & configuring some basic configuration files, automatically, on first run, after installation.
I tried different approaches & none of them were even close to what'd be needed to achieve a fully automatic installation process for a complex machine, especially if it's not a one-purpose server, but your personal dev machine.

For my Raspberry Pi servers, I use DietPi. It offers some kind of configuration, though it's not really declarative & is based on ancient UNIX/Linux style, which is basically anti-human. Still - way better, than having custom made pleb Bash scripts running.

I also work a lot with Borg. It's so versatile, it can be even used to make configurations happen, quickly.

The Linux distribution, which currently is closest to the ideal of having a completely declaratively configured operating system is, of course, NixOS. However, it has lots of issues, especially if you rely on compatibility & workarounds for special cases. It also takes a lot of work sometimes, to even achieve simple things.

This is where Goldboot offers a new level to declarative operating system configuration. It is another meta-level up, theoretically supporting any operating system. This is what makes this solution so strong.

Based on my experience, which I have summarised in that little diversion, I will use that Goldboot image & always, instantly pick up & record improvements, which come to my mind. If my image is pretty new & is still in the development process, I will constantly take notes of ideas, that come up, while using it. I will probably also build it quite frequently, to get a state of the OS, which I can use daily, without noticing too huge mistakes.
Once the state is reached, which allows me to use it as a daily driver, I will keep picking up small issues & take notes of them. Then, once they pile up, I will improve the configuration accordingly & build a new image.
Repeating that over a long period of time, I will reach a satisfaction rate of 99%, at some point. This will probably be as good as it gets & the image may then be considered "golden".

So, essentially, the image configuration evolving process is majorly made out of a huge amount of incremental, mostly little, changes to the configuration. It is a refining process, like polishing a statue or something like that.

The problem with setting up something from scratch to revisit past decisions (in a dry run environment) is time & nerves.
It takes way too much time & effort. Once you have evolved a pretty golden image in your head about your system & are confident in your decisions, then revisiting would be more harmful, than helpful.
The revisiting also is handled better anyway, while using it. If you revisit it passively, i.e. while just looking at the configuration or starting to manually set up the system, you are not really revisiting it. You are just revisiting the blueprint.
To really actively revisit your past decisions, you just have to use the particular feature in your image, that you have chosen to solve a particular problem & question it while using it.
This is how you actually, actively revisit the decision, while actually evaluating the behaviour, rather than just reading the blueprint of the solution.

Therefore, I think, that the approach of refining an image over long periods of time through mostly tiny, incremental changes is the way to go & the most approachable. It's the most natural & best to achieve.

[cilki]

@theAkito Yep, I think we're on the same page on all of this. When I setup a system from scratch, I might do some things better and some things worse than the previous time. The idea here is to "save" the things you've already done right and improve everywhere else.

On testing speed, since these images are built in VMs anyway, it would be easy to add a "try before you buy" feature where you can drop into a VM for testing without having to actually install the image on real hardware first.