Effortless and Robust Self Hosting

In times of geopolitical unrest and increasing internet enshittification, the ability to take ownership of our own data and to not be in a subversive position towards big tech is increasinly valuable. For this reason Defined Once decides to self host as much as it can and in doing so hopes to improve the status quo of self hosting.

Self hosting services (also referred to as home-labbing) often brings the clear benefit in terms of privacy and costs. However, since there is no free lunch in computer science, the maintenance burden is placed on the system administrator to keep the lights on (you!). In practice, this maintenance work either consists of spending time running commands in the terminal or wrangling with the sharp edges of an infrastrcutre-as-code (IaC) framework and often ends up being both. This proposal does not attempt to be another general purpose IaC automation framework, but rather takes an opinionated stance on deployment and then optimizes for developer experience. The main goals of this proposal can be summarized as:

• Each service to be deployed is just a directory containing at minimum a docker-compose file.
• Persistent data of any service is automatically backed up.
• All incoming network data is routed through a reverse proxy allowing the servers to sit behind networks that aren't accessible from the public internet.
• All configuration is tracked in a git repo which functions as the single source of truth from which all machines derive their state from.

From the perspective of a system administrator, when setting up a new service to the cluster, it should be as simple as writing a new docker-compose file and commiting the changes. Furthermore, there should be little to no manual effort in adding a new physical host to the system. Since this is about self hosting fallible physical infrastructure, as much of the initialization process needs to be automated.

Init

The genesis of any self hosting typically begins with some setup work. Steps like setting up a non-root user, doing some security hardening, mounting any physical disks and, of course, installing the necessary dependencies are common place. Automating even this part of the server lifecycle is crucial, since hardware should be treated as a ticking time-bomb waiting to fail. Thus, the initialization of new hardware during critical moments should be as easy and automated as possible.

The Worker

The most simple scenario is a single server hosting a collection of services. The server can be thought of as a "worker" that given some configuration data, runs some services. Configuration data is commited to a remote repository which is actively watched by the worker. The following example directory strucutrally organizes two services under ther called "office". It implicitly states that the documenso and landing-page instance should be up and running. The daemon running on the office server makes sure the two services are up and is able to collect logs from both using docker's existing infrastructure.

lab/
└── office
    ├── config.toml
    ├── documenso
    │   └── docker-compose.yaml
    └── landing-page
        ├── assets
        └── docker-compose.yaml

From a bird's eye perspective, the system is currently just being described with a single directory as shown in the diagram below. So far this is a rather uninteresting setup, because the services aren't available from the public internet, nor are there any backups. The value at this stage is locking the state of the configuration (identifiable by git hash) with the deployed services. Reasoning about the deployed "live" state of the system is built on top of reasoning about the commited files in the repository.

The Proxy

One of the key benefits of selfhosting comes from the fact that you physically own the hardware and data that you run. However, this is often in tension with the desire to expose a service to the public internet, since making a machine directly accessible from a home or office network requires forwarding a port which opens the great firewall into your personal network. A better option is to isolate the machine that needs to be public into a public data center and have it act as a reverse proxy to tunnel traffic directly to the worker nodes that need it.

There are a number of reverse proxies out there, but shout out goes to Pangolin as it packs all of our needed functionality into a single packge. With pangolin there's a worker-side process that facilitates the tunnel between worker and proxy. Pangolin also brings let's encrypt domain setup and even an optional unified authentication page that can gate-keep any service that doesn't pack its own auth. Our configuration repository remains mostly the same, we now have the additional option to keep some services behind the private auth. Furthermore, two certificates have automatically been provisioned for documenso.example.com and landing-page.example.com.

lab/
└── office
    ├── config.toml
    ├── private
    │   └── documenso
    │       └── docker-compose.yaml
    └── public
        └── landing-page
            ├── assets
            └── docker-compose.yaml

The Backup

In the worst case, a worker node can go completely offline and lose all of its persistent data. To defend against this case, separating backup data to a different physical machine is the first step in achieving a 3-2-1 backup strategy. In principle, more backup nodes can be added for more redundancy, but this example already sufficiently covers the added complexity for introducing jusst a single backup node. As shown in the diagram below, worker nodes (as well as the proxy) can send daily snapshots of their persistent data to the backup nodes.

The persistent data introduces a new axis of state that coexists alongside the configuration state tracked in git. By using snapshots for the persistent state we can enforce compatibility constraints between the configuration and what is currently stored on disk. For example, let's say a service "Documenso" has a configuration deployed with version 1.2.3. From this moment on, all backed up snapshots are tagged with the hash of that configuration. Then, we update Documenso to version 2.0.0 that includes some breaking changes as well as database migrations and we redeploy with a new hash. Now, it would likely break Documenso if we rolled the backup state back to a version before the 2.0.0 release. Since each snapshot is tagged, the system allows us to only rollback to snapshots that were tagged with a compatible configuration.

Self Healing

Imagine the case where worker is unavailable and the only option is to bring up a fresh replacement machine. Since the configuration and backups are tied to one another, one can arbitrarily deploy the application to a new worker node. In the case that the only worker goes down, a brand new worker can be brought up with no manual intervention. Since the configuration exactly pinpoints what needs to be deployed and how. By default, the latest snapshot is selected to restore data from, but the system admin can choose any compatible past snapshot to instantiate the new worker with. The same "resurection" concept can be applied to the proxy node as well as a backup node given that there at least two active backup nodes each containing full replicas.

Conclusion

This proposal purposefully omits any specific technical details, since the getting the architecture right first is what's important. The minimum viable cluster here runs with 3 distinct separate physical machines to achieve redundancy through isolation, while the number of worker, proxy, or backup nodes could be scaled up.