Saffron is a docker compose based homelab

It is deployable via static files, and managed as a git repository: Public Git Repository

Ever since my last homelab design post, I have been trying to find a workflow that takes better advantage of docker compose based services. After literal months of trying various versions of container management systems, performing unholy command line shenanigans, and bashing my head into my keyboard, I have settled on saffron, a simple, static folder structure with docker compose files & leveraging git SVM, running a modern web interface for easy configuration & modification.

And I am releasing it with all of my preconfigured docker compose services!

Documentation is on the wiki!

Features

• Media libraries for TV, movies, music, ebooks, & audiobooks. (+ subtitles!)
• Easy migration from any existing docker or docker compose deployment.
• Full torrenting suite with VPN integration.
• Automated backup to cloud storage.
• Smart home automation & integration.
• Network speedtests + hardware monitoring.
• Zero DNS or networking configuration outside of docker.

To deploy

Requires: git, docker, docker compose

# Grab saffron
git clone git@github.com:ivylikethevine/saffron.git
cd saffron

# Create file paths with correct permissions & start dockge
./install-saffron.sh

Why Docker? Why Compose?

# This compose file defines netdata, a Web UI for hardware monitoring.
version: "3"
services:
  netdata:
    image: netdata/netdata
    container_name: netdata
    pid: host
    network_mode: host
    restart: unless-stopped
    ports:
      - 19999:19999
    cap_add:
      - SYS_PTRACE
      - SYS_ADMIN
    volumes:
      - /containers/netdata/config:/etc/netdata
      - /containers/netdata/lib:/var/lib/netdata
      - /containers/netdata/cache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /etc/localtime:/etc/localtime:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
    environment:
      - NETDATA_EXTRA_DEB_PACKAGES=lm_sensors
networks: {}
# This is all we need to deploy an entire service

Containers are great for a homelab because, like their physical counterparts, they are repositionable and movable. Servers are a pain because configuration is tedious and complicated, but containers let us use someone else’s configuration.

Compose takes that further, allowing us to statically define services with configurations, easily network them, and allow complex services to easily run without complicated configuration on the host.

Alternatives

There are exisitng solutions for this problemspace, namely portainer and yacht. I’ve used both of these, but my primary issue with them for the simple homelab use-case is that the user does not maintain full control of the docker compose files. These solutions add a level of complexity on top of compose files, allowing (supposedly) one-click templates, complex networking, and many features that are no doubt useful in some scenario.

However, the homelab is a space where we need to maximize our compatibility and searchability. Portainer/yacht exist as supersets of docker compose, so if we can base our system on the lower level technology, we have more compability with the wealth of knowledge already on the internet. The level of tech-saviness required to grab someone else’s compose file is about as simple as copy/pasting an emoji on a laptop.

Docker compose also allows us the ability to reconfigure our homelab dynamically. It is easy to run multiple different types or configurations of containers, test them for a while, then stop & remove any eliminated contenders.

Dockge

In comes Dockge, a very nice tool that allows us to leverage the power of docker compose with very minimal difference from using a command line interface (CLI), which is (let’s be real here) unbearable UI/UX for 99% of people. Instead of running its own templating system on top of docker compose, it is a simple Web UI to edit our compose files, stop, start, and edit networking & environment variables. There are some downsides when compared to editing saffron directly on the host device inside of a code editor (as I do on swervy, my new homelab node), but with some slight workarounds, it is much easier to diagnose issues and find solutions using docker compose & dockge than portainer or yacht in my experience.

Tools We Can Use

Since we’re using docker compose, we can leverage a ton of useful existing tools to allow us to migrate existing container systems to saffron, such as:

• composerize - to turn docker run... into docker compose yaml (though dockge does have an implementation of this in the UI)
• decomposerize - inverse of above
• autocompose - to turn running containers into docker compose yaml

Source Version Management

Additionally, dockge allows us to create an entire git managed repository that houses our compose files, saffron. The repository contains dockge, in addition to preconfigured docker compose files for the services I currently run on my homelab nodes. They can be turned on or off, and the user maintains complete control of their compose files.

Git allows us to track our changes, and restore working versions when things (eventually) go wrong. Our .env files are not stored in git, so we can (mostly) safely keep personal directory pathings, vpn credentials, and other secrets out of SVM/git.

git commit -m "initial working version"
git checkout -b changeMediaServer # trying out new client
git commit -m "new client"
git commit -m "hot fix"
git stash # new client isn't satisfactory, return to previous
git checkout main # back to functional
git commit -m "different change"

	gitGraph
	    commit
	    branch changeMediaServer
	    checkout changeMediaServer
	    commit
	    commit
	    commit
	    checkout main
	    commit
	

Planned Features

saffron itself is a starting point, because it allows for easy development of future concepts. Some of the integrations I plan to work on are as follows:

• avahi - configuring this would allow internal containers to reach external <hostname>.local domains on LAN, which is my preferred method of networking
• traefik - easy docker integration allows for (in-theory) automatic routing of services (easily supported with labels, but conceptually possible as an automatic routing system)

So if you’re setting up a homelab, try Saffron.

Okay so obviously 3d printing is not new. I’ve been aware of it for about 10 years or so, and I remember vividly seeing so many of these plywood boxes in every makerspace & home workshop in the Las Vegas tech scene:

The Problem with the King

3d printing is an incredibly hard problemspace to work in. Consider how often a 2d printer fails to work correctly, and then the idea of adding the third dimension clearly makes everything harder. So, with most new technology, it sucked…

If you wanted to 3d print in 2012 you needed:

• Lots of Money
• Lots of Time
• Lots of Effort
• Lots of Ibuprofen (for 3d printer induced headaches)
• Lots of Patience

And even then, results were not that great. Filament composition was poor, the software & firmware was a mess, and the idea of creating a coherent UI/UX flow was unheard of in the consumer space. So I ignored FDM printing for a while…

Consumerism Succeeds

Until Amazon showed me an Ender 3 v2 Neo for under 300$ shipped to my apartment in 2 days and I just bought it. Conspicuous consurism got me.

Why the Ender 3 v2 Neo? It had a few quality of life improvements over other Ender 3 priced printers:

• Quiet Motherboard, Power Supply, and Stepper Drivers
• PEI-coated, Removable bed
• Auto Bed Leveling
• Metal Bowden Gears

But most of all, it has a massive amount of aftermarket support, both for troubleshooting and upgrading (physical and software).

Technology versus UI/UX

Over the past decade, 3d printing technology and software has evolved massively. The remaining hurdles for normal people using 3d printers all revolve around the user experience flow. Here is the typical flow:

1. Download model
2. Open slicer software
3. Process & slice (converting STL -> gcode, usually)
4. Export gcode to sd card
5. Bring sd card to printer
6. Use a small lcd to select file to print
7. Return in a few hours, usually with zero feedback on print progress/status unless you check the physical machine
8. Pry print off the bed

Compare to a modern 2d printer flow:

1. Download file
2. Print dialog
3. Remote print using CUPS/airprint
4. Pick up print-out

Additionally, 2d printers don’t have to have their output area (i.e. printer tray) before printing again, but 3d printers (except in very rare cases) require human intervention between prints. So if you are researching a 3d printer, I’d advise that to really have a worthwhile tool, there is more than just the printer. The following is my guide to the various levels of rabbithole you can fall down.

What to Fix

The main issues with a stock Ender type systems are simple:

1. Slow - Due to lots of engineering reasons, the physical max print speed of an Ender 3 is lower than other kinds. This is an inherent limit of the system, so we have to make the workflow around it as fast as possible to compensate.
2. Dumb - There is no wifi connectivity, remote monitoring, or any kind of web interface. Only a physical screen and dials. This can be a pro or a con, as it allows you to add your own interface (I use octoprint, more info below). I would prefer my printer be dumb and off of my personal network, since I then have control over the entire monitoring system.
3. Material Choice - Due to the hotend & bowden feed system, certain filament types are not feasible on a stock ender, such as many composites or flexible filaments. This is usually alleviated through large physical modifications to the print system, such as changing major components.

Level 1: Physical Basics

I specifically picked my printer because it came with a PEI magnetic steel bed. Basically, instead of scraping the plastic off a piece of glass, I take off a thin piece of steel, and flex it until the print comes off. The quality of life improvement is massive, but is also allows the use of multiple beds. For the first round of upgrades, I’d recommend the following:

• Extra print beds (allows swapping between prints, reducing downtime) - example
• A physical fire alarm to mount nearby - example
• Some simple printable upgrades
  • Bed locks to keep the bed level over time - example
  • Replacement extrusion caps for if/when you break them - example
  • Another extrusion cap - example
  • Display cover - example
  • Cable clips - example

Level 2: Smarten up that Printer

Now here is where things get interesting! We’re going to add a single board computer running Octoprint. There are a lot of options for an octoprint host, such as

1. Old Laptop/Desktop
2. (Some) Android Phones
3. Single Board Computers (or SBCs)

Hardware Selection

I’d recommend selecting an SBC if possible. While most of the octoprint configuration I’ve found useful so far has been entirely reproducable on the other systems, the easy ability to add sensors & switches (such as auto-on/off) is worth it. I ended up using a Raspberry Pi 3 Model B Plus Rev 1.3 because I had it on hand, but anything newer is also fine.

Webcam selection

On many SBC’s, there is often a CSI camera connector(s). These use this flat flex ribbon cable + Zero Force (ZF) connectors (quick tip: there are many other names ffor these cables/connectors):

While these are useful and plentifulm, I would recommend a USB webcam over these if you don’t already have one or more. It is much easier to disconnect USB than a CSI connector, which can be a hassle when working on the printer.

Putting it Together

For my build, I ended up with the following 3d prints (after many other options as my setup has evolved & changed):

• RPI 3B Case
• RPI Cam Case for MMS
• MMS anchor for Ender aluminum extrusions
• MMS Axis Changer
• Modular Mounting System/MMS (GoPro compatible mounts)

Another classic octoprint gotcha - make sure to tape over the +5v pin on the USB cable from the SBC to the printer. Otherwise weird power things can occur. Kaptan tape or electrical tape are both solid options. See this great explanation.

A good quality power supply is essential! save yourself the headache and buy one that specifically says its rating, such as this. A vast majority of SBC issues I’ve encountered are due to power supply weirdness.

A major feature that I wanted is to turn the entire printer on/off from the WebUI. This is easily done with a relay like this one. I have my SBC on the ‘always on’ outlet, then my printer & lamp on the ’normally off’ outlets. It’s also very easy to create your own header to attach to the SBC GPIO pins, such as this. (No soldering required!)

More SBC Gotcha’s

Now as far as software installation, I’ve tried multiple popular Octoprint install methods, and by far the best option is to just use the RPI imager tool, install Octoprint to a high quality SD card, and go from there. Docker is an option, but for this application, it’s easiest to use bare-metal. For SD cards, make sure to buy something as per this excellent article.

Configuring Octoprint

Now that we have Octoprint, we can configure it. Octoprint, like many pieces of good software, has a robust plugin system enabling most of the fancy functionality we want. The default setup for Octoprint is very basic, so here are two lists of plugins: 1 for any printer, and another that has some plugins specific to the Creality printers.

Download Plugins & Download Creality Temp Fix Plugins

Plus, since we’re running a simple linux based machine, we can use the following scripts to install all the packages we want, then setup NAS backups just like any other machine in my homelab.

sudo apt install -y zip git htop software-properties-common apt-transport-https wget xclip net-tools curl python3 python3-pip nodejs npm ca-certificates gnupg &&
  sudo apt update -y &&
  sudo apt upgrade -y &&
  sudo apt --fix-broken install -fy &&
  sudo apt autoremove -y
  sudo apt install -y cockpit cockpit-pcp nfs-common rsync cron

nas_name=maccy # user configurable
nas_path=/mnt/sandisk # user configurable

nas_hostname="$nas_name.local"
ssh_dir_path=/home/$USER/.ssh/
ssh_key_name=homelab
ssh_key_path="$ssh_dir_path$ssh_key_name"

sudo mkdir -p &&
  ssh-keygen -t rsa -C "homelab auth" -f $ssh_key_path &&
  eval "$(ssh-agent -s)" &&
  ssh-add ~/.ssh/homelab &&
  ssh-copy-id -i $ssh_key_path $USER@$nas_hostname &&
  crontab -l > cron-nas-backup &&
  echo "0 0 * * * rsync --mkpath --delete -ravh /home/$USER/.octoprint/data/backup $USER@$nas_hostname:$nas_path/$hostname" >> cron-nas-backup &&
  crontab cron-nas-backup &&
  rm cron-nas-backup

And with Octoprint configured, we can setup PrusaSlicer with some useful defaults and send our processed g-code directly to Octoprint, giving us fully remote control. I recommend PrusaSlicer because it has a vast array of settings, as well as incredibly good documentation.

And here is how we send files to Octoprint, no more SD cards required.

Level 3: Advanced Modifications

Level 1 & 2 are requirements, as far as I’m concerned. Again, my printer is physically slow, so we have to compensate by making everything else as seamless as possible. The next few changes are more advanced, and not directly necessary at the start.

1. Relocating the power supply to an external location, often a requirement for the next modification.
2. Creating an enclosure to house the printer, such as the LACK enclosure.
3. Adding a dry box to help print hydroscopic filaments, such as this or this.
4. Adding humidity & temperature sensors.
5. AI print failure detection, such as Obico or OctoAnywhere. I plan to cover my deployment of Obico on my personal homelab, as it is fairly easy to setup, but that is another post.

Conclusion

That covers the majority of the trial & error I’ve gone through during the past 6 or so months getting my beloved 3d printer to behave like a tool, not just a fun gimmicky hobby anymore!

P.S. Here is a great guide on how to diagnose easy print issues

Note on Links

None of these links are sponsored in any way, and I selected these arbitrarily based on convenience.

Part of the modus operandi of my work in the homelab space is to re-use old hardware, because:

1. It’s a fun challenge to work with sub-optimal hardware.
2. I’m cheap.

As such, here’s the current information on my homelab hardware, none of which is very specialized or powerful.

2024 Update: Swervy, the Node 202 Build

Role: Stable Saffron Host

• PcPartPicker
• Intel Core i3-12100 - 4 cores + hyperthreading, 3.3-4.3Ghz, Intel UHD 730
• TeamGroup Elite 1 x 16GB DDR4 3200MT/s RAM
• Asrock Z690M-ITX/ax
  • 1x Intel 1GBe
  • 1x Dragon 2.5GBe
• Corsair SF450W PSU *
• Fractal Design Node 202 *
• 1TB Intel 660p M.2 NVME **
• 2TB Crucial P3 M.2 NVME **
• 2TB Crucial SATA SSD **
• 3TB Seagate 7200RPM HDD **
• Geekbench 6 Scores: ~2100 Single Core/~6500 Multi Core

*: Gifted by a friend

**: Repurposed from other computers

Maccy, the 2015 Macbook Pro

Role: Experimental Saffron Host

• Intel Core i5-5257U CPU @ 2.70GHz: 2 cores + hyperthreading, 2.7-3.1Ghz
• 2 x 4GB DDR3 1867MT/s RAM
• 1x Gigabit Ethernet (using 1x Thunderbolt -> 1GbE)
• 2x Thunderbolt 2 Port (1x in use)
• 1x USB 3.0 (5Gbps) (in use with 1TB Sandisk Extreme SSD)
• 128GB SSD
• Geekbench 6 Scores: ~900 Single Core/~1800 Multi Core

Raspi, the Raspberry Pi 3B+ [MIA]

• Raspberry Pi 3 Model B Plus Rev 1.3
• Broadcom BCM2837B0, ARM 4 cores, 1.4GHz
• 1GB LPDDR2 RAM
• 1x Gigabit Ethernet
• 128GB Micro-SD
• Geekbench 4 (not 6) Scores: ~500 Single Core/~1200 Multi Core

Nessie, the 2012 Macbook Pro [Retired]

• Intel Core i5-2430M: 2 cores + hyperthreading, 2.4-3.0Ghz
• 2 x 2GB DDR3 1333MT/s RAM
• 1x Gigabit Ethernet
• 1x Thunderbolt 2 Port
• 2TB Crucial SATA SSD - Upgraded from original 128GB SSD. [Moved to swervy]
• Geekbench 6 Scores: ~500 Single Core/~750 Multi Core

Opie, the Orange Pi 3B [Water Damaged ☹]

• Orange Pi 3B 4GB
• Rockchip RK3566 - ARM 4 cores, 0.4-1.8Ghz
• 4GB LPDDR4 RAM
• 1x Gigabit Ethernet
• 1x USB 3.0 (5Gbps)
• 128GB M.2 2230 SSD
• Geekbench 6 Scores: ~150 Single Core/~500 Multi Core

The Backups Speech

Every discussion about file backups generally revolves around the same paradigm:

You need to have 3 copies of your data: the production data, a local copy, and an off-site backup.

Similarly, when reading about home servers, there are lots of pieces of (good) advice that talk about the importance of redundancy at every level:

• ‘Fast and redundant storage, best results with SSD disks.’ - Proxmox Recommended Hardware
• ‘We do not recommend installing TrueNAS on a single disk or striped pool unless you have a good reason to do so.’ - TrueNas Minimum Hardware Requirements

But the reality of the situation is my laptop has one internal drive and I want more NAS storage as cheap as possible. This led me to research USB external storage.

My Setup

As mentioned in my homelab design post, my NAS is a late 2011 Macbook Pro, and even with my swapped-in SSD, I have only 2TB of storage on a single drive.

But, thats not my full homelab. I have recently configured another laptop as per my previous post for 2 hosts:

1. nessie: a 2011 Macbook Pro with an upgraded 2TB SATA SSD
2. maccy: a 2015 Macbook Pro with a 256GB not-easily-upgradable NVME SSD

2TB is not insignificant, but most NAS solutions put that size to shame. Since we’re using laptops, we don’t have much internal upgradability (if any), so let’s explore USB storage on NAS’s, and see when it makes sense.

Rule #1: Check Your Speed

Most questions online about ‘how do I use a USB drive on my NAS?’ end in the same answer: don’t even bother.

And for a long time, this was a reasonable (if curt) answer because USB was unbearably slow. Here’s a table showing file transfer times on all USB versions, SATA 3, and Gigabit ethernet:

Before USB 3.0, USB storage was too slow for most applications. Even now, 5Gbps is not incredibly fast, but since an internal SATA 3 drive is around 6gbps, it is not as dreadful as it once seemed. Now, most servers don’t have high speed USB, but plenty of laptops do! maccy has USB 3.0, at 5gbps. However, since nessie only has USB 2.0, it is not viable.

So USB 3 is fast enough for me

The other issues with USB revolve around it’s resiliciency. System Admins will never recommend using external storage because it is a terrible idea for any business. God forbid a USB drive with the production database fails, or just gets unplugged when an intern gets caught on a cable!⁴

Critical data does not belong on a USB drive, but none of my data is really that critical. All of my development work is already hosted on github. My files are really just old schoolwork, pictures, and videos that I would like to keep safe, but don’t need 24/7/365 access with hardware redundancy. The best solution for me is just to run a USB drive and also backup remotely.

Rule #2: Don’t Fail to Boot

My homelab posts are the top 10% of the stuff I have tried, and USB drives have been fighting me for a long time. When designing a NAS with USB storage, remember that they will disconnect. It’s a ‘when, not if’ situation. Our implementation has to account for this and mitigate the issues that this can cause.

First off, the USB drive cannot be the boot drive. Booting off of USB is useful sometimes, but when you are already playing in the margins of tech, double dipping is a bad idea. If we allow our server to have a USB drive as a storage drive that is mounted later, we can gracefully deal with disconnections.

fstab, the file that lists how to mount all drives at boot, has a few options we need to enable. By default, most systems will either halt at boot or wait for a long timeout to pass before booting. If our USB storage is gone, we still want to be able to access the server’s web UI!⁵

With the cockpit UI that our system already has, all we really have to do is plug in the drive, format via the UI, and then work on our fstab options. There are many cool file systems, but I prefer to keep everything as simple as possible. One partition on the USB drive, ext4, no encryption. For nomenclature, I prefer for USB drives to be mounted at /mnt/<diskname>.

After basic configuration, our fstab line looks something like this:

UUID=f94b134d-4e14-4c05-8f30-19aaa477acb4 /mnt/sandisk auto defaults 0 0

We’ll add a nofail to this line, making it like this:

UUID=f94b134d-4e14-4c05-8f30-19aaa477acb4 /mnt/sandisk auto defaults,nofail 0 0⁶

This option means that when booting, we will not fail to boot if we are unable to mount our external drive. There are other issues to deal with, but this is the main one that could break our entire deployment.

Rule #3: Offsite Backups

Another data backup quote I see bounce around often is ’local copies are not a replacement for remote backups’, which is true, but is focused again at a business environment.

If a production hard drive dies, you need a copy of those files, right now. Servers typically do this via data redundancy over RAID (or other software solutions) with a drive already connected. Switchover is instant, the database never goes down, the business never loses a client due to a service outtage.

Again, at home, I am the client and my server needs are not critical to my ability to put food on the table. For me, a remote backup is fine. The disaster I am trying to prevent is not ‘what happens if hardware fails during peak business’, but ‘my old laptop drive finally died’ or ‘my apartment flooded’. So, 1 local copy and remote backups are enough for me.

There are a myriad storage providers, and a myriad of backup solutions. Here is what I have found to work for my uses and be cost effective:

Duplicati running inside a Portainer template from this repo - This docker image then mounts our data directory (/mnt/sandisk/), allowing it to access the data directory.

Backblaze B2 for bulk storage. My original pick was AWS, but B2 is easier for deployments that don’t utilize the AWS/AIM/CLI ecosystem.⁷

Configuration tips:

• I prefer to create a separate Application Key for each folder that my NAS uploads to B2. This way, each folder has its own configuration and is slightly more portable. Most of the backup software will upload files to the cloud provider in a binary format (not readable without the backup software itself).

• Make sure each Application Key only has access to a single bucket. Always keep permissions as minimal as possible, its just a good security practice.

• Encrypt your backups. It’s built-in to most backup software, so why risk it?

• Check your backups! Backups that do not work are not backups at all. Restore a file, delete a file, modify a file, download a few gigabytes. Test, test, test!

Rule #4: Fail Gracefully

USB drives are going to fail, so we need to design our system to fail non-destructively. We’ve already started to account for this with our new mount options above, which allows our system to continue operating if the USB drive is not found during boot, but there are a few other considerations:

1. Nothing Critical
   • Our system is designed with the ability for our USB drive to fail, but our software implementation also depends on us continuing this assumption. Like we already keep our OS on the built-in drive, we should keep all of our programs, software, docker images, etc. on our built-in drive.
   • The USB drive should only store data files & folders.
2. Don’t Break Remote
   • If our USB drive is not found, it is possible that some backup software could error out and assume that the files were deleted, thus causing a mess of our backups.
   • Always test your implementation for this oversight. (Even if the backup software handles this situation properly, adding in our docker environment could lead to weird & undocumented behaviour.)
   • I personally tested this on my duplicati/docker instance, and my deployment properly errors and stops the backup process when it cannot mount our data directory.
3. Don’t Break Local
   • Our backup system and our software design account for failure, so our last step is to fail gracefully as a NAS. Just like how we used specific fstab options above, here are the important options for an NFS NAS (USB or not):

maccy.local:/mnt/sandisk   /mnt/maccy/sandisk   nfs rw,auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

If we want to add more USB drives (such as for local backups), we can repeat the above proces, adding any number of /mnt/<driveName>’s as an NFS share via the Cockpit UI!

Rule #5: Don’t talk about USB Drive NAS’s

We have mitigated the downsides to USB drives, increased our homelab server storage and resiliency, and have a backup system for when the USB drives eventually break. Plus, we didn’t have to buy anything new! Our system fails gracefully in all of the situations covered above, and here’s how to fix pretty much every error condition I encountered during testing:

1. Turn off the server.
2. Replug the USB drive.
3. Restart the server.

Being a fault-tolerant, slightly janky system is good for home use, but it is always worth doing it right when actual business is on the line. Please do not apply these to production deployments!

Footnotes

Sustainable Software

Ok, so the title is Software Engineer or Software Developer, but writing code and building software is only part of the solution. The other half (or more) of the job is designing software solutions for humans to interact with. Software only is useful because we humans use it. Without us, the software is just some meaningless electricity in some far away warehouse. UI/UX design is important, but just as critical is developer operations (dev ops).

Dev Ops

• outlines a software development process and an organizational culture shift that speeds the delivery of higher quality software by automating and integrating the efforts of development and IT operations teams – two groups that traditionally practiced separately from each other, or in silos.¹

What is Dev Ops?

Don’t get me wrong, large scale production deployments can have very complex dev ops! However, even for a single developer, the tools and systems required are very easy, often free, and can massively improve productivity.

Here is my list of main dev ops goals. Some may seem obvious, but all of these are drawn from experience starting new projects as well as inheriting many projects of dubious quality.

• Source Version Mangement
  • Rule 1: software changes need to be easily assigned, tracked, and checked
  • Rule 2: environment variables need to be stored safely (important enough to state again)
• Deployments
  • Rule 1: deploying new versions should be automated via CI/CD and require no human intervention
  • Rule 2: staging site to test versions before production deployment
• Onboarding
  • Rule 1: new devs need to be able to download and deploy system within a day²
  • Rule 2: new devs should not be able to break production

The Fallacy of Saving Time

Dev Ops takes time to setup, but it is absolutely worth it. Here’s a story from my days as a contractor. At one company, it took 2 developers at least 4 hours each to deploy our application (at a generous minimum). Here’s how that math works out over just a single year:

That’s over a month of dev hours lost per year! Does any dev want to waste a month every single year running the same deployments? However, this is just the tip of the iceberg. If your system takes a day to patch, what happens when a major exploit is found in production?

This is a single example of poor dev ops, but time is money and similar math applies to dozens of scenarios.

So Let’s Do It Right

i. Source Version Mangement (svm/git)

Talking to my college friends, it seems a lot of CS majors do not learn Source Version Management (or SVM). Basically, we want a history of all of our code changes, and the ability to go back to specific checkpoints and fix things that break. There are a few tools for this, but I use git as the SVM and github.com to hold my repositories (as well as provide other dev ops tools, discussed later).

git is very powerful and very complex, but we only need a few commands to set up a useable CI/CD environment. This graph is a simple version of “git flow”, or a semi-standard way to use github branches.

	gitGraph
	commit
	branch develop
	checkout develop
	branch featureName
	checkout featureName
	commit
	commit
	checkout develop
	branch fixBug
	checkout fixBug
	commit
	checkout develop
	merge featureName
	checkout main
	merge develop
	checkout develop
	merge fixBug
	checkout main
	merge develop
	

main (previously defaulted to master): this is our production code. Never work directly from main. Never interact with main via CLI.

development: this is our staging code. This is where we start new branches

<namedBranch>: these are self contained changes to the project. Here we have featureName and fixBug, but often there are different categories of named branches such as feature/featureName or bug/bugName.

This is how we would setup a repository to follow basic git flow.

git config --global init.defaultBranch "main" # By default, git may still use `master`, so we set default to `main`.
cd projectFolder
git init
git push -u origin main # here we setup the remote copy of our repo
git checkout -b development # -b is only needed when creating a branch
git push
git checkout -b featureName

Now, we can work on featureName without impacting anything else. Additionally, other devs on our team can branch from development and any merge conflicts will be dealt with on the development branch before merging into main.

Once our feature is added:

git add . # the '.' convention will add ALL modified files. Only modify files that you need to. git restore fileName will undo any changes.
git commit -m "Added feature" # -m only allows inline commit messages. A commit is basically a checkpoint we can go back to later.
git push
git checkout develop

Now, we could immediately create a pull request from development into production, but it is preferable to have a staging or development site. A staging site is a live site that usually only devs/testers can access. So let’s deploy a staging site!

ii. Continuous Integration and Continuous Deployment (CI/CD)

My website is made with Hugo, a static site generator written in Go. It has a theme called Puppet, and I have it deployed to a custom domain (ivylikethevine.com) Originally I had used Github Pages alongside Jekyll, but found that the setup for a staging & production environment was tedious.³

Eventually, I switched to Cloudflare Pages, which I recommend as it allows us to access Cloudflare Zero Trust, a system to authenticate users before accessing webpages.⁴ Cloudflare Pages also defaults to creating preview pages, essentially staging sites per branch, and we can configure our staging site to deploy from the development branch when a change is pushed.

Luckily, Cloudflare has a pretty slick Github integration, so we can follow the instructions here. Note: only allow the minimum required permissions for a service integration. Cloudflare Integration should only have access to the repository(ies) that are deployed to Cloudflare Pages.

Custom Domains & DNS Pains

Github & Cloudflare pages host websites, but they do so using their own domains. Github uses a .github.io extension and Cloudflare uses a .pages.dev extension. We would obviously prefer our own custom domain, so what we have to do is change the DNS records for our site to point to the content deployed by our hosting platform. In Cloudflare, here is what my DNS settings look like for my website.

Our CNAME allow us to load the content of the pages.dev, but access it via our custom domain name. My domain name servers are with Cloudflare, which makes the setup very easy. If moving from Github Pages, make sure to remove the custom domain from the Github Pages site & to transfer name servers. We also redirect the www subdomain since it is still sometimes used, and we just want it to redirect to the same content anyways. We can also see that on our Workers/Pages configuration we have setup our custom domains.

Staging Environment

Now, we have automatic deployments from our main branch, as well as automatic preview branches! We could be done, but I prefer to just have one staging site, at least for now. In future, I will cover using Cloudflare Zero Trust to password protect all staging sites.

On the ‘Workers & Pages’ tab of Cloudflare, we can click on the details for our site, then head to “Settings”, then “Builds & deployments”. We can leave the top production branch alone, but here is my configuration for the preview deployments:

So now we only publish a live version of our development branch at https://development.repoName.pages.dev!

Two Environments, both alike in config

Having two environments now means we need to have configurations for each. Hugo allows multiple configs, and we only need two:

• config/_default/config.toml for development & staging
  • This config has 90% of the site settings, since we need most of them for the site to build correctly.
  • My staging website has the title set as “ivy (development)” & the pages.dev baseURL.
• config/production/config.toml for production
  • This config only holds a few settings that we only want on our live site, such as application keys for analytics/comments/etc.
  • My production website has the title “ivy (likethevine)”, as well as giscus (comments) settings, and my custom domain baseURL.

Hugo uses the _default config, but it can apply the production config if passed a flag during deployment. Luckily, since Cloudflare Pages supports Hugo out of the box, we simply need to set a few environment variables in the GUI. Back at the settings for our website, we can define the following flags. These will be passed to the hugo build command when we deploy:

Note: Environment variables are also useful for securing data that absolutely should not be in the main git repository, such as SSH/application keys.

iii. Prevent Disaster with Automatic Checks

Now that we have our SVM and our automatic deployments, lets talk workflow. Github (and other SVMs) provide many automated checks and systems to prevent the dreaded “new dev destroys prod” disaster. Branch protection rules are super simple and useful. Common rules include preventing direct pushes to main, requiring pull requests to be approved by others before being merged, and tons of more complex versions.

Here are the basic steps to working within a system like this:

1. Create a featureName branch off of development.
2. Add a feature/bugfix/etc, modifying as few files as possible.
3. Add files & commit at major checkpoints, leaving short but useful commit messages.
4. When a feature is ready for evaluation, push to the featureName branch, then open a pull request to merge featureName into development.
5. Have the pull request approved & merged by supervisor (or double check manually).
6. Test the staging site.
7. Deploy to production via pull request from development into main.

	gitGraph
	commit
	branch develop
	checkout develop
	commit
	branch featureName
	checkout featureName
	commit
	checkout develop
	merge featureName
	checkout main
	merge develop
	checkout develop
	

Reap the Benefits

So with all of that work, what do we really get? Time to revisit those goals:

• Source Version Mangement
  • Rule 1: software changes need to be easily assigned, tracked, and checked
    • stored in github following git-flow
  • Rule 2: environment variables need to be stored safely (important enough to state again)
    • dual config files locally as well as environment variables stored in our host
• Deployments
  • Rule 1: deploying new versions should be automated via CI/CD and require no human intervention
    • most deployments require a simple approve & merge of a pull request (via a Web UI)
  • Rule 2: staging site to test versions before production deployment
    • staging site automatically deployed from development
• Onboarding
  • Rule 1: new devs need to be able to download and deploy system within a day²
    • with a proper SVM, git flow, and environment configurations, onboarding will be a breeze
  • Rule 2: new devs should not be able to break production
    • branch protection rules, pull requests, and automated integrations mitigate or prevent most accidental destruction

The initial setup takes time, for certain, but now we will gain efficiency almost every single day from now on! Your managers, coworkers, and your future self will thank you, trust me.

Footnotes

The Background

I have been in tech for over 10 years (look at little me up there), and I have seen firsthand that a proper workflow is essential for any decent project. So, when I bought a 3D printer a few months ago, it was embarrassing to see my lack of workflow. Dealing with the dozens of STL files was a hassle by itself, but then, it came: Raspberry Pi SD card corruption. And it came again… and again… and again…

With least an hour of configuration per corruption, I finally bit the bullet and started setting up my own server. tl;dr? here’s the full setup script

Why a Home Server?

I have avoided making my own server for the past few years because I did not have a need for one. At Amazon Music, we had access to the internal architecture & work flows, and during my contracting work, our applications were usually relatively straightforward GIT/CI/CD Node/similar applications.¹ But now, I have graduated college and have the time & skill to setup my own, so lets do it — on the cheap.

Enter… the Laptop Server

No, a laptop is not the ideal server. They lack multiple ethernet connections, extra storage, and were not designed for this use-case. However, for personal-use deployment, they come with many benefits:

1. Power Efficiency
   • Laptops in general are built for battery life, and thus are designed to draw less power than a desktop/server often would.
   • Less power use = less waste heat. This is critical to me since the server must be in my bedroom to have access to ethernet.
2. It’s a Laptop
   • Small, easy to place wherever. (Mine sits under my TV)
   • Instead of a UPS, the laptop battery itself provides a healthy backup when power is cut-out (or more likely, when there are power fluctuations).
   • Built in keyboard/monitor for debugging when something breaks.²
3. I already have one. While I would love a full custom rack setup, the best server is the server you have right now.

The Hardware - Nessie, the 2012 Macbook Pro

For my setup, my server hardware is my old 2012 Macbook Pro 13":

• Intel Core i5-2430M: 2 cores + hyperthreading, 2.4-3.0Ghz
• 2 x 2GB DDR3 1333MT/s RAM
• 1x Gigabit Ethernet
• 1x Thunderbolt 2 Port
• 2TB Crucial SATA SSD - Upgraded from original 128GB SSD.

The specs are not incredible, and I was uncertain if this would be a performant system (spoiler: it works fine!). I upgraded the SSD with one I had laying around, since 128GB was not sufficient. Production servers should always have redundant storage and networking, but beggers can’t be choosers! It is possible to adapt the Thunderbolt 2 port to Gigabit ethernet, but dual networking is a rabbithole to follow another day.

The Software - Keep it Simple

• OS: Ubuntu 22.04LTS
• Host Management: Cockpit
• Container Management: Portainer-CE
• Virtualization: Docker
• NAS Mount Protocol: NFS

There will be a future post going more in-depth about the specific tech selection! Many popular alternatives were tested or considered, but I arrived at this setup because each of these technologies are well known, well developed, and have tons of community support & documentation.

Initial Installation

My servers all run Ubuntu 22.04LTS.³ Installation of Ubuntu is relatively easy, and the only install configuration required is “Install OpenSSH Server”. Aside of that, the default settings work fine. Installation Instructions.

Note: this guide assumes that the laptop is plugged into ethernet during the Ubuntu installation process.

After installing and rebooting, we need to ssh into the laptop to configure it. At this point, our system has a Dynamically Assigned IP address (such as 192.168.1.10), which we need to know in order to connect. If you have access to your router login, simply find the IP address assigned to the hostname of the server, in my case nessie. If you don’t have access to that, then login to the server on the laptop (w/ the username & password from installation), and run the following command: hostname -I | awk '{print $1}'. The result is the current IP of the host. The next commands can be run on the device itself, but I find it easier to connect to the host & then copy paste from my normal laptop (versus trying to manipulate text from a command line). So let’s configure us a server!

The Five Command(ments)

I. Thou art a Server

Although we’re installing a server OS, there are a few things we want to configure to allow our laptop to “act like a server.” This command does 2 things:

1. Prevent system from sleeping when the lid is closed.
2. Turn off the screen after 60 seconds (but not fully disable – to allow on-device debugging).

sudo echo $'HandleLidSwitch=ignore\nHandleLidSwitchDocked=ignore' | sudo tee -a /etc/systemd/logind.conf &&
sudo echo "GRUB_CMDLINE_LINUX_DEFAULT=\"consoleblank=60\"" | sudo tee -a /etc/default/grub &&
sudo update-grub

** The && chaining syntax means that the next command will not execute until/if the previous command executes successfully.

II. Thou Shalt Have Basics

Ubuntu 22.04LTS comes with many useful packages, but there are a few that I always end up using, so I’ve put them here. Of note:

1. The $nrconf line prevents a pop-up window listing services that should be restarted. This halts our config, so I have altered the configuration to list and not pop-up and halt.
2. avahi-daemon and avahi-utils are critical for our link-local addressing.
3. nfs-common is the package we will use to access the files on the NAS as if it were an external drive plugged.

sudo echo "\$nrconf{restart} = 'l'" | sudo tee -a /etc/needrestart/needrestart.conf &&
sudo apt update -y &&
sudo apt upgrade -y &&
sudo apt install -fy zip git htop software-properties-common apt-transport-https wget xclip net-tools curl python3 python3-pip nodejs npm ca-certificates gnupg avahi-daemon avahi-utils nfs-common &&
sudo apt autoremove -y

III. Thou Shall have Containers

In the Homelab/Selfhosted/NAS space, containerized applications are common (and awesome!). Docker lets our server run containers, meaning we can run software on our host server without having to configure the host for every single application. Each container is separate and (more or less) sandboxed.⁴ In the case of deploying templates, containers are essentially “applications” that run on our server.

sudo install -m 0755 -d /etc/apt/keyrings &&
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg &&
sudo chmod a+r /etc/apt/keyrings/docker.gpg &&
echo \
 "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |
sudo tee /etc/apt/sources.list.d/docker.list >/dev/null &&
sudo apt update -y &&
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin &&
sudo usermod -aG docker $USER &&
newgrp docker &&
sudo systemctl enable docker

IV. Thou Shalt be Web Monitored

For this server, two different monitoring solutions are needed.

1. cockpit - managing the host device & NAS file sharing
   • we also install cockpit-file-sharing from 45Drives, which allows easy NAS setup from the cockpit webUI
2. portainer - a containerized application that allows the monitoring & deployment of other containerized applications

sudo mkdir /portainer_data &&
docker volume create portainer_data &&
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/portainer_data portainer/portainer-ce:latest &&
sudo apt install -y cockpit cockpit-pcp &&
curl -LO <https://github.com/45Drives/cockpit-file-sharing/releases/download/v3.2.9/cockpit-file-sharing_3.2.9-2focal_all.deb> &&
sudo apt install -y ./cockpit-file-sharing_3.2.9-2focal_all.deb &&
rm ./cockpit-file-sharing_3.2.9-2focal_all.deb

V. Thou Shalt Not Break DNS

After hours of blood, sweat, and tears, I’ve arrived at a networking configuration that satisfies the following:

1. Work with cockpit. (Requires use of NetworkManager for proper updates & network monitoring in the Web UI, not networkd)
2. Don’t break DNS. (Can’t run my own DNS server/sinkhole)
3. Use a named address & a dynamic IP. (<hostname>.local with DHCP)

sudo echo "network:
version: 2
renderer: NetworkManager
ethernets:
$(ip -o -4 route show to default | awk '{print $5}'):
dhcp4: true
link-local: [ ipv4 ]" | sudo tee /etc/netplan/00-installer-config.yaml &&
sudo netplan apply

Essentially what I am doing here is overwriting the default wired netplan with my own configuration. The renderer is set to NetworkManager to work with cockpit’s UI. The line $(ip -o -4 route show to default | awk '{print $5}'): is a nifty way to grab the current active ethernet connection and configure it (versus unconfigured or virtual interfaces). On my macbook, it evalutes to ens9, the name of the built-in ethernet interface. We enable dhcp4, since my home network only uses ipv4, not ipv6, and my router uses DHCP instead of static IPs (as do most home networks).^{5 6 7}

The link-local: [ ipv4 ] line is also crucial, as it works with avahi-daemon/avahi-utils to tell other devices on our network that nessie.local can be used, not a dynamically changing IP address. Alternate solutions involve editing files on every host (which is a hassle), or creating a custom DNS server (which is also a hassle and I could break the wifi for my roommate, which would be very bad!)

After the new netplan configuration is applied, our ssh session break or disconnect, so close the terminal and wait a few minutes. Then, reboot the machine (from command line: sudo reboot). When it finishes booting (which may take a few minutes, internet interfaces can take a while to start), you should be able to access it via https://<hostname>.local:9090 for cockpit https://<hostname>.local:9443 for portainer. At this point, all management can be done via cockpit’s terminal or portainer’s UI!

Congratulations 🎉

Now just configure the software we’ve installed & enjoy:

• Initial Portainer Setup

• Optional: Change portainer templates to 3rd party repo to allow more 1-click application deployments, such as https://github.com/lissy93/portainer-templates

• Configure SAMBA on the server for Windows Clients

• Configure NFS on the server for linux clients

• Optional: Mount folder at boot on Linux client with the following command (requires sudo apt install -y nfs-common first):

  sudo echo '<hostname>.local:/<nfs_folder> /<local_mount> nfs rw,auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0' | sudo tee -a /etc/fstab &&
  sudo mount -a

  • <hostname> - ex: nessie
  • <nfs_folder> - the name of the nfs folder created above
  • <local_mount> - the folder on the client device to alias to the server
    • often the /media/<folder> or /mnt/<folder>. ex: /mnt/nessie
    • folder must be created before the fstab file modification⁸, ex: sudo mkdir /mnt/nessie

Next up, we’ll configure offsite backups via a containerized application since our server has ZERO redundancy!

Footnotes