Most ops folks take the “typical” computer boot process for granted. On today’s typical x86 laptops, desktops, and servers, the process appears pretty straightforward. Push the power button, wait for the BIOS/UEFI to do some magic, and the computer will attempt to start an operating system from whatever media it’s configured to use. If you want to boot from a new media type (such as USB, SATA, and NVMe drives, or PXE network boot), it’s typically as easy as adding the boot media to your computer, changing an easy to configure option in the BIOS, and continuing like normal.

Other CPU architectures, and in general single board computers (SBCs), do not have these luxuries. Generally speaking, when a SBC boots, board designer and/or operator must directly provide the CPU with all code and configuration for the entire boot process, often down to the very first instruction that is executed. This is usually more than a simple “run the operating system on hard drive one”. It typically entails telling the CPU how to load the operating system, as well as configuring the hardware for this process. In this post, I will cover what this process looks like for the RK3588, and I will show how I added NVMe boot support to the Blade 3 board.

The control board of the Cluster Box uses a program called nodectl to handle out of band management for the nodes. This includes things like powering nodes on and off, and providing serial console access. It is (very) specific to this hardware, and unfortunately not open source. In this post, I will detail how I reverse engineered the binary and rewrote it in Go, fixing some bugs along the way.

Unexpected problems

I had originally planned on this post being about the first boot and setup of the box. Unfortunately, I hit several major issues and may need to RMA the control board. I’m in talks with the manufacturer about the issue, but I don’t have a resolution yet. I’ll complete and upload a setup post at a later date.

With a brand new Mixtile Cluster Box in hand, it’s time to open it up and get to work. In this post, I will share some pictures of the product, and some additional technical information not available elsewhere.

I’ve been having some issues with my home Kubernetes cluster lately. One of my servers has been consistently triggering a kernel panic on reboot, one is constantly OOM killing processes, and my rack sucks an enormous amount of power at idle.

I have also been neglecting my updates maintenance over the past year or so. I use a combination of Renovate, GitHub, and Flux to manage automatically deploying updates to the services I host, but breaking changes require manual intervention. Rather than trying to update my existing stack, I’ve decided to replace (most of) it.

In this series I am going to cover (nearly) everything I know regarding writing mods for .NET games. I’ve written and contributed to several mods over the years for different games, including Minecraft, Planetbase, Bloons Tower Defense 6, and Spiritfarer. This series will primarily focus on buildings mods and modding support from scratch for Unity-based video games. This post will detail broad topics regarding modding applicable to all software.