Disaster

Feb. 13 update: I can receive emails now. (I still can’t send, unfortunately, but I can just reply with any other email address. So functionally speaking, my email works!)

As you may have noticed, this website and everything related (puzz, Math Advance) have been down for a week or so. Here is the story, roughly, of why that happened.

When I went home for break, I bought a Raspberry Pi and set up a server back home. Then I moved all the websites and email onto there, with no clear backup plan for what I’d do if everything went bust. Not everything was fully migrated, but I decided it was good enough. Satisfied, I shut down my virtual Ubuntu server that had been running for several years and called it a day.

Well, everything went bust.

A few days after I leave for college, I realize my emails are suspiciously empty. That was the first sign that the installation was somewhat botched, and upon investigating, the problem is with the mailcow docker process. Okay, maybe mailcow just doesn’t work on Raspberry Pis. So I spin up a test server, and it seems like Alpine is actually the issue. So I can’t install mailcow on the Raspberry Pi. That leaves me two options:

• Set up another VPS just to handle mail,
• or mystically figure out how to install Postfix (grr I wish the documentation were better).¹

I am a busy college student. Life has many demands. So after trying the second route for a bit, I decide that getting my email working couldn’t be that important. It could wait…

But then everything, and I mean everything, came crashing down.

I notice this when I check on my puzzle website. It’s down. Hmm, odd. My first suspicion is that the connection in our house died, and that indeed was the culprit. I couldn’t SSH into the server, and nor could my family. This posed a big problem: I stupidly left the Pi plugged in somewhere where no monitor could realistically be moved to it. Because Raspberry Pis are pieces of shit, they don’t have a power switch. And no one can SSH into it. This means, effectively, that no one can see what is going on in the screen.

So my brother has to blindly type out commands that I dictate (none of which he really understands) to try and attempt to either reconnect to the internet or shut the Pi down (so it can safely be moved to a monitor). Neither works.

Exasperated, I decide to take the risk and instruct him to unplug the Raspberry Pi. I read online that it usually would not corrupt the data on the SD card. Apparently it did. The damn thing would not boot, or so I was informed. Either way, being a coast away meant that I couldn’t look at it myself. So I needed to set up something in Pittsburgh.

I think about setting up an Orange Pi. But then I realize, a) our room doesn’t have an Ethernet port (at least I have not been able to find it in a semester and counting), b) I don’t have a monitor or a USB keyboard, and c) it would be a massive pain in the ass. So I go back to where I started: a virtual cloud server.

Lessons

• Always have direct access to your hardware.
• Don’t host somewhere with shoddy wifi.
• Plan for and simulate failures. What happens if the Wi-Fi is out? What can you do?

All these seem so obvious in retrospect. It’s the kind of thing that they tell everybody running a server anywhere. Yet it seems, for better or worse, that I have to learn the hard way: reliability above all else.

What now?

Only dennisc.net and puzz.dennisc.net are up as of right now. Here is what I plan to do:

• Get email working. This is the top priority.
• Get mathadvance.org back up. This isn’t really too hard to do, but it’s also not that important.
• Get mat.mathadvance.org back up. Past contests are on there, so this is quite important, but also trying to host a Node JS application seems quite difficult. (There are some really big dependency hell issues, mostly because Node 16.7.0 -> Node 17+ contained a breaking change that fixed SSL of all things, and I have not been able to figure them out yet.) So it’s high priority, but also realistically high effort.
• Get the rest of the random stuff up, like git.dennisc.net and mast.mathadvance.org (which effectively have no content).

It’s likely that I pay for this cloud server for the foreseeable future. (In the time between installing my old server and the new one, the minimum price went down from $10/month to $6/month.) But when I move into a real house, it is also quite possible that I give the whole Raspberry Pi thing another go. As annoying as system administration is, it can be rewarding to have a whole homegrown setup as opposed to outsourcing the server somewhere far away. And with more free time in the summer, it is quite likely I will have the motivation to try it again.