jonwatson blog

broadcasting useless crap to the internet since 1200 baud

I work for one of those very rare companies that lets me try to do things I’ve never done before. I’m many years into this Linux sysadmin thing (La Cosa Linuxstra) and you don’t get very far into this gig without having some scripting chops. Sysadmin code is generally only as elegant as “whatever stops this emergency right now” and I’m the champ of hackey code that keeps stuff running. What I have less experience with is designing complex, easily maintainable code.

Despite that shortcoming, I’ve been entrusted to write some software automation. I’ve spent the last several months creating a bot that takes loosely formatted customer input and tries to produce useful responses. The challenges have been great all throughout the stack. At the bottom, I’ve had to become more familiar with developer stuff like pull requests (wha?) Somewhere in the middle I’ve had to shelve some really powerful concepts because the underlying infra doesn’t exist. At the top, I’ve spent some tense moments trying not to cringe with embarrassment as my fledgling bot gives an absolute asinine answer to a fairly straightforward request.

Bots are like kids. They’re never really grown up. But you can still learn a lot along the way. Here’s some of the things I’ve learned on this ride.


Log file showing invalid SSH logins

Brute force hacking is the easiest, least effective, and messiest method of all the ways to attempt to gain access to a system. It leaves a really obvious trail, and it’s fairly easy to stop unless you’ve become the target of a large organization that really is out to get you.

By definition, brute force hack attempts are simply some variation of just trying to guess a proper username and password combination. I will look at attempts to break in to a Linux box via SSH, but the principals are the same regardless of the attack target.

Linux tracks log ins in two files and there are two separate commands to read them: last and lastb. The last command shows successful logins and system reboots, whereas the lastb command shows unsuccessful logins. Let’s take a look at the latter.


A porcelain frog holding a pile of books

The general idea of remote work is that you do the same job you would do in the office, but you don’t have to actually go to the office. This removes all the problems with people and politics of the office. That’s viewed as a huge benefit, but the reality is that many people only keep their jobs because of the people and politics of the office. Remote work strips all that away and leaves you standing naked in a meritocracy where only your skills matter.

I’ve worked remotely for 7 out of the last 9 years. For 4 years I was a remote contractor left to my own devices. I spent 2 years working as a remote worker for a non-remote company and I’ve spent the last year-ish working as a remote worker for a remote company. While sitting at home looks the same in all cases, each of those situations were very different from each other.


Tutanota app screenshot

I have used Proton Mail for quite a while because it was really the only horse in the race a few years ago. Hushmail was OK, but too expensive (although I did use it for a year) and Tutanota was too immature – my biggest complaint being that it did not support 2FA at the time which was too glaring a deficit for me to overlook. Fast-forward a year or so, and I am moving to Tutanota.

My choice to move to Tutanota is almost entirely based on cost. Proton has a lot of features but I really only use the custom domain feature, and its associated aliases support. For those features I pay about $8 CAD/month. For the same level of plan on Tutanota I paid $18 CAD for the entire year.


CCTV camera

The “Internet of Things”, or IoT, refers to the ever expanding offerings of traditionally non-Internet connected things that can now be connected to the Internet. The array of things you can connect to your home wifi network is staggering and, to be honest, pretty dumb. Internet connected toasters, light bulbs and even hot tubs are all available to lurk on your home network and send god only knows what data about you to god only knows where.

Your home network should be a safe place where only trusted devices have access. Traditionally, this has meant your own computers, your own smartphones and perhaps a few other devices such as gaming consoles. The problem with attaching a new device to your trusted network is two-fold: does it make attacking my network easier and what is it doing with the data it collects?


Door knocker

I was thinking about port knocking the other day (yep, that’s how I roll) and while I consider it to be a valid security layer, it occurred to me that it would be pretty easy to set up a poor implementation of it that was susceptible to being gamed. Here’s how that thought process went.

Caveat: This is a proof of concept and has many points against it which I outline at the end of this post.

For the uninitiated, port knocking is a process whereby some port on a server can be fire-walled off until some pre-determined set of ports are ‘knocked’ on, and then the firewall can be reconfigured to open some other port. A practical example is a server where you need SSH access, but you don’t want to leave the SSH daemon running wide open to the world all the time. You can use a port knocking daemon like knockd, coupled with an IPTables firewall to protect that port. The normal configuration would be to have the SSH daemon running on some arbitrary port and have the firewall dropping connections to that port until a valid set of ports are knocked on, and then the IPTables would be rewritten, usually temporarily, to allow connections to the SSH port.


Binary map

Note: I wrote this in October 2016 after the Dyn DDoS attack.

There is a lot of blame to go around in the aftermath of the Dyn DDoS attack on Oct 21st. A good chunk of the bots look like Internet of Things (IoT) devices that were recruited by the Mirai botnet code. Mirai has dropped the traditionally high costs of building a botnet to near zero which means we’re seeing progressively larger and more effective DDoS attacks each week.


The Monoprice Select Mini is a very popular 3D printer. It’s inexpensive, small, and very hackable/fixable. For those reasons, it is a popular 3D printer among newbies and experienced printers alike. Having said that, it has one very significant flaw that affects pretty much 100% of units. The Mini comes with a heated bed which is a nice feature for an inexpensive printer. However, the wiring design is terrible and the movement of the bed while printing breaks the wires that control the bed temperature, usually within a few weeks. Mini owners are lumped into two camps on this – those who rewire the bed so that it will not break again, and those who just don’t care that the bed no longer heats. I was in camp 2, but eventually wanted to graduate to printing with ABS. It’s easy to print PLA on a 3D printer without a heated bed, but it’s much harder to get good ABS prints on a cold bed. So, I sent my printer back RMA to get it fixed up good as new, and then promptly rewired the bed when it came back to ensure the wiring would not break again.


Fiber patch panelIf you're working in data centers, you're going to need to know something about fiber optic connections. Fiber is the most common type of connection used for that last mile between backbone providers and your equipment. Just like any other cable, fiber cables have to connect to your equipment and this article is about two commonly used fiber connectiors: the SC and LC connectors.

A quick primer

First, some background. My exposure to data centers is generally a single cabinet in general population. My DC provider gives me rack, power, and a patch panel. The patch panel is the central part of this article.

The patch panel is generally mounted at the top of the rack because most data centers run their cables over head. The part of the patch panel that I use is referred to as the A-side and the other side is referred to as the Z-side.

When I deploy into a new DC, my backbone provider gives me a Letter of Authorization (LOA). I take that LOA and order a cross connect from the DC using that letter. The DC techs need that LOA because it is their authority to connect my backbone provider to my patch panel. The DC techs will run cable from the ports my backbone provider has specified on their patch panel in the bowels of the DC (the Z side) to the ports I dictate on the patch panel in my rack (the A side). I then show up and start plugging things into my patch panel (the A side).


Write Freely is the self-hosted version of I stumbled across on the Fediverse and immediately took a liking to it. It's spartan, clean, and has literally no distracting features to play with; it's perfect for people that put content first.

One of the features I liked as a user of was the ability to have multiple blogs. Write Freely has that feature as well, but not when you run it in single-user mode. Putting it into multi-user mode does indeed give you multiple blogs, but it also activates a landing page for users to log in, pay for services, etc. I was looking for a single-user setup with multiple blogs. That combination is not available out of the box so I documented what I did.