An Ancient Memory
Sitting around a table eating lunch with strangers in an earlier epoch…
“So, what’s your deal?” I asked.
“Arms dealer,” he replied.
“Do tell,” I probed.
“I work for Intel,” he offered.
“Ah, I see now.”
I miss conferences. I miss lunch.
Of late I have found myself, somewhat flippantly though maybe more accurately than comfortable, describing my cyber career arc as “from solider, to security guard, to arms dealer, to mercenary”. I don’t doubt that the aforementioned conversation played a role. Much truth is said in jest.
Last Night’s Prelude To A Dumpster Fire
So last night I’m at the part of S1:E4 of Halt And Catch Fire when, ironically, just as they are using a disk degaussing power surge as plot device, the dual port NIC in my bespoke home network router, uh… more or less halts and catches fire. I did not realize at the time the cause of the spinner that appeared in the Netflix iPad app as I stood at the kitchen counter shoveling delivery lamb korma into my pie-hole but soon enough I would receive a hint of the drama to come.
I poked at my iPhone and realized that it, too, lacked its usual spunk, and so shut off its WiFi and bonded the iPad to its hotspot. This would at least allow me to finish my 2021-style dinner-and-a-show experience, albeit first with herky-jerky performance, and subsequently after an interlude for some buffering a tolerable enough experience.
After nomming the noms and watching the dramas I headed downstairs to find that die blinkenlichten wurden traurig (not blinken at all actually) da (presumably) sie haben blowenfusen und poppencorken mit spitzensparksen or something to that effect. I concluded as much, in any case, after the tried and (mostly) true off-and-on-again remedy failed to work.
/sbin/ifconfig -a * ALL YOUR FRIENDS ARE DEAD *
Evincing a whiff of optimism and preparedness I rummaged through a nearby pile of stuff, found an unopened duplicate dual-port NIC (which makes me think of a former girlfriend who would regularly remark “two of everything!” of my purchasing proclivities), tossed it by the computer, and went upstairs to sleep, perchance to dream fitful dreams of imminent network debugging drama.
Running through my usual shutdown procedure I pulled up the Hue app on my iPhone and… “can’t connect to bridge”. Right, I turned the WiFi off… Ok, turn it back on and… no dice. Ahhhh fuhhhhhhk, this is what I get for setting the Eero’s “DHCP & NAT” setting to “Bridge” — now all my smart lights are dumb lights and I have to go around fiddling with hard switches to shut them off. Graceful failure NOT SO MUCH but clearly a problem for Future Andrew to solve. Off to bed!
Today’s Trials And Tribulation
Wake up, feed the kitties, swallow the vitamins, jump in the shower, annnnd we’re off to the races! The clock is running because there is a meeting later in the morning where reasonable bandwidth piped into my mountain home via a wired connection is a pre-req for Zoom. But first — take a snapshot!
Pop the case, swap the card, button it up, replug the cables annnnnnd… the damn card moved because I failed to screw it down. MULLIGAN. But finally happy looking lights. At least preliminarily happy.
And now the real fun begins. FIRE IT UP!
Gah, you flabby weak ass old BIOS battery, can you not go a hot minute unplugged without forgetting what time it is?
Sorry, I meant what DECADE it us. Good grief I hadn’t even finished college by then. You hadn’t even been BORN by then, Osprey. (that was 2009 since you clearly forgot)
Anyway… Boot it up, log on in, see what the what and… some kind of “can’t establish network connection” dialogue pops. The nearby Eero plugged to one of the ports on the dual-port NIC, meanwhile, continues to glow red. Gonna play hard apparently.
Guh. How do networks work at this level? Oof — gonna have to spool up the tape archives in my brain for this.
IP addresses… Computers love IP addresses. Let’s see if…
sudo systemctl status isc-dhcp-server.service * EVERYTHING IS AWFUL. I CAN'T EVEN BIND TO A NIC.
OK, Linux, what’s your deal…
/sbin/ifconfig -a * THERE ARE TWO FAMILIAR LOOKING NICs. ONE HAS A PROMISING IP. ONE HAS NOTHING. IT IS DARK. YOU MAY BE EATEN BY A GRUE. *
I presume the NIC with the 10.X.X.X IP got an IP address from my cable modem and the sad one has somehow gotten borked. I open up the graphical interface for networking whoop-dee-doo and see that the other card is configured for DHCP in a world where nobody is going to tell it who it should be. And so I go digging to find what I should tell it to be.
cat /etc/dhcp/dhcpd.conf * ME TELL CLIENT ROUTE TO 192.168.0.1. YOU TELL NIC BETTER BIND THERE. *
OK. I tell NIC bind there. Things still don’t want to play. I reboot Osprey. I reboot Eero. Things still sad. Rising sense of panic that I won’t make my work call. Feel like failing interview question and buzzer about go off.
Did the port identities auto-vivify backward? Do I need to just flip the cables and then all will be well?
/sbin/ifconfig -a * enp8s0f1 AND CABLE MODEM HAVING GREAT TIME. enp8s0f0 HAVE IP BUT NOTHING SHOW FOR IT. * cat /etc/iptables/rules.v4 * HAHA. NOPE. GUESS AGAIN. * -A FORWARD -i enp8s0f0 -o enp8s0f1 -j ACCEPT -A FORWARD -i enp8s0f1 -o enp8s0f0 -m state --state RELATED,ESTABLISHED -j ACCEPT
Ok, so cabling is fine, NICs seem superficially fine… DAFUQ GOING ON.
At this point I send Slack message of shame that I probably won’t make the meeting.
Osprey can talk to the Internets but none of the devices on my home network bridging through the Eero to Osprey want to play ball and all the Eero nodes are showing the sad red light of doom.
Y U NO ROUTE?
sudo tcpdump -I enp8s0f0 EVERYBODY: ARP ARP ARP ARP ARP ARP ARP ARP ARP ARP ARP ARP ARP ARP ARP ROUTER: *CRICKETS*
OMG I told enp8s0f0 to take 192.168.0.1 but left its net mask as 0.0.0.0. Let’s set that to 255.255.255.0 and…
sudo tcpdump -I enp8s0f0 * EVERYYYYYBODY PAAAARRRRRRRTY!!!!111one!!!! *
FML. I just barely make my meeting with… negative five minutes to “spare”. We speak of things about ten layers up the tech stack, ruminating on DevOps tool chains operating in software defined data centers to field CND awesome sauce that with any luck will keep our nation’s power grid safe from the script kiddy hooligans and nation state nightmares alike who roam the Internets sowing their seeds of chaos. FOR GREAT JUSTICE.
The Ever Deepening Stack
The foregoing may evince a degree of masochism on my part and, yeah, maybe that’s a thing, but actually there exists a larger purpose. I feel blessed to have grown up with our present tech ecosystem in a manner that renders it just barely comprehensible in all its resplendent glory. And I feel thus obligated to keep the various facets of this compendium accessible in moments of need, in moments of crisis, in moments of… metaphor shear.
I appreciate that as Computer Science education was bifurcating, with many institutions corralling students to the higher level Java programming language, my undergraduate alma mater felt we greenhorns ought cut our teeth by hand-crafting algorithms in the relatively low-level C language, then dive even deeper into Assembly programming, before clawing our way up to higher level languages such as Lisp and Prologue, then suffer the whiplash of plunging back down into operating systems, computer system architecture, and circuit theory. I deem myself further lucky to have received such a solid grounding in distributed systems and cryptography later in graduate school.
The grubby outside world would in the fullness of time provide plenty of opportunity to program in Java. Why waste our precious time at The Citadel thus?
Software has grown so complex over the intervening two decades. And what a luxury to grow up alongside it after such a solid foundation, getting to gradually metabolize an assortment of innovations as they arrived on the scene. I empathize with the sense of overwhelmed-ness that new graduates to the field must experience. I am grateful that, for now, I can cling by my fingernails to being a generalist practitioner who can navigate the majority of the tech stack involved in bringing a modern application into existence. I am aware that in the fullness of time this may prove a futile effort. For the moment, anyway, I find myself doing such things as learning the React framework so I can fight my way back to claiming that I am “Full Stack” with a straight face instead of limiting myself to mentioning UX technologies from 2010 (jQuery FTW! 0xA000 or bust!).
At the risk of tipping my hand, one of my long-time favorite interview questions for software engineering candidates is — there is a browser with a URL in the address bar, someone mashes the enter key, and a web page loads; tell me all the things that happened.
Let’s have a fun jam session seeing how full stack both of us really are. Maybe we can start with an emergent flow of electrons bubbling up to the firing of an interrupt handler and then rocket through multiple layers to TCP and TLS handshakes facilitated by myriad other network protocols (DNS, and ARP, and BGP, oh my!) and maybe from there plunge back down to a hard disk head swinging across a rapidly spinning magnetized platter (haha, just kidding, it’s not 2010 anymore (or is it in some places and why?)) and then return to a layer where HTML files land in a browser to set off an explosion of additional requests, rendering operations, and callback firings.
I (heart) Serverless, but The Cloud is just somebody else’s computers, and things always be breaking. The more you know how this shit show works the more you can slice through the problems the world flings your way like a hot knife through a distributed trace.
A programmer should be able to change an ethernet board, plan a sprint, butcher a legacy system, design an algorithm, write a SQL query, set an interrupt handler, comfort the debugger, allocate memory, free memory, solve race conditions, pitch manure, program a GUI, cook a tasty meal, fight efficiently, and die gallantly.
Specialization is for insects.
Or at least that’s what Heinlein would say. But damn is it hard and getting harder to be a generalist.
My more recent adventures as “arms dealer” and “mercenary” certainly seem to drive this home. Increasingly my ability to help various ecosystems along seems to involve operating at two poles, namely as the hyper-generalist architect and meanwhile the hyper-focused specialist in a few select areas.
There is just too much for any one person to know.
Best to have a high level map of the terrain, a few areas of hands-on expertise, and a wealth of colleagues to fill in the gaps. But maybe don’t neglect the low level arcana either. Ain’t nuthin like being able to solve your own problems when the familiar metaphors turn out to be nothing more than comfortable lies and your world is on fire.