Distributed System Fail

The other day when returning to my hotel room from near-100F heat I found myself willing to spring the $2.50 the vending machine wanted for a Vitaminwater.  I lacked the paper or metal currency to complete the transaction but thankfully the machine accepted plastic.  Or at least appeared to.  Except someone botched designing fault tolerance into the system.

I swiped my Visa but after the system hung with the text “authorizing transaction” for a minute it barfed with “communication error”.  So I tried my AmEx instead in the vain hope that the networking failure was not at the last hop.  Alas, no dice.  Here I was, willing to pay retail++ for a Vitaminwater but the system did not want to take my money.  Because it couldn’t talk to the mothership and maybe I wasn’t good for the $2.50.

Now, I’m pretty sure that when a credit card authorization service gets interrogated its implementation is to default to authorizing the transaction (as long as it is reasonably small) if a non-confirmation of the transaction does not come back within a few seconds because the wheels of commerce as a whole need to keep turning regardless of whether some pissant network service feels like being helpful at a given moment.  The vending machine’s own architecture would have done well to draw inspiration from the larger system’s design choices.

It seems entirely reasonable that the vending machine instead might have handled an inability to reach the authorization service by letting the sale complete and caching the transaction for later transmittal.  To deal with an attacker it might cap a given card to a single cached transaction so that someone cannot simply disconnect the machine and then use a single bad card to clean out the machine without paying.  And maybe to thwart a really determined attacker who is carrying a wad of bad cards and for some reason is willing to squander one card per stolen beverage you could cap the total number of system-wide cached transactions.  And, hey, don’t a lot of vending machines these days come equipped with cameras that could be retroactively used to nail a really determined attacker (as well as deter shenanigans in the first place)?  Surely such a simple arrangement would have sufficiently contained the attendant threats to make the operator willing to complete a transaction that would net him $2 at risk.

To be fair, though, I have no experience in the vending machine business, except that sometimes I just want a cold beverage but the machine doesn’t want to play.

Kudos To BGE UX Devs

On my long pre-move checklist was terminating my BGE gas and electric service.  I was anticipating this somehow being a Kafkaesque-ly unpleasant experience but the BGE mobile workflow really nailed it.  It started with the UI best practice of making the actions that are most common across the user base yet rare for an individual get prominent placement on the page and then followed up with a very clean workflow.  Hats off!

Death By Plummeting

I am grateful I was not too thoroughly immersed in my phone when I encountered this on the sidewalk in Baltimore near the intersection of Eastern Ave and S Chester St @1030EDT on 2016-07-08. 

I stopped and spoke with a member of the BGE crew and suggested that this was probably not an adequate setup in a 2016 world where distracted walking is the norm.  He politely thanked me for my input and said they did not have enough gear to do any better.  I suggested that they reallocate one of the several cones they had placed in other locations of much less obvious utility.  He seemed disinterested and when I looked back two blocks later it was clear they were not going to improve the situation.  The fact that this was not merely an oversight but rather an intentional act of disregard for safety makes it especially troubling.

What I would really love to know is…  If someone fell down this shaft, how long might it be before someone even realized?

Edit @1130EDT: This has become Baltimore City 311 issue #16-00496948.


Automated Target Recognition

As I wended my way through airport security in Atlanta this morning I partook of the informational video loop telling me about the great security technology the facility has in place. Chief among the topics was Automated Target Recognition. Immediately after that I passed by body-armored officers toting silencer-equipped assault rifles. Leave it to the government to create the most anxiety producing branding time and again.  Could you maybe not make people feel like they’re just about to be lit up by skynet?

Sadly this is utterly unsurprising.  At least they did not have a logo as terrifying as the Total Information Awareness program of some years ago.


I would love to be the artist commissioned by the government to provide a graphic representation of the Automatic Target Recognition system in action.  I think I would start with guy-crushed-by-gate as inspiration and work from there.


Everything Is Amazing And Nobody Is Happy

As a software engineer who works in a DevOps-ish fashion within a large enterprise environment as the lead of a highly integrative product, the day-to-day can prove a maddening existence of stuff seeming to break incessantly. Our own software evolves with implications we cannot always predict with latent bugs sometimes not becoming evident until months later when obscure conditions emerge in the larger environment, other systems change their interfaces or performance characteristics in not-entirely-coordinated ways, and even ordinarily transparent things such as DNS or an underpinning SAN can go sideways without warning and ruin your evening. A drama free day can seem the exception to the rule and a drama free week laughably implausible.

In my personal life I will now-and-again have a wow moment when I actively ponder how seamlessly the technology that mediates my everyday existence on the outside (usually) functions. Sure, there are glitches, but on the whole it still amazes if you’ve got the right perspective. The sheer number of things that must all somehow work together boggles the mind. If you’ve actually tried to keep a system-of-systems in good working order then you know what I mean.  The Internet Of Things keeps on truckin’ thanks to the hard work of an army of unseen and unsung back office heroes.

I was having such a moment this morning on my drive to the airport at the end of a skiing vacation in Utah. I stopped for gas, filled the tank with liquid fuel from the energy grid, swiped a token that serves as my gateway into a massive payment network, and I was on my way to return my vehicle to the “car cloud” whence I had borrowed it a few days earlier via a reservation over the Internet and subsequently to partake of the miracle of human flight which has become so economical that I can without hesitation employ it for something as frivolous as teleporting a couple thousand miles for a few days of skiing at a cost equivalent to a handful of nice meals out on the town. This quick jaunt to the airport, meanwhile, included using a pocket-sized supercomputer that integrated trivially with the borrowed car to play digitized music selected by a big data crunching artificial intelligence while merging real-time data from an ad hoc network of other such mobile supercomputer sensors and an array of satellites to guide me to my destination.


Everything is amazing but nobody is happy.

And now I am posting this bit of prose via a medium that can deliver it to millions of viewers. Never mind that I’d be lucky to have 100 people read it.

P.S. People might be unhappy because the little girl in the seat behind them on the airplane is using some amazing technology to watch a personal in-flight movie but doing so without headphones. Inconsiderate poopfaces will always find a way.

How to piss off your users and never be the wiser

I just had an exasperating experience trying to pay my Johns Hopkins hospital bill online, exasperating enough to compel me write about it, and exasperating enough to make me reflect on software I have written.

It started off with my opening an envelope and looking at my billing statement.  I was resigned to paying via snail mail until I noticed some small print saying “go to hopkinshospital.org to pay your bill”.  So, I went there, and was _not_ greeted by a nice big obvious “pay my bill” button, but rather had to click “billing and insurance” from among seven tabs at the top of the page, which when clicked took me to another page headed with the unhelpful text “For answers to your billing and insurance questions, please visit the comprehensive payment information provided by Johns Hopkins Medicine”, and further down which was a list headed with “I want to…” which did include the item “Pay My Bill”, but in slot seven of eight along with other such items as “make a gift”, “apply for a job”, and “refer a patient”.  WTF?  Fine, whatever, I clicked “Pay My Bill”… which took me to another page with a blob of preamble text followed by _another_ list, this time with _twelve_ (!) items, of which item seven was “view/pay your bill online”, which I would swear I had already indicated was the desired task to perform. Clicking this “no, really, I would _love_ to just pay my bill” link I was brought to another page where I was instructed to click on one of three pictures of bills with the captions “Blue Bill”, “Green Bill”, and “Clinical Practice Association Bill”. Blessedly not being color blind, in which case I would have been SOL, I clicked “Green Bill”, which took me to another page where I had to choose whether I was trying to pay a bill for “The Johns Hopkins Hospital”, “Johns Hopkins Bayview Medical Center”, or “Howard Country General Hospital”. I clicked the first link, and was taken to an external site at jhhs.ixt.com, finally arriving at something looking like a place to enter payment information.

This external site was also brain damaged. After clicking “pay without enrolling” I was presented with the opportunity to enter my “account number” and “medical record number”. I should only have had to enter a single identifier, a statement ID, but whatever. I entered that information and was brought to another form that had pre-populated my billing amount and that required I do nothing more than hit continue. I was then presented with a delightful form with the form fields configured in such a way that hitting the tab key caused me to alternate between address fields and credit card information fields, because, you know, it totally makes sense to enter my street address, and then my credit card number, and then my city, and then my credit card’s expiration date, and then my zip code, and then the CC’s CVV. Bah, whatever, power through! SUBMIT! Ignore the fact that the enter key never works and I must always use the mouse to click on some stupid button. What, what? “Invalid dollar amount”? But you pre-populated that value for me. And you won’t let me edit it. WTF?! I JUST WANT TO BE DONE WITH YOUR STUPID WEB APP.

Should have just gone the snail mail route… Sunk costs fallacy here we come… No, screw that, I’m not going to fight with this, I’m just going to mail the damn payment and never try to pay a Hopkins bill online again… I’ll use the 17th Century tech, thank you very much…

Except that’s not how this story ends…

And I skipped one of the earlier parts of the story…

You see, at the very beginning of the web app experience I was presented with an in-page pop-up window that asked “do you want to participate in our stupid survey at the end of this ordeal?”, and I of course clicked the “yeah, right, PASS!” button. By the time I was greeted with “invalid dollar amount”, however, I was just about ready to hunt down and murder the developers of this system, and damn it I really wanted to say so in their stupid survey. But, NO! Too late. You said you didn’t want to do so at the beginning of the workflow so now you can’t. WTF? If you had told me I would have been so pissed off by the end of this experience I would have clicked the “hell, yeah” button, but at that point I didn’t know that your god forsaken web app would be so cartoonishly bad that I would be be willing to wage a jihad just to let people know.

So I tried to walk through all the steps again, on the forlorn hope that I would be asked to participate in the survey again, but no such luck. Employing my l337 h4x0r skills I cleared my cookies and went at it once more. BAM. We’re in. “Yes”, I clicked, “I will take your stupid survey”. And then, oddly enough, I was actually able to get all the way through the app with no “invalid dollar amount” nonsense, managing at last to pay my bill.


And stupid it was, the survey… After forty some questions that wanted me to provide a one-to-ten score on such nebulous things as “how would you rate your overall experience”, I found, after much scrolling, a free-text box for “other comments”, which of course would only allow 1000 characters because God Forbid I fill their database with an excessively chatty diatribe on their prodigious failings. So pithily did I expound upon my grievances. But as I was about to hit “Submit” I inadvertently hit the button on my trackball that maps to the browser’s “Back” button, and my input was duly erased. “Fuck you, shitty web app”, I said, “fuck you forever”, and I went on with my life.

Damn it, Hopkins… Your medical personnel are so awesome. Did you spend all your karma points on them and have none left over for your software development staff?

But maybe I’m the asshole. Or maybe, rather, I’m _also_ an asshole, I mean. I write imperfect software, and, really, how great are my channels for getting user experience gripes back to me? How good are the methodologies I have put in place at getting users of my systems to complain productively or, failing that, detecting automatically when the user experience has broken down? Maybe mediocre at best. I talk with users as much as I can manage. I encourage them to email the dev list to ask for support. I have logs of user actions, but, honestly, I don’t yet exploit those logs to the degree I ought. I think I earn some points for having a “Get Help” and “File A Bug” link on every page, but that only provides a mechanism to capture data from one’s most proactive and outspoken users. No doubt there exists a deep vein of ore to be tapped in the form of user visit logs that document specific actions and timestamps. How better to see, at scale, the way one’s user experience breaks down in the hands of real users?

Thus grows the “things I always want when I start a new software project” list, to include not just such things as “a good graphing package” and “a solid testing framework”, but also a means for mining the user experience quantitatively for the application’s shortcomings…

Spinning Up On DCS UH-1H

I got it into my head recently that now is as good a time as any to work on some bucket list items. One such item is flying.

To that end I hopped onto Steam and acquired DCS, i.e. Digital Combat Systems.  By “acquired” I mean “downloaded for free”.  DCS provides a highly realistic flight simulator environment with a variety of aircraft available as purchasable modules.  Fancying helicopters, I opted for UH-1H Huey.

I also wanted some reasonable gear with which to do the flying.  Looking around for a Joystick, the Saitek brand came up repeatedly.  The X-55 controller seemed to offer the latest revision of what I sought, but a bunch of reviews marked it as a step down from the X-52.  I ultimately acquired an X-52, doing so via eBay on account of availability issues, and rounded out my kit with a set of Saitek’s Pro Flight Combat Rudder pedals.  I find both pieces of hardware suitably pleasing.  One forum commenter, a professional helicopter pilot,  indicated that he liked the X-52 but actually preferred the X-45, though I could not find one for purchase at the time.  One of his reasons involved the centering spring on the X-52 that acts to bring the stick back to neutral, something that when piloting helicopters is not particularly desirable.  Various other comments mentioned the possibility of removing the centering spring, but I’ve not yet found the bravery for such a thing.

I have thus far spent a moderate amount of time in the training environments.  These consist of a mix of demonstration by an “instructor” and the opportunity to take the controls and work through a scripted session.  The scripting is not incredibly rich, but it’s decent, and will get you up and running.  I won’t profess any particular greatness, but I can pick up to a controlled hover, take off, land, and do a tolerable job of hitting targets that aren’t firing back at me.  In other words, I have enough experience to have gained a substantial amount of respect for the actual professionals that fly these things.

So far I have bumped up against a few bugs in the game that represented an in congruous lack of polish on an otherwise extremely sophisticated game.  Firstly, the weapons training script would jam at the step where it wanted you to take one of the door gunner positions, on account of the helicopter for the training mission not actually having any door guns installed.  I had to dig up a Youtube video that walked you around that bug by firing up the in-game mission editor. Hrm. After that I got stuck again not being able able to aim the door guns or use the flexible sight until a forum post yielded the bizarre fix of disabling and then re-enabling the “TrackIR” checkbox under the special options section. Bah. Having to do these things was a little bit ridiculous, but I managed.

There were other control-related challenges as well, though perhaps these were not the game’s fault, but nonetheless represented additional stumbling blocks.  One such problem involved the joystick not controlling pitch and roll if my Logitech G-27 Racing Wheel was also plugged into my computer.  Sigh.  And then there was the confusion of the rudder pedals defaulting to messing with my roll if I activated the toe brakes.  Good grief.

But…  Onward!  Having gotten through all of those initial challenges I’m really enjoying figuring things out (like how differently loaded and unloaded helicopters handle).  One of these days I think I shall take lessons in an actual helicopter.  I believe I’ll limit my experience with helicopter combat operations to the simulator, but it would be awesome to get up in the air eventually.  It’s pricey, at an apparent $300/hour for in-the-air time as per some preliminary Googling, but once I’ve done enough simulator time that I’m a little less concerned about killing myself I believe I’ll go for it.