Distributed System Fail

The other day when returning to my hotel room from near-100F heat I found myself willing to spring the $2.50 the vending machine wanted for a Vitaminwater.  I lacked the paper or metal currency to complete the transaction but thankfully the machine accepted plastic.  Or at least appeared to.  Except someone botched designing fault tolerance into the system.

I swiped my Visa but after the system hung with the text “authorizing transaction” for a minute it barfed with “communication error”.  So I tried my AmEx instead in the vain hope that the networking failure was not at the last hop.  Alas, no dice.  Here I was, willing to pay retail++ for a Vitaminwater but the system did not want to take my money.  Because it couldn’t talk to the mothership and maybe I wasn’t good for the $2.50.

Now, I’m pretty sure that when a credit card authorization service gets interrogated its implementation is to default to authorizing the transaction (as long as it is reasonably small) if a non-confirmation of the transaction does not come back within a few seconds because the wheels of commerce as a whole need to keep turning regardless of whether some pissant network service feels like being helpful at a given moment.  The vending machine’s own architecture would have done well to draw inspiration from the larger system’s design choices.

It seems entirely reasonable that the vending machine instead might have handled an inability to reach the authorization service by letting the sale complete and caching the transaction for later transmittal.  To deal with an attacker it might cap a given card to a single cached transaction so that someone cannot simply disconnect the machine and then use a single bad card to clean out the machine without paying.  And maybe to thwart a really determined attacker who is carrying a wad of bad cards and for some reason is willing to squander one card per stolen beverage you could cap the total number of system-wide cached transactions.  And, hey, don’t a lot of vending machines these days come equipped with cameras that could be retroactively used to nail a really determined attacker (as well as deter shenanigans in the first place)?  Surely such a simple arrangement would have sufficiently contained the attendant threats to make the operator willing to complete a transaction that would net him $2 at risk.

To be fair, though, I have no experience in the vending machine business, except that sometimes I just want a cold beverage but the machine doesn’t want to play.

Leave a Reply