Category Archives: Uncategorized

Thoughtless Development

Back when I was a boy, we ran servers on bare metal and we liked it.

And then there were containers.

And then there was AWS Lambda: “Run code without thinking about servers.”

Dwindling are the folks who might even know what “lsof”, “ps”, “top”, “nc”, “traceroute”, “df”, and “ldd” are, much less when to use them.

Actually, Lambda is pretty great, and I use it a lot, but damn does it make it easy to grow your attack surface and forget that you’ve done so.  And, at the end of the day, there are servers, and that reality has implications for availability and latency in whatever system you are building.

Meanwhile, infra-as-code faculties have proliferated, and many folks are using them, but the siren’s song of infra-as-clicks is quite strong, and the potential to create a non-repeatable mess in the cloud provider of your choice is great.

Be Strong.

But let’s get more concrete…

Today I was in the pantry at the office and on the TV I saw some talking head with a green-screen behind him on which three logos were painted in a repeating pattern: New England Patriots, Dunkin Donuts, and…  Zudy:”No Code Apps”.


Football, donuts, and faux enterprise software development.  LOL WUT.

Zudy’s marketing hype is intense: “No Code Enterprise Apps; Join The No-Code Evolution; Build game changing apps in days”.

Oh, FFS.  It was bad enough that we had to endure the No SQL shenanigans for about a decade before Make SQL Great Again got legs.  Now we’re going to pretend that we can develop apps without even thinking?

Spoiler alert: creating apps is easy; developing them over time once data has begun accumulating and people have begun broadly using them is hard.

We are witnessing a proliferation of shiny technologies that make it easy to bring new capabilities into existence, with the promise of old baggage being jettisoned, but we are not seeing commensurate faculties to manage and evolve these capabilities as we attempt to navigate a full system lifecycle.

I’m sorry, but the majority of the code written for a mature software system centers on logging, testing, data modeling, exception handling, security hardening, performance tuning, configuration management, release management, and inter-version compatibility.  This is the inescapable bread-and-butter engineering work of taking the kernel of an idea to a robust system that can handle day-to-day usage by an army of users in a way that is not completely maddening.

This is not new.  But the frequency with which products like this crop up is increasing.  We see examples of it in such offerings as SplunkPhantom, and NiFi.  And yet the well of uncomfortable truths tells us that “you’ll never find a programming language that frees you from the burden of clarifying your ideas”.

But, fear not…  If you get yourself wrapped around the axel, Zudy has an “AppFactory” and is more than happy to “Let Zudy’s experts build your apps for you.”  Congratulations.   You just built yourself a thicket of tech debt and hired some third rate contract programmers who will hold you hostage in perpetuity.

There are two kinds of enterprises: the kind who create and manage software deliberately and wittingly, and the kind who do so accidentally and unwittingly.  Which will you be?

Rage Against The Machine

At last week’s Strata Conference the buzzword exhibiting the highest frequency count appeared to be “Explainable” as prepended to “Artificial Intelligence”.  We have collectively transcended “can we make it work?” and landed squarely in “why did it make that decision?” territory.

In highly regulated industries the government applies a strong back pressure on non-explainable algorithmic decisions.  This serves as a check against runaway and impenetrable automation of decision making.  Yet clearly not all AI-driven industries that can exert an enormous impact on our lives find themselves subject to such controlling forces.  And from one country to another the degree of regulation for a given industry can vary greatly.

The UAE’s Daman gave an interesting talk on how they applied Natural Language Processing techniques to non-textual data in the healthcare claims adjudication space.  The strategy appeared to enjoy substantial and measurable success.  What creeped me out, though, was their seeming heavy reliance on customer complaints to act as the corrective force on falsely flagging claims as invalid.  The presenter offered the opinion that if a customer did not fight a claim rejection then the claim was probably invalid or unimportant anyway.

This feels like data scientists engaging in cost externalization to customers who exist in a fairly disadvantaged position and who must now fight back against a maddeningly opaque decision engine.  This appeared especially so in the case of Daman who apparently controls 80% of the health care market in the UAE (cited by one of the presenters as a reason why this particular data set was super cool to work on).

What force would stop such a company from taking the next logical step in profit optimization?  Auto-tune the rejection of valid claims to the sweet spot where statistically customers don’t fight it because getting their due does not justify the cost.

There has been much talk of how we must not allow the “Kill Decision” to fall into the hands of robots in warfare.  How easy it would be to make the same mistake in less sensational contexts.

Social Engineering

As an opportunistic hobby I will occasionally engineer my way into “illicit” access to my own stuff as reminder of how vulnerable I am to shenanigans.

Tonight I returned to my hotel room and found my key card unwilling to open my door. It was not that authorization had failed, but rather authentication, as neither the red nor green light came on. I reduced the theory space to a fried card by swiping it on someone else’s door which also gave no recognition of it. I suppose it could also have been that _all_ readers were dead, but that seemed unlikely as there was not a line of irate guests at the front desk as I passed it moments earlier. And I suppose it could have been awkward if that room’s occupants had showed up just as I swiped at their door, but #whatevs.

“I think my card is fried, room XXX”, I said, and handed it to the desk attendant. “Name?”, he asked. “Andrew”, I replied, giving as little information as possible, and not offering my ID, which he did not request. “Oh, yeah, totally dead. You put it next to a phone or something?”, he remarked. “Maybe. Not sure”, I replied noncommittally. He programmed up a new card and handed it over, no more questions asked.

I looked at the physical card afterward. There is no identifier imprinted on it.

So I am pretty sure all I need to go into any arbitrary room in this hotel is knowledge of someone’s name and a room card over which I have dragged a magnet.

The things a less scrupulous person could do with so little… Maybe snoop the guest ahead of you for their name and have access to a card that was “lost” from a previous visit? In you go. Maybe not worth the trouble to steal someone’s wallet. But maybe to leave a little Novichok behind?

Assigning Credit

I was listening to a The Motley Fool podcast this morning in which their product promotion segment referred to the Eero home WiFi system and provided a promo code of “fool” to get free shipping off of the Eero web site.  I have of late been having a crappy experience with my NetGear router and so thought I would give it a try.  I went to the web-site on my iPhone and…  it didn’t load.  I went on with my morning and a little bit later tried pulling it up on my laptop.

While there I decided to search for it on Amazon to see what its reviews were.  Lo and behold it was playing very positively.  And its list price was $50 cheaper.  And I could have free shipping with a guaranteed Tuesday 2 January delivery.

But it felt crappy to learn about the product from TMF and not have them get credit.  And I’ve been hearing increasingly unhappy stories about vendors getting bullied by Amazon over pricing.  So I went all the way to the check-out step on both Amazon and on Eero to  see what the delta in cost and experience would be.

To Amazon’s lower list price would be added $22 in taxes.  Either that was already in the Eero website price or it was not being taxed.  For shipping, Eero’s site would provide free one-day FedEx shipping (thanks to the promo code), but also said that “orders ship within 1-2 business days”, leaving with poor clarity on when the item would arrive.

So, with a $28 higher price, a worse shipping experience, an unclear product return workflow, and order history fragmentation, I couldn’t bring myself to buy directly from Eero.  If the total difference in experience had been just a small delta in price, say within $20, I would have probably on principle purchased directly through Eero, but the holistic Amazon experience was too superior to pass on, and their site reliability and review system pulled me into their gravity well on this purchase before Eero could close the deal directly.

Oops.  It’s actually even worse.  The 5% cash back my Amazon Prime Rewards Visa will give me almost entirely closes the effective pricing gap.

Ironic Fail: The TMF podcast had as one of its topics the runaway dominance of Amazon in the e-commerce space




FedEx Hell Week

A few months ago I sprung for a “Yoko” yoke by Virtual Fly.  I was happy enough with its construction that I decided to gift myself for Christmas a set of their “Ruddo” pedals.  I can report that they are of a similarly awesome quality.  Both are a substantive upgrade from the Saitek items I had previously, providing a tactile experience fairly authentic to a Cessna.

While I am happy they arrived in time for my year-end loafing, I am mildly perturbed not just by FedEx leaving them sitting on my door stoop on Wednesday, but the clear fakery in which they engaged to do this.  Both the yoke and pedals were shipped internationally and required a signature.  Naturally my schedule required that I pick up the yoke from the FedEx facility.  And it was perhaps folly on my part to order the pedals so close to Christmas that FedEx would be operating in #fuckit mode.  It seems as if they showed up at 1343, logged a delivery exception, filled out a door tag, then decided to leave it anyway at 1345 _and_ claim that I signed for it.  This is a little terrifying since on other occasions FedEx has mis-delivered my stuff to the local high school, which regrettably shares an identical street address modulo s/Court/Ave/;.  I can’t wait until the when they both mis-deliver an important package _and_ fake a signature.

It seems as if “another attempt will be made” meant “60 seconds from now”.


But, all’s well that ends well…  This time…

Drowning In Garbage

I love the convenience of Amazon.  Time is what I have in shortest supply.  Having a stream of the things that I need or want showing up on my door step at the click of a button is enormously valuable to me.  But I am nonetheless horrified at the growing ratio of packaging to product.  And I imagine that the last mile logistics companies shifting from a small number of large deliveries (to retail stores) to a large number of small deliveries (to end-consumer homes) does not help the per-util carbon footprint.  I found myself thus despondent when I received my ostensibly eco-friendly rechargeable batteries delivered by themselves in an ungodly amount of non-biodegradable packaging this evening.  It didn’t even register on my brain that there was a recycling label on the bag before it went into the trash and I suspect that even when this stuff is theoretically recyclable the vast majority of it goes into landfills.  Ugh.


Analog Device Automation

I’ve got three Marpac white noise generators distributed around my bedroom.  It is a mild nuisance to have to walk around the room and switch each one’s state at the boundary of a sleep session.  And so I acquired an Aeotec “Smart Switch” for each one of them and an Aeotec “Minimote” to operate them under the yoke of my growing Home Assistant driven Z-Wave network.

I was a little bit muddled by the Minimote’s setup until I realized that you can use it either as a primary or secondary Z-Wave controller and it was the latter I wanted which entailed pressing the “Learn” button as opposed to the “Include” button when joining it to the existing network.  The other configuration was to set it to “Scene” mode vs “Group” mode, the former giving you two distinct “scenes” per button with a short and long press.

I only made one configuration change to the Smart Switches out of the box.  Specifically, I configured parameter 81, “Configure the state of the LED”, to “When the state of the Switch changes, the LED will follow the status (on/off) of its load, but the LED will turn off after 5 seconds.”  This provides useful visual telemetry without leaving you with glowing orbs in the room you’d like to be dark for sleeping.

There were some other bumps along the way…

The switches had some identity crises at the outset.  I started with node id 14, added one that became 19, and added another that became 21.  14 then disappeared and, I think, came back as 22.  Now 14, 19, and 22 seem to be the healthy ones and 21 is AWOL.  Things seem to work now, but I was a bit perplexed for a while, and not sure what exactly happened.  I would probably have to have been tracking raw device identifiers to truly understand.  Maybe I can do that forensically later.

Initially I imagined just using a single “scene” on the Minimote and invoking the “switch.toggle” service, but this was not fault tolerant, the failure mode being that if a hiccup in command distribution prevented all devices from receiving a broadcast then a subsequent toggle command would cause a subset of the switches to be on and a subset off.  Ugh.  So I’m consuming both of the “scenes” of one of the buttons to do an explicit “switch.turn_on” or “switch.turn_off” command.

Lastly, occasionally, as captured in the below video (listen closely and you hear two of the three devices switch off immediately), there is a bit of delay between button-press and response, but generally it is immediate.  This has me developing the suspicion that I am getting some network congestion. I modified configuration parameter 111, “Group 1 Interval”, to 60 for all of my MultiSensor devices to get them to submit reports every minute instead of what appeared to be an hourly out-of-the-box configuration.  Now I am getting the reports I want but possibly congestion once per minute.  Maybe I need a lower reporting frequency and some staggering?  I also note that this upped frequency sabotages having set the bedroom-located MultiSensors’ parameter 81, “LED report”, to “Disable” to prevent obnoxious blinking in response to the motion sensor, because the LED will also blink whenever it sends its period report.  UGH.  I ended up using the l337 hack of putting electrical tape over the LED in those devices.

    name: "Smart Switches"
      - switch.aeotec_zw096_smart_switch_6_switch
      - switch.aeotec_zw096_smart_switch_6_switch_2
      - switch.aeotec_zw096_smart_switch_6_switch_4
  - alias: Start White Noise
      platform: event
      event_type: zwave.scene_activated
        entity_id: zwave.aeotec_dsa03202_minimote
        scene_id: 1
      service: switch.turn_on
      entity_id: group.smart_switches
  - alias: Stop White Noise
      platform: event
      event_type: zwave.scene_activated
        entity_id: zwave.aeotec_dsa03202_minimote
        scene_id: 2
      service: switch.turn_off
      entity_id: group.smart_switches