The Good, The Bad, And The Data

Firing up my phone a few moments ago to read something whilst my biological functions went through their morning boot-up procedures I found myself reading an NYT article on the facet of the TrumpCare debate centering on coverage for existing conditions, which was interesting enough in its own right, but I appreciated on so many levels the ad-placement that occurred within the piece, a not-so-chance occurrence whose irony in the context I found myself savoring.


I was imagining the various bullets that this Dean Kozlowski character might have included in his year-end performance self-appraisal write-up, all of which his bosses would have thought were great, but for which the general public would have a mix of reactions:

  1. fused public and private data sets to discover under-served markets
  2. established customer attrition patterns and enacted plans that improved retention
  3. deployed voice-stress analysis software in call center to triage active cases in real time
  4. determined precise pricing pain points for specific communities and raised premiums just below the threshold where they would drop their policy
  5. proactively detected and dropped high-risk policy holders
  6. established data driven protocols for refusing policies to high-risk applicants

As someone whose hat collection includes the “Data Engineer” one, I find myself by turns proud of and creeped out by the power of data, and wonder what lies ahead as we learn how to wield this power in a responsible and sustainable fashion.  The conversation has only just begun.


The Perfect Storm

Donald Trump _almost_ did not win the US presidency.  Almost.  And yet, even if he had lost, the forces that brought his candidacy right up to the brink would still have been extant. The world is proceeding through a phase change that promises economic violence and social unrest. Much of it is now inevitable because avoiding it would have required implausible clairvoyance regarding the coming storm and decades’ worth of preparation. The best we can do now is to emplace cushions to absorb the unavoidable and make course corrections that reflect the rapidly changing realities. The possibly unprecedented challenge takes shape in so much happening in a compressed timeframe.

Civilization is no stranger to disruptive phase changes that destroyed professions wholesale and entirely eliminated certain ways of life. This isn’t even the first time that we’ve been in real danger of having the world end in fire or ice. The present circumstances, rather, distinguish themselves by the breadth and rapidity of the coming changes that threaten to leave large swaths of the population without a producer role in the economy, without the tools to navigate the information-sphere effectively, and generally without stability or a sense of purpose.

While much of the bounty of what we enjoy in the world stems from globalization and automation, poor cognizance and management of the secondary effects of these things have yielded mounting negative consequences. To the extent that we have addressed these things it has been at the far ends of the spectrum, either denying their existence or importance on the one hand, or engaging in ineffective knee-jerk policies on the other. To accept the truth of things would be to acknowledge that they provide substantial material improvements to consumers in the short term, they jeopardize integration with the economy for all but the most elite and esoteric producers in the long run, and they come with a web of secondary and tertiary effects that impact everyone increasingly over time, with key areas including education, journalism, democracy, the environment, and the nature of warfare.

Consider for a moment the collection of jobs hanging very immediately in the balance and their corresponding contributions to total employment. According to the American Trucking Association, 3.5 million people are presently employed as professional truck drivers, and approximately another 5 million are employed in non-driving roles related to the industry. The US rail industry, meanwhile, accounts for approximately another quarter million US jobs, as does the cab driving profession. What happens when the self-driving car research of Google, Apple, Uber, and others comes to fruition in the coming decade? What will these people do?

The US retail sector, meanwhile, accounts for another 5 million jobs. What will be left of the retail sector once Amazon has finished its various automation efforts? Its warehouses are packed with robots that are leaving increasingly little for humans to do. It is making inroads into becoming its own shipping company. And, in a nod to the few remaining benefits of the brick-and-mortar experience, Amazon is now piloting fully automated stores where shoppers can walk out with the desired merchandise and an array of sensors determines their purchases in a completely non-disruptive fashion that requires no on-site employees.

And if that weren’t enough, there is even promising work being done in moving robots out of factories and warehouses and into the more delicate work of food preparation and nursing care.

Stack that on top of the countless unemployed or underemployed folks ejected from the manufacturing sector.

Now consider the voting bloc that these people will form when they are collectively unemployed in what is effectively the blink of an eye on political timescales.

And if you think you’re safe because you’re in some manner of white collar profession, tell that to the droves of financial services people being displaced by robo-brokers. How long can even the lofty medical profession of radiology last in the wake of systematic big data collection and machine learning? How long can any but the uppermost crust of IT professionals survive as cloud technologies mature to the point that organizations no longer operate their own infrastructures or employ the requisite support personnel?

To remain relevant as a professional in the coming economy will require a deeper knowledge stack than at any prior time. And, if that weren’t enough, the requisite stacks are undergoing a period of extreme volatility, making it all the more difficult to stay current for even the most able and motivated individuals. Keeping the general public adequately educated will require not simply an enormous investment in education but rather a fundamental re-tooling of our education system because the models of the past are unworkably inefficient and non-scalable.

And, perhaps even worse than the world’s education system’s inability to keep the general public gainfully employed is the sub-par job it is doing innoculating people against demagoguery and extremism in all its forms, ranging from ISIS to Trump. One of the nubby problems, of course, centers on our struggle to agree on what good education even looks like. In some cases, perhaps only violence is the answer, such as when dealing with people who think that women should not be educated. In others, perhaps only strategic concessions will work, as is the case with people who don’t believe in evolution, in which case the best bet may be to agree that we should teach neither religion nor evolution in public schools, instead focusing on the low-level fundamentals of language, logic, mathematics, philosophy, and scientific method, in the hopes people will stumble upon truth and reject falsehoods on their own.

Without effective public policy to ensure that the general public possesses solid educational grounding we are not going to solve any of our problems. We are in grave danger of having most people exist in increasingly insular mini-worlds that serve as echo chambers of their own pre-existing preferences. Humans are born with an innate tendency toward confirmation bias, something that once upon a time was an adaptive behavior in environments where dangers were very direct and very lethal, but now is a liability regularly exploited by those who wield information as a weapon and have a profit motive to do so.

And there is a viciously nasty cycle in all of this.

Twitter has left people with an attention span of 140 characters (“SAD!”). Google has fostered a belief that “research” consists of typing a search term into a web browser and reading the first three results. Facebook is selecting which of your friends’ posts to highlight based not on what will broaden your understanding of the world but rather on how long you’ll stay engaged with the site and clicking advertising. Real journalism outlets, meanwhile, organizations that pay professionals to do substantive research and write thoughful and balanced analyses, not only find themselves struggling to engage with a readership that can meaningfully digest their content, but worse still find themselves in a brutal pincer maneuver, on the one side being starved for cash as the big tech companies wring the lion’s share of advertising revenues by repackaging news products without properly compensating the creators of the content, and on the other side by sensationalist politicians who are openly hostile to anything that offers even a whiff of intellectualism or accountability.

As people find themselves bereft of a sense of purpose due to economic disengagement, subject increasingly to highly sensationalistic context-free information, and without the tools to either re-engage with the economy or critically question what they see, we find ourselves with a recipe for desperation, extremism, xenophobia, short-sightedness, and authoritarian rule. And we find ourselves in these circumstances when we can least afford it, at a time when we have the most powerful weapons of war, the most environmentally impactful industrial capabilities, the largest ever human population, the greatest density of sensors, data, automation, and telephony, and the greatest concentration of wealth and power in the hands of the few.

There is, meanwhile, a creeping non-democratization to our world on every axis of existence. Our wars will be fought by robots, not people. Our information gateways are on a trajectory to become ever smaller in number and greater in power, though we’ll be duped into thinking that we’re in a more democratic information age as these gateways preside over a growing information anarchy. Our food will be produced by an increasingly consolidated agricultural-industrial complex whose engineering prowess in creating addictive amalgams of fat, sugar and salt is matched only their ability to do it right under our noses as we tolerate the Krafts of the world doing things far more insideous than the Marlboros ever did.

And maybe most dangerous of all is the risk that we’ll stop understanding how our world works, we’ll stop being able to route around the failures of our automation, and the likelihood of failure will grow as system complexity reaches intractable levels. Humans are pretty inefficient when it comes to most tasks as compared to a specialized robot or software agent that has been honed over time. But they are remarkably fault tolerant and adaptable, properties we generally don’t yet see in the automation we have as yet created, and our tendency seems to be dangerously skewed toward increasing featurefulness as opposed to robustness.

To maintain a coherent and sane world we probably have to focus on a few key things:

  • Rate And Scope Control: Societies must invent and evolve or they will be destroyed by the ones that do. But they must also take great care not to allow their rate of change to outstrip their ability to digest those changes and reconfigure to a new way of being. We need to dream up ways to balance the imperative of progress with the ability to understand and handle the implications. This will be a difficult balance to strike. We are in thrall to our technology. The fruits of progress are immediately obvious to us and the costs are much more difficult to see and understand. I don’t know, for instance, that we should have a tax on robots, but it’s not the craziest idea I’ve ever heard, and if that isn’t the right kind of solution for a certain kind of problem, then we at least need to be tossing around ideas like it, and we should be thinking about things like this across a variety of domains.
  • Charging Full Price: It is currently far too easy for powerful entities to reap enormous gains while misappropriating the labors of others, externalizing the negative impacts of their production processes, and free-riding on basic infrastructure, none of which is sustainable in the long run. This can take such diverse forms as technology companies looting and undermining journalism, energy companies being subsidized by military spending, mining and manufacturing firms not bearing the costs of the environmental damage they do, transportation systems charging prices that only reflect fuel and labor costs as opposed to carbon emissions, pharmaceutical companies benefiting from basic research while not adequately paying for it, food distributors selling products that are creating a public health crisis, or corporations offshoring their profits while riding atop local infrastructure. Mind you, corporations are only behaving rationally within the system of government we have created, and the price of all the benefits of a free market society is that corporations will do everything they can to maximize profits. We shouldn’t expect them to behave differently within the current system. We should change the system.
  • Investment In The Basics: The public should on the one hand be very leery of hand-outs and yet on the other hand seriously consider how to reasonably ensure that a demonstrably motivated individual has the tools to be an effective participant in society. Done under the auspices of the old regimes such a thing would be impossibly expensive. But done with the power of the technology that is precipitating the disruptions it may actually be possible.
    • With education, we must dial down the absurdly expensive direct education model, instead moving to a hybrid approach that includes self-directed computer-based study, one-on-one tutoring, group lectures, and the automation to dynamically structure the experience to the individual.
    • With medicine, we need to implement and deploy pervasive real-time monitoring of health metrics and feed these to analytics that can catch problems early enough that simple and inexpensive interventions are the norm instead of staggeringly expensive crisis care, we need to decouple healthcare plans from employers, and we need extreme transparency in the costing models for medicine so that people can consume it rationally like any other service.
    • Foundational in turn to both of these things is ensuring proper basic nutrition for everyone so that they have the energy and focus to meaningfully participate in schools and workplaces instead of being chronically ill and consequently excessively consuming medical services while being unproductive. We should accomplish this through a combination of greater transparency in product labeling, more basic research into the impacts of nutrition on health, and intelligently crafted subsidies that ensure reasonably healthy food is affordable for everyone instead of the idiotic subisides we have right now for the worst food products (I’m looking at you, corn).
    • And foundational to all of this is a surge on ensuring that everyone has access to reasonable quality basic infrastructure in the realms of transportation, energy, clean water, and communications network access, things that will become easier as we roll out self-driving vehicles, create more smart devices, emplace more fiber and cell towers, and intelligently instrument all of the processes so we can troubleshoot and optimize them.

These are the unsexy but really important foundational matters we need to address to resolve our present predicament but we’re distracted by an administration that came to power by fanning the flames of fear and hatred in an appeal to our worst selves instead encouraging us to roll up our sleeves, cast aside extremism, and get work creating a sustainable future in which everyone can participate.

Airport Security Workflow Overhaul Fail

The Hartsfield-Jackson (ATL) airport recently emplaced a rejiggered passenger-and-luggage screening system.  Superficially the individual changes appear clever, but in the aggregate they create a more chaotic and stressful experience for travelers, seemingly without accomplishing the intended benefits.

Having cleared the snaking line and ID validation, travelers will find that they can approach the conveyor belt at one of several parallel stations, each of which offers a private staging area where you can prep your items.  Another conveyor belt, meanwhile, feeds empty trays below the upper one for you to retrieve.  The rest of the process will appear familiar.  Sort of.  At least in theory.

These changes attempt to address what can be two of the more vexing and delay-inducing problems, firstly the exhaustion of available bins, and secondly the inability of a traveler to effectively prep their items for the belt before occupying the belt in a way that prevents its use by other passengers.

The first issue becomes evident as soon as you attempt to retrieve a tray.  The belt that serves this function starts up unpredictably and may do so when you are in the midst of retrieving a tray, as happened to me, causing the tray to pitch and roll substantially and thereby get wedged.  I could also imagine, though it did not happen to me, someone experiencing a mild to moderate hand pinching as a consequence.

The second issue manifests as you attempt to prep your items.  The prep station only accommodates a single tray at a time.  Meanwhile, you compete with several other passengers to push items from the prep station onto the shared belt in an uncoordinated fashion.  If you have more than one tray’s worth of stuff, your items will inevitably get separated.  If the shared belt engages in the middle of your transferring an item to it, your item may get wedged.  I ended up with three trays (roller-board, bag/shoes/belt/jacket, two side-by-side laptops) that ended up uncomfortably far apart.  It was quite unsettling.

The third issue manifests not as the result of a change but rather the lack thereof.  I think the new arrangement does actually move passengers along the earlier stages of processing more efficiently.  Unfortunately, there were no changes apparent in the baggage collection phase of the architecture, resulting in even more chaos.   TSA personnel were admonishing travelers not to re-dress/re-pack at the belt.  People ended up awkwardly putting their luggage onto the floor in crowded conditions.  I believe I witnessed an especially good illustration of the problems as I arrived in a burst of travelers at the checkpoint that seemed to have followed a lull.

While the arrangement certainly makes some strides on optimizing for throughput, it does so in a way that leaves you feeling even more like harried cattle than the previous arrangement did, and worse still the optimizations appear to have been applied only locally in a way that did not calibrate the system to avoid bottlenecks.

Perhaps the most unforgivable failure in all of this was the seeming lack of instrumentation data or readily apparent customer feedback mechanisms.  I hope that this is being done as a mere pilot project at only a single of the airport’s several checkpoints.  But if it is then the validation of the pilot project’s effectiveness would not seem to be playing out in a meaningful manner.  I chatted up a couple of my fellow travelers at the end of the process to make sure the exasperation I experienced was not unique to me.  It wasn’t.  Sadly these two conversations appeared to be the closest thing to a survey being conducted on the experience.

When I mentioned to another traveler the forlorn hope that this was just a pilot project and that the airport would learn what a disaster it was before rolling it out to all of the checkpoints, he remarked “It’s the government, so I doubt it.”. I’m afraid that that my intuition says that he is right.  This could be done a whole lot better but seemingly the right people are not designing the system or overseeing its implementation.

Death To System Initiated Pop-Up Windows

Pop-up windows that manifest without a user-initiating action and grab user-input focus do not merely present a severe nuisance that can manifest in unintended system actions but also by consequence increase the attack surface by providing an avenue to subvert sandboxing and configuration management faculties.  Modern operating systems should phase out this faculty and replace it with a more friendly and secure one.

In lieu of unpredictable pop-ups, operating systems should gather user input requests from system processes and applications into a shared queue of action requests, require a GUI context-switch for users to operate on them, and provide an inobtrusive alert faculty that announces the presence of input requests without hijacking input focus.

I have been thinking of this for years but only just now found myself angry (and perhaps time-rich) enough to write something when I was typing something into Safari on my Mac Air when Flux contrived to pop an update request dialog box that my in-flight fingers accepted without my having any opportunity to know what was happening until it was too late.  Decidedly not cool…

Microsoft Windows has made some useful inroads into security by requiring more explicit user decisions when applications attempt to gain execution, but even that falls somewhat short by manifesting as a focus-grabbing pop-up that could fall prey to the above-described problems.

We can do better.

AdTech Integration Fail

Every time I swipe my American Express Blue Cash Preferred card at the pump of a nearby gas station the in-pump TV runs a commercial that encourages me to apply for that very same credit card.  I wonder whether the swiped credit card has absolutely zero input into the displayed ads or if the decision logic is totally blowing an opportunity to perform meaningful targeting.  Certainly AmEx is not getting the best of their advertising dollars in this exchange.

It’s Package Managers All The Way Down

I just wanted to run the angular-cli on a recently built Fedora 24 box.  Unfortunately getting it installed was an exercise in coaxing along a poorly coordinated collection of package management software.  Ultimately what works is…

sudo dnf install git
sudo dnf install fedora-repos-rawhide
sudo dnf install --enablerepo rawhide nodejs libuv --best --allowerasing
sudo npm install -g angular-cli

… which provides the grounding for a successful run of…

ng new angular2-fundamentals

… but getting there meant slogging through angular-cli first blowing up because it assumed that Git would already be installed and then blowing up again because the version of NodeJS installed to support NPM’s own dependency was not adequate for angular-cli’s requirements.  Ugh.  This is the manner of hazing inflicted on you to make sure that you really want to use the software.  And even now, having superficially seemed to succeed, I’m not even sure I really did, as the install of angular-cli itself threw some “errors” that maybe were just warnings.  Who knows…