Building The Foundations Of Oppression

From the Financial Times…

The UK, France and Italy have promised to hold technology companies to account if they fail to block online terrorist or extremist content, with Emmanuel Macron, the French president, promising to “name and shame” companies that do not take robust action.

The meeting came after Mrs May and Donald Trump, the US president, discussed the need for internet companies to behave in a socially responsible way, move quickly to remove terrorist material and to stop it from appearing in the first place.

Mrs May challenged the internet companies to develop technology that would stop “evil material” appearing on the web or remove it within two hours of it appearing online. “Industry needs to go further and faster in automating the detection and removal of terrorist content online, and developing technological solutions which prevent it being uploaded in the first place,” she said.

Would this look any different, in shape if not actual configuration, from the machinery already used by totalitarian regimes to deploy pervasive and aggressive thought police against their own populations to prune “undesirable” content from the Internet?  Why should we trust all future leaders who would inherit such capabilities to use it responsibly?  How do we define what constitutes responsible use?  Why should we even trust today’s leaders when in the US they make a regular thing of demanding “apologies” for perceived insults and suppressing thought by not just butchering the funding for basic scientific research at government agencies but also redacting and possibly even destroying official records that do not fit their agenda?

If cynical realpolitik rooted in a desire to look tough on terrorism is not the base motivation, or maybe an actual desire to take things in a totalitarian direction, then heading down this path is a horrifying exercise in double-think and failure to consider secondary and tertiary effects.  The EU is certainly trying to have it both ways, on the one hand hammering Google and FaceBook for wielding their monopoly powers to unfair commercial advantage, and on the other hand demanding that they weaponize their Internet gateway status on behalf of the government.

Or maybe this isn’t double-think at all.  The most terrifying explanation is that governments are entering a negotiation with these tech behemoths with the idea of sanctioning their exploitation of their monopolies for commercial purposes if only they will consent to having their filtering powers bent to government will, a Faustian bargain if ever there were one.

God save us from a world where skynet is “automating the detection and removal of terrorist content online” and “prevent[ing] it [from] being uploaded in the first place”.  Could that possibly read more creepily?  How much ignorance of history, technology, and current events does it take to suggest such a thing seriously?  Or do they know exactly how this will end?

Deflection, Ideology, and Disaster: The Worst Of All Worlds

With the epic Hurricane Irma meting out destruction in Florida, Texans only just beginning to unwind the damage of Harvey, the memory of Sandy still stinging for New Yorkers, and Katrina a shameful blight on our recent history, the world is actively pressure testing our social contracts in ways that illustrate serious rifts at best and magical thinking at worst.

Meanwhile, there exists a natural temptation to broach climate change matters in the wake of a natural disaster such as Irma, but the reality is that exploring such a complex topic in the context of a single story would be irresponsible, and is best left to a numerical analysis of trends across a statistically significant collection of data points.

And yet there is a confluence of several factors that should have us scared independent of the culpability of humans in climate change on our planet:

  1. population density, especially in vulnerable coastal areas, is way up
  2. the transportation infrastructure around population centers is taxed to its limits
  3. population centers cannot operate self-sufficiently beyond a trivial time period
  4. population centers depend on fragile infrastructure to satisfy their needs
  5. a shocking proportion of people living in the most vulnerable areas are uninsured
  6. the terraforming around population centers eliminates natural weather shielding
  7. tight political and market cycles make engineering for fault tolerance difficult

The consequences of these realities could not be more palpable than for those struggling to find gasoline, water, and lodging as they flee the affected areas via jammed roads.  And the commentary from various thought leaders staking out their positions could not be less helpful, with government officials lamenting the cruelty of price gouging within the local economies and market religionists saying that allowing market-based pricing is all that is required to best allocate scarce resources.  To be blunt, the former are deflecting responsibility, the latter are making context-unaware arguments, and neither are meaningfully engaging with the reality on the ground.

When a big box store leaves pricing structures untouched in the run-up to a natural disaster, it is not doing so altruistically.  Rather, it is prioritizing optics over all else, and as consequence we see footage of hoarders carrying cartloads of regular-priced bottled water out of the stores.  Some of these purchasers are just panicked end-consumers, likely buying more product than they strictly need, doing so at the expense of other end-consumers who will find empty shelves.  Others are opportunistic re-distributors engaged in arbitrage between the storefront and the ultimate consumers who will pay huge mark-ups after finding the stores picked clean.  Both cause churn and uncertainty during a demand shock and do nothing to synchronize the supply-side with the local population’s needs.  The demand signal does not make its way usefully upstream.

But the crux of the folly of those making market-based arguments could be summed up in the one word that they are ignoring: latency.  Free markets, judiciously managed, are a great way to equilibrate the needs of producers and consumers.  But that equilibrium takes time: time for the demand signal to reach producers and time for the goods to flow from where they are produced to where they are needed.  This reality is perhaps well suited to dealing with the aftermath of a disaster, where the dramatically inflated local rates for professionals in the construction industry can serve as a nationwide draw until the crisis resolves, convincing, say, a plumber from California to spend a year living in Texas, but less well suited to sorting out a situation where the difference between life and death may be measured in hours, every manner of infrastructure is already completely overwhelmed, and the last thing affected regions need is a poorly managed inflow of people and goods just as officials are desperately reconfiguring highways to be one-way out of the area.  Allowing prices to swing during a crisis might help with the rationing of existing local supplies, but in a way that can feel capricious and cruel, and that does nothing to ameliorate the root cause of the misery, a supply shortage.

The 2008 financial crisis taught us some really painful lessons about what it means for market players to be too big to fail in the realm of banking.  We ultimately concluded that we still mostly want market forces steering things but that we also want certain safeguards in place to prevent cascading failures from spiraling out of control.  So-called “stress tests” became part of the national discourse.  We perhaps ought be drawing similar conclusions pertaining to distribution networks around our preparedness for disasters in the physical realm, natural or man-made, because when the proverbial excrement hits the fan things will be happening way too fast if we have not prepared and tested, doing so with a thoughtful blend of market forces and government planning and validation.

Unfortunately, to the extent that there is currently any interaction between government and market entities on such matters, it is just to have the former threaten criminal prosecution around “price gouging” for which there is no clear definition, and the savvy players among the latter group to take a hands-off approach that catalyzes situational anarchy.  With a continuation of this approach in the face of increasingly extreme environmental conditions we can only expect worsening outcomes.

Acquiring resources quickly in an emergency can be fantastically expensive and dangerously unreliable.  Helicopters to send in relief ain’t cheap or highly available.  Preparing proactively for an emergency, meanwhile, is not without its costs, and entails less resourcing for fun things during sunny days, which requires a certain degree of discipline.  Locating warehouses full of critical supplies near population centers means ponying up for the use of scarce real estate and maintaining inventory every day as a form of insurance payment.  Somebody always has to pay.  The main questions are who and when which will ultimately govern how much.

The situation we’ve currently got is one in which politicians engage in grandstanding and face-saving, established wholesalers and retailers are by and large helpless or indifferent, opportunistic profiteers make a quick buck without improving the ecosystem, and the masses suffer.  By the time a case of water is going for $100 we have really screwed up.  We could do a lot better in future situations by taking a more proactive stance.

Imagine a public-private partnership that harnesses the strengths of each sector by doing  things like the following:

  1. incentivize private sector distributors to maintain larger warehousing and distribution footprints in the vicinity of disaster prone areas to absorb demand shocks
  2. incentivize disaster zone residents to leave their homes in a more proactive and orderly fashion to smooth the load shocks on transportation infrastructure
  3. perform regular stress tests that posit certain scenarios, query the present availability of resources, simulate the outcomes, and phase in performance penalties
  4. establish data anonymization and aggregation procedures that allow for the fusion of public and private data sets to both predict and manage the play-out of disasters
  5. begin pulling the levers that will require those who choose to live in high-risk areas to bear fuller costs for their decisions so as to encourage rational risk-taking, to include meaningfully enforced requirements on insurance that ultimately cause either changes in behavior or fair passing-along of costs

Consider one of the specific things that happens during mass exoduses from disaster areas, specifically that highway toll collection halts and traffic streams through unmonitored.  That’s pretty reasonable given the hand the government agencies have at the moment.  But imagine what we could do if over time we completely phased out the acceptance of cash tolls, had the corresponding expectation that every vehicle transiting a toll station could be tracked to an owner, and implemented some manner of “toll” that incentivized people to leave more proactively.  This wouldn’t necessarily be easy, because the people most likely to procrastinate are probably the ones least able to take time off from work, but with some cleverness it could be done, perhaps using a progressively-structured income-linked tax credit as well as emplacing laws that prevent retribution by employers against employees who take advantage of the arrangement.  The trade-off relative to having to perform expensive and risky rescue and relief operations in disaster areas would likely be worth it.

It is probably not particularly controversial to say that $100 cases of water in disaster zones is icky and heart-breaking.  Making sure that those kind of things are less likely to happen in the future is going to take some serious thinking, hard work, and proactive sacrifice.  Harsher times are coming.  For now, the system we’ve got is the system we’ve got, and people need help.



The Good, The Bad, And The Data

Firing up my phone a few moments ago to read something whilst my biological functions went through their morning boot-up procedures I found myself reading an NYT article on the facet of the TrumpCare debate centering on coverage for existing conditions, which was interesting enough in its own right, but I appreciated on so many levels the ad-placement that occurred within the piece, a not-so-chance occurrence whose irony in the context I found myself savoring.


I was imagining the various bullets that this Dean Kozlowski character might have included in his year-end performance self-appraisal write-up, all of which his bosses would have thought were great, but for which the general public would have a mix of reactions:

  1. fused public and private data sets to discover under-served markets
  2. established customer attrition patterns and enacted plans that improved retention
  3. deployed voice-stress analysis software in call center to triage active cases in real time
  4. determined precise pricing pain points for specific communities and raised premiums just below the threshold where they would drop their policy
  5. proactively detected and dropped high-risk policy holders
  6. established data driven protocols for refusing policies to high-risk applicants

As someone whose hat collection includes the “Data Engineer” one, I find myself by turns proud of and creeped out by the power of data, and wonder what lies ahead as we learn how to wield this power in a responsible and sustainable fashion.  The conversation has only just begun.


The Perfect Storm

Donald Trump _almost_ did not win the US presidency.  Almost.  And yet, even if he had lost, the forces that brought his candidacy right up to the brink would still have been extant. The world is proceeding through a phase change that promises economic violence and social unrest. Much of it is now inevitable because avoiding it would have required implausible clairvoyance regarding the coming storm and decades’ worth of preparation. The best we can do now is to emplace cushions to absorb the unavoidable and make course corrections that reflect the rapidly changing realities. The possibly unprecedented challenge takes shape in so much happening in a compressed timeframe.

Civilization is no stranger to disruptive phase changes that destroyed professions wholesale and entirely eliminated certain ways of life. This isn’t even the first time that we’ve been in real danger of having the world end in fire or ice. The present circumstances, rather, distinguish themselves by the breadth and rapidity of the coming changes that threaten to leave large swaths of the population without a producer role in the economy, without the tools to navigate the information-sphere effectively, and generally without stability or a sense of purpose.

While much of the bounty of what we enjoy in the world stems from globalization and automation, poor cognizance and management of the secondary effects of these things have yielded mounting negative consequences. To the extent that we have addressed these things it has been at the far ends of the spectrum, either denying their existence or importance on the one hand, or engaging in ineffective knee-jerk policies on the other. To accept the truth of things would be to acknowledge that they provide substantial material improvements to consumers in the short term, they jeopardize integration with the economy for all but the most elite and esoteric producers in the long run, and they come with a web of secondary and tertiary effects that impact everyone increasingly over time, with key areas including education, journalism, democracy, the environment, and the nature of warfare.

Consider for a moment the collection of jobs hanging very immediately in the balance and their corresponding contributions to total employment. According to the American Trucking Association, 3.5 million people are presently employed as professional truck drivers, and approximately another 5 million are employed in non-driving roles related to the industry. The US rail industry, meanwhile, accounts for approximately another quarter million US jobs, as does the cab driving profession. What happens when the self-driving car research of Google, Apple, Uber, and others comes to fruition in the coming decade? What will these people do?

The US retail sector, meanwhile, accounts for another 5 million jobs. What will be left of the retail sector once Amazon has finished its various automation efforts? Its warehouses are packed with robots that are leaving increasingly little for humans to do. It is making inroads into becoming its own shipping company. And, in a nod to the few remaining benefits of the brick-and-mortar experience, Amazon is now piloting fully automated stores where shoppers can walk out with the desired merchandise and an array of sensors determines their purchases in a completely non-disruptive fashion that requires no on-site employees.

And if that weren’t enough, there is even promising work being done in moving robots out of factories and warehouses and into the more delicate work of food preparation and nursing care.

Stack that on top of the countless unemployed or underemployed folks ejected from the manufacturing sector.

Now consider the voting bloc that these people will form when they are collectively unemployed in what is effectively the blink of an eye on political timescales.

And if you think you’re safe because you’re in some manner of white collar profession, tell that to the droves of financial services people being displaced by robo-brokers. How long can even the lofty medical profession of radiology last in the wake of systematic big data collection and machine learning? How long can any but the uppermost crust of IT professionals survive as cloud technologies mature to the point that organizations no longer operate their own infrastructures or employ the requisite support personnel?

To remain relevant as a professional in the coming economy will require a deeper knowledge stack than at any prior time. And, if that weren’t enough, the requisite stacks are undergoing a period of extreme volatility, making it all the more difficult to stay current for even the most able and motivated individuals. Keeping the general public adequately educated will require not simply an enormous investment in education but rather a fundamental re-tooling of our education system because the models of the past are unworkably inefficient and non-scalable.

And, perhaps even worse than the world’s education system’s inability to keep the general public gainfully employed is the sub-par job it is doing innoculating people against demagoguery and extremism in all its forms, ranging from ISIS to Trump. One of the nubby problems, of course, centers on our struggle to agree on what good education even looks like. In some cases, perhaps only violence is the answer, such as when dealing with people who think that women should not be educated. In others, perhaps only strategic concessions will work, as is the case with people who don’t believe in evolution, in which case the best bet may be to agree that we should teach neither religion nor evolution in public schools, instead focusing on the low-level fundamentals of language, logic, mathematics, philosophy, and scientific method, in the hopes people will stumble upon truth and reject falsehoods on their own.

Without effective public policy to ensure that the general public possesses solid educational grounding we are not going to solve any of our problems. We are in grave danger of having most people exist in increasingly insular mini-worlds that serve as echo chambers of their own pre-existing preferences. Humans are born with an innate tendency toward confirmation bias, something that once upon a time was an adaptive behavior in environments where dangers were very direct and very lethal, but now is a liability regularly exploited by those who wield information as a weapon and have a profit motive to do so.

And there is a viciously nasty cycle in all of this.

Twitter has left people with an attention span of 140 characters (“SAD!”). Google has fostered a belief that “research” consists of typing a search term into a web browser and reading the first three results. Facebook is selecting which of your friends’ posts to highlight based not on what will broaden your understanding of the world but rather on how long you’ll stay engaged with the site and clicking advertising. Real journalism outlets, meanwhile, organizations that pay professionals to do substantive research and write thoughful and balanced analyses, not only find themselves struggling to engage with a readership that can meaningfully digest their content, but worse still find themselves in a brutal pincer maneuver, on the one side being starved for cash as the big tech companies wring the lion’s share of advertising revenues by repackaging news products without properly compensating the creators of the content, and on the other side by sensationalist politicians who are openly hostile to anything that offers even a whiff of intellectualism or accountability.

As people find themselves bereft of a sense of purpose due to economic disengagement, subject increasingly to highly sensationalistic context-free information, and without the tools to either re-engage with the economy or critically question what they see, we find ourselves with a recipe for desperation, extremism, xenophobia, short-sightedness, and authoritarian rule. And we find ourselves in these circumstances when we can least afford it, at a time when we have the most powerful weapons of war, the most environmentally impactful industrial capabilities, the largest ever human population, the greatest density of sensors, data, automation, and telephony, and the greatest concentration of wealth and power in the hands of the few.

There is, meanwhile, a creeping non-democratization to our world on every axis of existence. Our wars will be fought by robots, not people. Our information gateways are on a trajectory to become ever smaller in number and greater in power, though we’ll be duped into thinking that we’re in a more democratic information age as these gateways preside over a growing information anarchy. Our food will be produced by an increasingly consolidated agricultural-industrial complex whose engineering prowess in creating addictive amalgams of fat, sugar and salt is matched only their ability to do it right under our noses as we tolerate the Krafts of the world doing things far more insideous than the Marlboros ever did.

And maybe most dangerous of all is the risk that we’ll stop understanding how our world works, we’ll stop being able to route around the failures of our automation, and the likelihood of failure will grow as system complexity reaches intractable levels. Humans are pretty inefficient when it comes to most tasks as compared to a specialized robot or software agent that has been honed over time. But they are remarkably fault tolerant and adaptable, properties we generally don’t yet see in the automation we have as yet created, and our tendency seems to be dangerously skewed toward increasing featurefulness as opposed to robustness.

To maintain a coherent and sane world we probably have to focus on a few key things:

  • Rate And Scope Control: Societies must invent and evolve or they will be destroyed by the ones that do. But they must also take great care not to allow their rate of change to outstrip their ability to digest those changes and reconfigure to a new way of being. We need to dream up ways to balance the imperative of progress with the ability to understand and handle the implications. This will be a difficult balance to strike. We are in thrall to our technology. The fruits of progress are immediately obvious to us and the costs are much more difficult to see and understand. I don’t know, for instance, that we should have a tax on robots, but it’s not the craziest idea I’ve ever heard, and if that isn’t the right kind of solution for a certain kind of problem, then we at least need to be tossing around ideas like it, and we should be thinking about things like this across a variety of domains.
  • Charging Full Price: It is currently far too easy for powerful entities to reap enormous gains while misappropriating the labors of others, externalizing the negative impacts of their production processes, and free-riding on basic infrastructure, none of which is sustainable in the long run. This can take such diverse forms as technology companies looting and undermining journalism, energy companies being subsidized by military spending, mining and manufacturing firms not bearing the costs of the environmental damage they do, transportation systems charging prices that only reflect fuel and labor costs as opposed to carbon emissions, pharmaceutical companies benefiting from basic research while not adequately paying for it, food distributors selling products that are creating a public health crisis, or corporations offshoring their profits while riding atop local infrastructure. Mind you, corporations are only behaving rationally within the system of government we have created, and the price of all the benefits of a free market society is that corporations will do everything they can to maximize profits. We shouldn’t expect them to behave differently within the current system. We should change the system.
  • Investment In The Basics: The public should on the one hand be very leery of hand-outs and yet on the other hand seriously consider how to reasonably ensure that a demonstrably motivated individual has the tools to be an effective participant in society. Done under the auspices of the old regimes such a thing would be impossibly expensive. But done with the power of the technology that is precipitating the disruptions it may actually be possible.
    • With education, we must dial down the absurdly expensive direct education model, instead moving to a hybrid approach that includes self-directed computer-based study, one-on-one tutoring, group lectures, and the automation to dynamically structure the experience to the individual.
    • With medicine, we need to implement and deploy pervasive real-time monitoring of health metrics and feed these to analytics that can catch problems early enough that simple and inexpensive interventions are the norm instead of staggeringly expensive crisis care, we need to decouple healthcare plans from employers, and we need extreme transparency in the costing models for medicine so that people can consume it rationally like any other service.
    • Foundational in turn to both of these things is ensuring proper basic nutrition for everyone so that they have the energy and focus to meaningfully participate in schools and workplaces instead of being chronically ill and consequently excessively consuming medical services while being unproductive. We should accomplish this through a combination of greater transparency in product labeling, more basic research into the impacts of nutrition on health, and intelligently crafted subsidies that ensure reasonably healthy food is affordable for everyone instead of the idiotic subisides we have right now for the worst food products (I’m looking at you, corn).
    • And foundational to all of this is a surge on ensuring that everyone has access to reasonable quality basic infrastructure in the realms of transportation, energy, clean water, and communications network access, things that will become easier as we roll out self-driving vehicles, create more smart devices, emplace more fiber and cell towers, and intelligently instrument all of the processes so we can troubleshoot and optimize them.

These are the unsexy but really important foundational matters we need to address to resolve our present predicament but we’re distracted by an administration that came to power by fanning the flames of fear and hatred in an appeal to our worst selves instead encouraging us to roll up our sleeves, cast aside extremism, and get work creating a sustainable future in which everyone can participate.

Airport Security Workflow Overhaul Fail

The Hartsfield-Jackson (ATL) airport recently emplaced a rejiggered passenger-and-luggage screening system.  Superficially the individual changes appear clever, but in the aggregate they create a more chaotic and stressful experience for travelers, seemingly without accomplishing the intended benefits.

Having cleared the snaking line and ID validation, travelers will find that they can approach the conveyor belt at one of several parallel stations, each of which offers a private staging area where you can prep your items.  Another conveyor belt, meanwhile, feeds empty trays below the upper one for you to retrieve.  The rest of the process will appear familiar.  Sort of.  At least in theory.

These changes attempt to address what can be two of the more vexing and delay-inducing problems, firstly the exhaustion of available bins, and secondly the inability of a traveler to effectively prep their items for the belt before occupying the belt in a way that prevents its use by other passengers.

The first issue becomes evident as soon as you attempt to retrieve a tray.  The belt that serves this function starts up unpredictably and may do so when you are in the midst of retrieving a tray, as happened to me, causing the tray to pitch and roll substantially and thereby get wedged.  I could also imagine, though it did not happen to me, someone experiencing a mild to moderate hand pinching as a consequence.

The second issue manifests as you attempt to prep your items.  The prep station only accommodates a single tray at a time.  Meanwhile, you compete with several other passengers to push items from the prep station onto the shared belt in an uncoordinated fashion.  If you have more than one tray’s worth of stuff, your items will inevitably get separated.  If the shared belt engages in the middle of your transferring an item to it, your item may get wedged.  I ended up with three trays (roller-board, bag/shoes/belt/jacket, two side-by-side laptops) that ended up uncomfortably far apart.  It was quite unsettling.

The third issue manifests not as the result of a change but rather the lack thereof.  I think the new arrangement does actually move passengers along the earlier stages of processing more efficiently.  Unfortunately, there were no changes apparent in the baggage collection phase of the architecture, resulting in even more chaos.   TSA personnel were admonishing travelers not to re-dress/re-pack at the belt.  People ended up awkwardly putting their luggage onto the floor in crowded conditions.  I believe I witnessed an especially good illustration of the problems as I arrived in a burst of travelers at the checkpoint that seemed to have followed a lull.

While the arrangement certainly makes some strides on optimizing for throughput, it does so in a way that leaves you feeling even more like harried cattle than the previous arrangement did, and worse still the optimizations appear to have been applied only locally in a way that did not calibrate the system to avoid bottlenecks.

Perhaps the most unforgivable failure in all of this was the seeming lack of instrumentation data or readily apparent customer feedback mechanisms.  I hope that this is being done as a mere pilot project at only a single of the airport’s several checkpoints.  But if it is then the validation of the pilot project’s effectiveness would not seem to be playing out in a meaningful manner.  I chatted up a couple of my fellow travelers at the end of the process to make sure the exasperation I experienced was not unique to me.  It wasn’t.  Sadly these two conversations appeared to be the closest thing to a survey being conducted on the experience.

When I mentioned to another traveler the forlorn hope that this was just a pilot project and that the airport would learn what a disaster it was before rolling it out to all of the checkpoints, he remarked “It’s the government, so I doubt it.”. I’m afraid that that my intuition says that he is right.  This could be done a whole lot better but seemingly the right people are not designing the system or overseeing its implementation.

Death To System Initiated Pop-Up Windows

Pop-up windows that manifest without a user-initiating action and grab user-input focus do not merely present a severe nuisance that can manifest in unintended system actions but also by consequence increase the attack surface by providing an avenue to subvert sandboxing and configuration management faculties.  Modern operating systems should phase out this faculty and replace it with a more friendly and secure one.

In lieu of unpredictable pop-ups, operating systems should gather user input requests from system processes and applications into a shared queue of action requests, require a GUI context-switch for users to operate on them, and provide an inobtrusive alert faculty that announces the presence of input requests without hijacking input focus.

I have been thinking of this for years but only just now found myself angry (and perhaps time-rich) enough to write something when I was typing something into Safari on my Mac Air when Flux contrived to pop an update request dialog box that my in-flight fingers accepted without my having any opportunity to know what was happening until it was too late.  Decidedly not cool…

Microsoft Windows has made some useful inroads into security by requiring more explicit user decisions when applications attempt to gain execution, but even that falls somewhat short by manifesting as a focus-grabbing pop-up that could fall prey to the above-described problems.

We can do better.