No perfect person exists for a given role, just the best one at a point in spacetime. So, too, it goes for process, technology, and culture. Thus to successfully steer an ecosystem leaders must emplace the right team at the outset, sense phase transitions proactively, and modify conditions smoothly. They must also reliably recognize when their purview includes activities requiring shrewd partitioning. Act too soon and one creates headwinds that stifle innovation and undermine purpose. Act too late and the enterprise finds itself bogged down in a swamp and breaking promises. Timing is everything.
Navigating The Phases
Engineering activities operating under an iterative delivery model endlessly cycle through a well known set of phases to release new units of functionality, but they also simultaneously progress linearly through a higher order set of phases. We interest ourselves with these latter phases presently, both with regards to their inherent nature and the kind of people and culture that best support them. Note in particular that although an individual may have competency in a range of phases, competency in one typically comes at the expense of competency in another, and yet “Flexers” serve a key role in smoothing phase transitions.
POC: To create a Proof Of Concept, inventors lay out some novel idea and endeavor by hook or by crook to prove vague technical feasibility. We must emphasize creativity and risk taking while shielding the activity from bureaucracy. The outputs of this phase may justify further investment but we must hold them in suspicion as we would scientific research not yet peer reviewed.
MVP: Progressing to a Minimum Viable Product requires enough engineering rigor to avoid squandering the confidence and enthusiasm of initial contacts and yet also a ruthlessness for cutting just the right corners to expedite time to market. We must emphasize practicality and resourcefulness while quashing unduly speculative investments. In reaching this threshold we begin to attract early adopters who will help us shape our product. We also find ourselves increasingly at risk of getting too far out over our skis.
Scaling: To enter the Scaling phase involves a transition from building something people want to building something enough people will purchase at a price that supports operating a sustainable business that can stand up to compliance scrutiny and hostile action. To navigate this phase entails managing the scaling of multiple facets — infrastructure costs, product complexity, team size, and customer composition. We must emphasize prioritization, automation, reliability, security, and unit economics. Increasingly we require the services of technical specialists to navigate arcana and managerial generalists to control complexity.
Sustainment: Often foolishly characterized as the unsexy “Maintenance” portion of the lifecycle, this phase in fact offers the most complexity and peril as burnout, bureaucracy, and complacency enable the competition to eat our lunch. Managed poorly it marks inevitable decline. Should the right people seize the moment, however, the enterprise may evolve to a fractal shape, one effectively consisting of a company of companies. At this point many such “companies” may exist out-of-phase, a substantial amount of politics become inevitable, and a critical personnel archetype must emerge to walk the knife’s edge of stability and relevance.
Managing A Company Of Companies
Having established a viable business model, a company will then find itself contending with the following inextricable challenges:
- Protecting existing equities
- Maintaining market relevance
- Managing cultural conflicts
To borrow the terminology of Safi Bahcall’s Loonshots, we increasingly find ourselves at risk of conflict between the “Soldiers” struggling to operate the business of today and the “Artists” striving to imagine the business of tomorrow. And to manage this challenge we may seek to integrate these disparate groups while creating a single homogeneous culture — a terrible mistake. The Soldiers will experience the Artists as anarchists, the Artists will experience the Soldiers as bureaucrats, productivity across all measures will tank, and attrition will ensue. Far better to recognize the unique values of each, craft an organizational architecture that provides homes for both, and foster Flexer individuals and squads to maintain dynamic equilibrium at the boundaries.
In the context of the aforementioned phases, Artists gravitate toward the POC end of the spectrum and Soldiers toward the Sustainment end, while the ideal Flexer will have the bulk of their skills in the Scaling portion and yet also have at least some competence and inclination in the adjacent MVP and Sustainment realms. Perhaps the most salient challenge herein consists of giving both Soldiers and Artists a home base of compatible individuals while avoiding a toxic antagonism stemming from a “throw it over the wall” approach around transitioning prototype functionality into production systems. Instead of experiencing such an abrupt phase transition we can instead employ Flexers to continually be asking the Soldiers “wouldn’t it be nice if you could do this? and the Artists “have you taken into account these constraints?” so that innovative capabilities have a smooth path to landing in sustainment organizations.
Tangled up in all of this are vexing challenges around the following…
- Cross-cutting technology and techniques
- Access to systems, data, and users
- Respect for process and equities
- Inertia, complacency, and possessiveness
Separating The Timeless And The Transient
Huge risks exist in nascent enterprises around premature investment in infrastructure and enforcement of rigor. What good are those outlays in comprehensive test suites, hyper-scalable architectures, and bulletproof security when you realize you were building the wrong product and have to pivot hard? And yet, while we ought fold rigor into our applications in a dynamic and phase-aware fashion, we should also view certain certain basic technology and techniques as foundational and universal, shaping them as early and as generically as possible.
We will doubtless discard most of the code written as we navigate the idea maze on our way to a valuable product. The incentives and rationale for a minimalist approach are strong. Nonetheless we would do well to tease out two timeless elements from all that will likely prove transient…
- Accessible data warehousing
- Flexible DevOps tradecraft
Why? Two simple needs…
- Continual reassessment
- Sustainable velocity
Accessible data warehousing provides the basis for generating intelligence on both how we are performing today and where we may want to be tomorrow as well as the raw material to build new applications. Flexible DevOps tradecraft, meanwhile, allows us to consistently sustain velocity despite shifting requirements and growing complexity. At the outset we will likely have only the crudest idea of what we will ultimately build and so let’s have the humility to stack the deck in our favor with the right foundational investments.
Meanwhile, as a company grows, an interesting question emerges around whether to take a “mission striped” or “function striped” approach to organizational architecture as pertaining to specialists supporting cross-cutting concerns. The correct answer is often “both”, especially when it comes to the realms of DevOps and Data Engineering, but perhaps equally as often with Security Engineering and UX. In particular, an organization and its specialists may benefit from establishing Centers Of Excellence around certain practices, expecting that practitioners will both have long-term assignments with product focused areas as well as participate in and draw on pools of similar specialists. These CoEs ensure that individuals receive appropriate mentorship and peer review, tooling and tradecraft exhibit convergence, and the enterprise reduces risks around knowledge silos, burn-out, and general immaturity.
Safe Spaces
Artists need the flexibility and chaos of a mad scientist’s lab. Soldiers thrive on the order and predictability of a well run factory floor. And each group requires the other to sustain long-term vitality. Instead of glibly and universally promoting a “bias toward action”, organizational architects must visualize risk strata and think in terms of the Fast/Slow Problem which dictates that entities that need to change quickly must not be collocated with entities that need to change slowly.
Matters inevitably become fraught, however, when Artists attempt to gain access to realistic data whose safeguarding lies in the remit of Soldiers. In some cases such data may represent the company’s Crown Jewels and in others pose perilous compliance risks. Various techniques exist to smooth the process while preventing spillage, spanning sanitization, sandboxing, and field-level access control, all aided by elastically scalable data warehouses that are isolated from user-facing production operations and can handle bursty non-quite-planful loads. Mature DevOps processes with fast cycle times further ease the struggle by allowing exploratory analytics a fast path into environments with production-level data controls.
That promotion of code, however, represents the second and previously intimated problem around transition and ownership. Far too easily a “prototype” becomes “mission critical” only to have the Artist progenitors become bored and the inheriting Soldiers behold what the former hath wrought with horror. We might lay the blame squarely on the Artists for taking a “throw it over the wall” attitude around responsibility hand-off to Soldiers but in fact that cultural toxicity often results from and pairs disastrously with the cookie licking and change aversion of the latter which foments a secrecy on the part of the former. In many places “it’s better to ask forgiveness later than permission now” which foments guerrilla engineering, but that is a hallmark of the aforementioned toxicity and indicative of a lack of Flexers that empathize with both groups while gently coordinating their activities.
And all the while the siren’s song of micro-service architecture calls to us, promising us clean boundaries in which small teams can have their own safe spaces to tinker and make technology choices unilaterally. Beware, however, that while such an architecture can serve as a valuable technique for scaling the number of participants building a complex system, it requires a keen understanding of several techniques, demands a certain maturity of DevOps tooling, imposes a per-service tax, and when unmanaged increases the risks of Balkanization. As with all matters, timing is everything, and attempting to adopt a micro-services architecture prematurely nearly always yields instead the dreaded Distributed Big Ball Of Mud.
Conclusions
The growth-oriented organizations that last are the ones that recognize that their internal structure must both continually change and avoid homogeneity. Be sure, however, to file this under “necessary but insufficient”. Far too often, alas, re-orgs result not from a careful consideration of the present context but rather the predilections and politics of the day’s power brokers, yielding violent pendulum swings between the extremes of mission-striped and function-striped architectures. Predecessors are blamed, reshuffles are made, and, finally, three envelopes are prepared…