Why Agility embraces not changing as much as change
“The art of progress is to preserve order amid change, and to preserve change amid order.” Alfred North Whitehead
We live in the digital era. We are interconnected like never before. We are constantly deciding what to process amongst an overwhelming amount of data about almost anything. And the rate of change keeps accelerating due to our global economy which is locked in a vicious competition cycle of year on year growth.
Riding the crest of this tsunami of information and change are the software developers who are tasked with orchestrating it. Agile is the mantra in modern software development. Embrace change. Change is the only constant. But is it? That is certainly true on first glance and at the macro scale. Our codebases change. Our technologies change. Our businesses change. The society that our businesses serve change. But at the micro level it is as much about the things that don’t change as those that do.
The problem with too much change is that change represents novelty. This introduces uncertainty, unpredictability and inefficiency. If everything changed all of the time then we would have no assumptions to reason, no basis to function. So how do we maintain a sense of stability amongst all this change? By establishing things that don’t change, or in other words invariants. These invariants give us stability but also enable wider change. To understand more let’s rewind back to when our society started to use technology in such a way that triggered the acceleration of change.
Interchangeable parts
In 1801, Eli Whitney demonstrated something quite extraordinary to a group of people which included the current and future presidents of the United States. Using only a screwdriver, he assembled ten different locks to the same musket. Nowadays this might not seem so impressive, but this was in an era of craft production where no two parts were the same and assembly was a time consuming process of the utmost skill to alter each part so that they could be combined into a finished product. The prospect of mechanisation to quickly and consistently produce a multitude of weapons was intriguing from a military perspective. The interchangeability that the precision of the mechanisation facilitated would not only simplify assembly, but allow the potential for a layman to repair guns simply by replacing a like for like part.
Whitney had stumbled upon something that we would now call modularity. Where parts can be combined reliably due to conforming to an invariant interface. Any parts that meet that interface can then combine.
He was not the first to discover this. He probably got the idea from French pioneering of standardised parts. And the broader concept of modularity had existed since the ancient times – the Egyptians for example made bricks out of mud and straw in a mould. A very simple example of modularity. But in the context of the Industrial Revolution this was about to play a key part in our socio-technical evolution.
Although Whitney would go on to waste a considerable amount of the War Departments time and money without mastering interchangeable parts, he had sown a seed and by the 1850s the rifle industry had started to establish a reputation for machine tooling such as die forging, milling machines and turret lathes. All of which allowed consistent production of parts which facilitated interchangeability. This trend would spread to other growing industries such as sewing machines and bicycles. By the start of the Twentieth Century the true potential of modularity would be unleashed to allow production on an altogether new level of scale and complexity.
“I’ve had the same broom for 20 years…I looked after it well…This old broom’s had 17 new heads and 14 new handles in its time.” Trigger, Only Fools and Horses
Mass production
Henry Ford is probably best remembered for making the motorcar available to the masses through the moving assembly line. But it was his mastery of interchangeable parts that made this possible. His Highland Park factory was a mass of machine tools such as moulds, jigs, dies, fixtures and gauges. The modularity this enabled created economies of scale such as long run production of parts and highly specialised roles. This drove down costs making Ford cars accessible to more people which created a virtuous cycle of increasing economies of scale leading to increasing demand. Mass production had been born.
Decomposability
The scale of production in Ford’s factory was all the more impressive when you consider the complexity involved. By the mid 1920s thousands of workers coordinated to assemble the nearly 5000 parts that made up each Model T car and build nearly ten thousand cars a day. Modularity allows people to work on something simultaneously. Once an interface is agreed, people can work independently as they are abstracted from the complexity other side of them which allows them to focus on a more manageable sized area. This has enabled us to create ever more complex products.
A mere forty years after the peak of Ford mass production the same principles of modularity would allow man to go to the moon. Modularity in the mammoth complex design effort which involved up to 400 thousand people. But also in the finished product where millions watched the iconic separation and docking between different Apollo space craft modules.
Variety and change
Ford had exploited modularity to allow never before seen economies of scale of a homogenous product. But he was soon to be eclipsed by General Motors who would show that modularity was equally effective at producing variety and change. They offered a pricing ladder of different cars with more offerings in colour and stylings. They also instituted an annual model change. This appealed to the consumer demand for individual identity and social climbing with people buying what they could afford and then aspiring to buy the next car up the ladder. To achieve this economically, modularity allowed parts to be shared between different models and therefore maintain economies of scale. This capability to offer standardised parts which can be mixed to provide a unique product has proven to be one of the most enduring benefits of modularity. A perfect balance between economy and customisation. Just ask an IKEA customer who can choose from a seemingly infinite combination of unique products to meet their taste and circumstance by purchasing from a standardised set of parts in different product ranges.
Nowadays the effects of modularity are all around us – repairs on your car; vacuum cleaner attachments; installing apps on your phone, or plugging electrical devices into sockets. In software development, that is equally true.
Modularity in software development
The aforementioned benefits of modularity are all very familiar to software developers – decomposition of complex solutions, better efficiency through reuse, facilitating easier maintenance and innovative change. The primary preoccupation of software architecture and design is deciding what not to change to facilitate wider coordination and onward maintenance. Modularity is exploited at many scales. The class for example exposes a public interface to callers, the invariant, whilst encapsulating private data and methods which can be refactored (the variant). With larger systems, teams may expose APIs for other parts of the system to call or even for other systems to interact with. They need to ensure backward compatibility (i.e. invariance) so that they do not cause incompatibility with their many clients but the implementation behind that can vary (i.e. variant).
But modularity is just one of the many tools of invariance in the software developer’s toolbox.
Code
The most fundamental invariance of software development is the code itself. Each line of code is an invariant that acts upon a variant, the data, or as it may be referred to in the code the variables. In this way a software developer can reduce a complex business process that encompasses a potentially infinite set of circumstances, of data, to a simple algorithm that might be a small amount of code. This is one of the superpowers of software development.
Immutability
As software solutions have grown in complexity, it has become increasingly difficult to predict or control the impact of change. Particularly where conflict occurs due to concurrent changing of something shared. This has led to the popularity of another invariant. That of immutability where things are designed to never change. No change means no conflict and so no unexpected results.
Immutable data types prevent conflicts in data shared between concurrent executing threads. With more and more concurrency this is one of the growing appeals of functional programming.
At a large system scale a similar benefit is gained with design approaches such as event sourcing and tools such as Kafka to ensure that many of the conflicts of changing distributed data are avoided by making the data immutable.
Immutability is also the key to tools such as Git which ensure code changes are stored, tracked and integrated reliably and consistently as multiple incoming changes occur and branches are merged. Likewise with Docker to ensure the environment the software runs on is consistent.
Even collections of code such as classes can be written to avoid change. The issue with changing a class is that it likely increases its entropy which needs further work to reorganise it (i.e. refactoring). Also the complexity of software means that when a change is made, it is often difficult to predict the consequences. Side effects or bugs can occur. But if the class never changes then no entropy and no side effects. This is the idea behind the Open Closed principle which aims to design classes that do not need modified but instead can contribute to new behaviours by being extended via their modularity to combine with other classes in innovative new ways.
Building on invariants
Wherever you find a high performing software team, amongst the constant change you will likely find a stability drawn from many invariants. Invariants in the design of their code and systems such as modularity and immutability. But also in many other respects too.
Technologically— faced with the cost of closing a continual knowledge gap due to an ever emerging stream of technologies, applying them economically through the invariance of tried and tested principles, patterns and heuristics. Those of good design and craftsmanship.
Socially — whilst the business scales creating vastly different demands of the communication structures, the constancy of trusted relationships at the local level by retaining long lived stable teams.
Evolutionary — amongst the relentless change of business priorities, establishing routine through the cadence of feedback loops such as daily standups, retrospectives and quarterly OKRs.
So if you want to master change, look past the turbulent surface to the deeper more stable foundations.
References
From the American System to Mass Production, 1800–1932: The Development of Manufacturing Technology in the United States – David Hounshell
My Life and Work – Henry Ford
My Years with General Motors – Alfred P. Sloan
