What Product Development Can Learn From Music Production and Movie Making
The big debates in software development and product management tend to be about finding the “right” process to develop software products. There are as many defenders of any given method as there are detractors. It’s also the case that many of the most widely practised and applied product development methods fail to deliver their promised benefits consistently. Something is wrong.
Because I am also from a music production and film making background, I think there are practices from those disciplines that can inform software product development, which the prevailing process frameworks mysteriously overlook. This article is a speculative discussion about why film makers and music producers don’t have a single method (encapsulated in a formal framework) and which practices software product development is not incorporating into any of their process frameworks. Product developers are missing several tricks.
If you asked a music producer why they were not applying agile processes, peer programming of their digital audio workstations and synthesisers, discussing user experiences of their songs and estimating story points for every bar of music they create, they’d look at you as if you were from Mars. Talk to a movie director, or the editor working in the cutting room, about the benefits of sprints, product backlogs and burn down charts and they’d give you a quizzical, “Huh?”. Yet, it’s abundantly demonstrable that great records have and continue to be produced and increasingly complicated movies get made, which delight audiences. How are these people getting all of that collaborative creative work done, on time and budget, without an agile framework to follow?
Why does software development, singularly, seem to need one?
Getting to the Essence
The proliferation of software development methods, over the past several decades, has resembled the Wild West, with many home-grown, ad-hoc methods proposed as the one true righteous way. What these frameworks have had in common is that they define their own terms, non-interchangeably with every other framework. They start from different assumptions and make different assertions about goals and values. Each has different strengths and blind spots. No single framework seems to encapsulate the full end-to-end process, from vague hallucinated idea to production-quality result, shipping at scale. They tend to be as restrictive as they are prescriptive and in codifying the rules, fail to deal adequately with innovation and variance. One size hardly ever fits all. In short, it’s a confusing mess and there is no agreed lingua franca to describe what are essentially the same methods and practices, which they share in common.
A very brave soul, Ivar Jacobson, took on the task of trying to distil the essence of all of these methods and practices into a common language and framework of understanding. It’s an ambitious and commendable goal. I’m not expert in his resulting descriptive schema at all, which has been named “Essence”, but my understanding is that it seeks to regularise all of the best things that various frameworks for software development bring to the practice of product design, carefully separating the methods from individual practices. It is, at its heart, a variety of pattern language which seeks to propagate best practices to arrive at optimal outcomes.
In seeking a common denominator and a lingua franca, the hope is that product development with software can be made better, faster, cheaper and happier. Whether or not these are adequate goals remains debatable. To my mind, they’re lacking in ambition and human values. I would like to see software development focus on personal empowerment, equality, diversity, universal thriving and regenerative economics, for example. The choice of what you optimise on is a highly charged political position, in reality.
Competencies
Practices require competencies. If you don’t know what you’re doing, it’s going to be hard to be an effective practitioner. Many software development frameworks gloss over this important point. They only work if the practitioners are competent, yet competency is something you continually grow. Nobody ever reaches the point where they are 100% competent in anything, let alone everything required by a given practice. Also, no two people develop at the same rate or along the same dimensions, so even among pools of practitioners of the same practice, there is a tremendous variance in breadth and depth of skills. Some software development methodologies deny this entirely, or make claims that if you follow the method, your individual competency won’t matter, or else that the method itself, if followed faithfully, will magically confer competency on the incompetent. None are really true. Obtaining of mastery in any practice is a painstaking process that takes years of learning and experience. You can’t obtain competency by following the right recipe. It’s never that simple.
What is the Essence?
Essence, it is claimed, is the common ground of all methods and frameworks (presumably for software product development, though this is not stated explicitly). It doesn’t include any roles, any specific artefacts or any process. Competencies are included because they are needed. Dimensions of progress are called “alphas”, so that they can be measured. It’s a new construct, unique to Essence, that allows measurement of progress and health in an agnostic way, irrespective of which method or framework is being used. Alphas have states to do that. Each state has a check list measuring real outcome, not artefacts produced or activities done. There are seven alphas: Opportunity, Stakeholders, Requirements, Software System, Work, Team and Way of Working. All seven have to be progressed for a software product development to be successful.
Importantly, Essence has the ability to add both tacit or explicit practices.
What Do Other Creative Industries Do?
It’s generally accepted that making a record or a feature film is a creative undertaking that involves complexity, artistic risk and collaboration, where many practitioners of different competencies come together to create a work that couldn’t be realised by any single individual, much like software product development. The purpose of all three activities is to make something that consumers love. A side effect of satisfying a paying audience is that people make money. Indeed, it’s only possible to make money if the audience is willing to pay for it and they’ll simply withhold their cash, if the offering is not good enough. The stakes are comparable.
Interestingly, these other arts get by just fine without the formalism and abstraction of frameworks. There is no equivalent of Essence in music making and film production. Why does software development regard itself as exceptional? Why do product developers see the need for elaborate frameworks of understanding of their own methods and practices, whereas developers of musical or cinematic products do not? Clearly, there are successful films and records made all the time, without the benefit of a formally described development framework. Is the software industry fooling itself, overcomplicating their creative process or otherwise trying to compensate for deficiencies in mastery of competencies and creative vision?
Given the existence of successful films and records, can the methods used by music producers and film directors inform product managers about how to make products that audiences love? What can product developers learn from music production and movie making (if anything)?
Where Do Methods Come From?
Practices and methods are invented all the time, thanks to imagination. This is a good and healthy thing and is the reason why we never get bored of the films and music offered to us by their producers. There’s always something new and innovative that balances expectation with surprise. Trying to insist on just one right, true way to make a record or a film would be a ludicrous proposition, to those that spend their working lives making them.
Software development, in contrast, seems constantly fixated on striving to find one, or a handful of useful development methods that fit every circumstance. You have to do Scrum just right (just to take one example), or your development shop isn’t really agile. Who cares? What if Scrum is a very bad fit for what you’re trying to make? I will suggest, in this article, that the lack of method diversity is something that holds software product development back severely. Innovation is stifled, if you aren’t willing to entertain alternative methods to create something, without obviously sacrificing quality and customer appeal.
I contend that this obsession with legislated software development methods is wrong-headed in the extreme. A plurality of methods, or plans of attack, if you like, provides a healthy, creative diversity in how things audiences love will become manifest. Sticking to just one way of organising development guarantees that every product will be a reflection of that internal organisation. Other interesting and valid products will simply not emerge, if the organising principle of every software development shop is identical.
Methods can’t be imposed from above, by the recognised authorities, standards bodies or groups of august industry experts. Nor should they be. The notion of doing so is somewhat of an absurdity, born of a desire to create the illusion of control. The method a creative visionary uses to realise their particular vision is a signature of the artist and is often influenced by the content of the creative work — what you are trying to make or accomplish drives which method you have to choose, in order to make creating it possible. In software product development, as in film and music, a method appropriate to producing the envisioned work should be chosen. It’s a primary creative choice.
That’s not to say have no method. No film or record could get made if producers adopted no methods at all. The choice of development (or production) method is a key creative decision, which if encumbered by insistence on a singular methodological framework, kills innovation and diversity at a stroke.
Here are some examples of method and practice invention and diversity, from the worlds of film and music. There was no visual effects industry of any significance before George Lucas and Star Wars. He had to invent new methods and practices, in order to realise a new visual grammar. Following a standard film making method would not have allowed Lucas to convince audiences they were visiting a credible alternative world, set in the future, replete with fantastical creatures, hyper-drive space travel and planetary-scale fortresses. Creating this illusion absolutely required that new methods be invented.
By way of analogy with George Lucas’s visual effects and computer generated imagery, if you consider current software development frameworks, they all seem to lack any recognition of the important work flow from user experience designer into development. You won’t find much support for this practice in Scrum, or any of the other software development methods. It’s a blind spot, yet the most beloved software products in the market have somehow developed their own method that takes the skill and competence of designers and marries that to software development practice. Can that be Essentialized? I don’t know.
Paul McCartney and George Harrison, half of the Beatles, had the same high school music teacher, who assessed them both as having no particular promise, musically. Following his classical music biased curriculum of education, which ignored anything and everything to do with music recording, song writing and individuals creating their own musical identities, he adjudicated them on the basis of a standard method, against which they came up short. Had these two Beatles accepted that there was only one true method of creating music, audiences would never have had Sergeant Pepper’s Lonely Hearts Club Band.
Brian Wilson of the Beach Boys received an “F” for his high school musical composition, a song called “Surfin’ Safari”, which went on to earn him millions. He was out of step with the standard, accepted method of composing. In the recording studio, Wilson again broke with traditional methods, leaning heavily on overdubbing (something which Les Paul invented for himself and others had later adopted) and the use of unusual instruments and musical arrangements. If he’d stuck to what the record producers of the day considered to be best recording practice, none of that wonderful music could have been made.
It has been shown that jazz pianists have different brain wiring to classical musicians, because they plan what to play, more than how to play it — the exact opposite of what classical pianists do. There wouldn’t be any jazz at all, and probably very little musical improvisation, if the only method of playing music available and insisted upon was that of classical musicians. Indeed, there would have been no classical composers either. Mozart’s relationship to music composition methods was his own. He created it, as he went, to serve his own purposes.
In all these cases, the imposed methods and practices, against which the artists resisted, were demonstrably not the exclusive way of making the creative work. That should carry an important lesson for software product development.
Does Creative Vision Emerge from Method?
One of the most egregious aspects of modern software development methods, in my view, is their attitude toward the initial creative vision. Some imply that a creative vision will emerge, if you iterate often enough, in front of enough customers. That approach relies on the customers knowing what they want, which often isn’t the case at all. As a method of working out what to make, it’s a bit directionless and random.
Others believe the creative vision arrives at the start of the development effort, fully-formed, in the mind of the founder and thereafter is simply executed upon, by developers, reproducing the idea faithfully. This is nonsensical too.
None of the existing development frameworks, as far as I am aware, provide guidance for ideation, nor how to select good ideas from the bad. There is scant recognition that even if a creative vision is the starting point, there is considerable interplay between the method of development chosen and how the initial creative vision evolves to become the delivered product, as a result of that choice.
Most (if not all) software development methods fail to codify how to create a good product vision and product concept. Maybe you can’t. There are methods, like Jobs To Be Done, which might get you some part of the way there, but I don’t think they’ve been Essentialized.
In any case, the solution elements chosen, the technology stack and the competencies of the people involved in the collaborative development shape the product vision beyond its initial conception. That’s inevitable, as it is in music product and film making. The actors and musicians bring their own slants to the creative endeavour and the final product ends up quite different, though similar, to what was initially intended.
Software product development is not immune to this, but agile and other methods fail to recognise the effect. Some pretend to entirely suppress it and homogenise the very personal quirks of the developers involved. Suppressing developer individuality and brilliance is not necessarily the best thing for customers.
Where’s the Pre-Production?
In music and film production, selection of which creative vision to pursue is loosely called “pre-production”, when the vision is refined, prior to the actual making of the work. When you hear about scripts being “in development”, it means the concept is undergoing refinement, revision and modification, not the film itself. There are pre-production stages involved in film and music that don’t have good analogues, in software development. Anything that resembles “big design up front” is considered bad and methods gurus claim it won’t be useful for the finished product anyway. This is only asserted in software construction.
In other creative pursuits, the planning done at the concept stage is crucial to even getting the go-ahead to spend serious money making the work. Good pre-production planning can save a lot of wasted time and expense, once production begins. Changes on paper are far cheaper to make than changes to performances made on location, already affixed in recordings and video footage. They keep doing it, because it’s necessary and it works.
Pre-production also helps you select or invent the production method you’re going to follow. If film makers had religiously followed the methods of silent film producers, they would have never invented talkies, or had a way to incorporate the recording of live dialogue, and its studio-quality replacement, to match the actors’ silent performances. It took pre-planning.
Similarly, if Disney had charged in and started animating two-dimensional frames of film together, before deciding to change the traditional method of making animated films to incorporate a new, multi-planar, rostrum camera, the first colour animated feature film would have looked a lot less three-dimensional, less visually interesting, flatter and not so innovative. The ability to choose and change methods, before starting on the work, was a crucial factor in his success.
In agile software development methodologies, this kind of preparation and forethought is not officially valued at all, except in sprint planning, which misses the big picture, while planning the small. Changing the development method to suit the creative goals is heresy. Conservatism is baked-in to the organisation, immutably. Any suggestion of taking a look at the whole product concept, in order to plan how to tackle it in its entirety, is considered to be something akin to waterfall. What’s missed is the opportunity to invent the new methods and practices necessary to create something that has never been available to customers before.
Where are the Rough Cuts?
When film and music producers want to see what their product will be like, they start with a story board or track sheet to visualise it on paper. Often, those story boards are animated and timed, to create a framework for the picture being made. There is a concept called the “rough cut”, where the dailies are cut into the rough edit, sometimes even interspersed with static storyboard place holders, to see how the product will play out, when finished. These rough cuts are emphatically not the final product and the audience/paying customers are almost never shown them. They exist only to visualise the overall shape of the product.
What is the equivalent in software product development? There are pre-production wire frames, proofs of concept and prototypes, but seldom are these a close representation of the intended finished product. Because of incremental development, developers often have no idea what the whole product will be like, even while delivering early iterations of the product to market. The sacrifice is user experience consistency and cohesiveness, which much agile software development suffers from.
Since the product development methodology fails to provide a tangible artefact to demonstrate how everything is supposed to fit and work together, there is no way for developers to make software today that will work nicely in the context of work they will do in a year’s time, for example. That would be fine, if they could rework initially-delivered features to match the newest thinking, but often that’s deemed cost and time prohibitive.
In rough cuts, the structural hierarchy of the film can be visualised and modified early, before it’s too expensive to change. It permits good story telling, through the development of character arcs and sub plots. The big picture shape of the product is available as a guidance system to all the detailed, painstaking work that needs to be done to finish the film. It’s as if there is an architect’s and structural engineer’s blueprints available.
In music production, rough mixes made at the end of each day’s work, allow music producers and musicians to listen to the music they’ve made and adjust what is and isn’t working, before the audience gets anywhere near the finished product. Where is this equivalent in software?
Software product development tends to begin before the product concept is even clearly articulated and it evolves chaotically, as customer complaints begin to arrive. There is little sense of a settled product concept intention, prior to coding and few artefacts to guide developers toward some semblance of user experience consistency. The method most programming shops follow is to hope that a cohesive product emerges from multiple release iterations. In my experience, it rarely does. As valuable as customer feedback and their reactions, against continuously delivered iterations of the product is, you can’t let that fill the vacuum of not having thought about the whole product intention.
Similarly, if you decide you need to pivot the big picture product concept, count on having to do a lot of rework of previously released software. Otherwise, even by following an agile methodology to the letter, you will wind up with an offering that is neither one product concept nor the other. It will be a weird, unsatisfactory hybrid, which doesn’t do anything particularly well. We most frequently call products in this state “bloatware”. You’ve all used it.
Sweetening
A common practice in music production is audio sweetening, where the performance recorded is adjusted so that it sounds more polished and professional and in general, more pleasing to the ear. This is accomplished by the addition of reverberation, adjustment of the dynamics of the signal and equalisation of the component frequencies. Sweetening is performed in the context of other audible parts of the work, so that each piece of music recorded sounds distinctive, but blends together seamlessly, with other instruments, in a cohesive mix.
In film production, although the lighting and camera settings are crucial, there is a post production process called colour grading, which seeks to balance the visual look of the footage in the context of the whole film. Colours are emphasised and de-emphasised to give all the footage a consistent overall quality.
In software development, integration builds bring individual programmers’ work together, but there is no “signal processing” to sweeten the work submitted and make it sit together cohesively. There is a missing refactoring practice, in most software development methodologies. There isn’t a formal step where, at integration time, the work of the contributing developers is modified by another developer, with a view to making things look and work in coherent and consistent ways. The idea would be that, at integration time, a specialist programmer would rework some of the submitted code for one reason only — to make it all appear like its part of the same product concept, not the work of different people working in isolated opposition to one another.
How did software development miss this useful step in all of its practices, whereas both film and music production ensure that sweetening always happens in high quality productions? Could it be because of software product development’s lack of method diversity and fixating on one true way too soon?
Who Delivers the Vision?
In record production, the record producer is supposed to be the responsible adult, on the hook to deliver a product of marketable quality, heeding the instructions of the record company, but also playing umpire and diplomat with the collaborating artists. Their job is to deliver the creative vision in product form. They can only do that by not diluting anybody’s artistic integrity too badly, yet overseeing the planning and production, so that the work gets done.
When it comes to making films, the director is the man (or woman) in charge and its their job to ensure delivery of a pleasing film, compelling enough to cause people to want to see it, with the fewest compromises to integrity possible. They have to get the product made too, wrangling the many collaborators into effective contributors.
Software product development places all the responsibility (and blame) on the shoulders of the product manager. They, like record producers and film directors, are responsible for the quality and saleability of the work and for getting it done when promised. Whereas being an outstanding director can win you an Oscar and being a top-flight record producer can earn you a Grammy award, there doesn’t seem to be any accolades or pubic recognition for highly effective product managers. Often, they don’t even get share options. I think that’s a pity. The jobs are comparable in scope and difficulty.
In agile methodologies especially, everybody is supposed to be responsible for the product, with the net result that nobody really is and nobody can agree on what the product should be or adjudicate on that authoritatively. It’s left to the product manager to try to sort out the mess, amongst the myriad stakeholders.
What product development can learn from music production and movie making is that one person, representing the creative vision, needs to have the final say. Nobody should second guess the final adjudication of the product manager, even if you disagree with it, provided all views have been acknowledged and aired.
Who Conceives of the Creative Vision?
Every product starts as an idea, but whose idea is it? In films, it’s usually the script writer. Most films start with a script, which becomes the basis of the whole movie, though it is revised many times, by many people, during production. In music, the product begins in the mind of the songwriter or composer. In software product development, things are less clear cut.
Often there is a recognised product visionary — a Steve Jobsian character — who becomes the public face of the idea. It may not be their original idea, but as the public representative of it and the brand ambassador, they tend to have a very large say in what goes into the product and what’s left out. They also become the guardian of product quality, fit and finish, which in their absence, has a tendency to gradually slide.
This role is far more than being a product owner, in the sense of an agile participant that grooms and prioritises customer feature requests. It’s a deeper skill and requires somebody that can act creatively. Some software methods forget to put anybody in charge of the creative vision overall.
Performance Coaching
The product owner role is routinely filled by a product manager, but they also find they are required to be a developer performance coach too. Software methods tend to fixate on this aspect of the work, forgetting that developers have to bring excellence to the party, just as a musicians and actors have to, in their creative endeavours. Yes, directors and record producers have a coaching role to play too, but there is a minimum quality bar that applies in music and film that appears to be more relaxed in software development.
Excellence takes years to mature, but software methods often try to deny or ignore that. Instead, they try to turn poor developers into good ones, through religious adherence to the process, while overly-constraining virtuosos. Often, though, bad developers don’t learn to be good ones just by following a software development methodology. There is more to it than that, as anyone that has worked with outstanding developers will tell you. Many developers learn on the job, on production code, so their mistakes are inflicted directly on paying customers.
There isn’t even an equivalent of an apprenticeship, where you formally study under a master developer either, really. Ad-hoc mentoring takes place, but young, cheaper, inexperienced developers often do the majority of the work and it shows. Music and film producers tend to keep the amateurs and learners out of the production, until they get better. Software shops just let them loose.
Honing the Craft
When musicians want to perfect their craft and improve their musicianship, they play live gigs. Actors have live theatre in which to hone their skills. The ability to perform what you do, spontaneously (but carefully rehearsed), with no opportunity to edit or correct errors, in front of a critical audience, is actually a very powerful forcing function. It makes you learn fast and become a master of your craft.
Though software developers are exposed to live paying audiences in production, they don’t have to produce code on demand, in front of an audience that is waiting to be entertained. There really isn’t such a thing as a live software development gig. Hackfests might be the closest equivalent.
Product development can learn from music and film that the requirement to produce your craft as a performance, live, in front of people, without the ability to retract anything your do, is a fabulous way to become a very good developer. Coding jam sessions, where coders work together improvisationally, probably ought to be a thing.
Creating Structure and Clarity
In film, the person whose job it is to maintain the clarity of the structure of the story is the film editor. They tell the story through selection of shots and camera angles, ensuring that transitions and timing maintain the right mood, according to the dialogue and acting. In music, the arranger is responsible for putting the parts and instruments in the right order and balance. They work on getting the overall impression of the work and its timbre to be emotionally affective (sometimes the producer is the arranger). In software development, the nearest equivalent is the software architect, though many agile methods have evidently begun to eschew the role, claiming it undermines team egalitarianism and permits too much big design up front.
In a software development team, anybody that volunteers to create clear structure throughout the entire product is frequently resented as a jumped-up, arrogant know-it-all, whose skills are scrutinised minutely, in an effort to prove they are no better than any other developer on the team. That’s not the point of the role. The point is that somebody should care about delivering a product that has clear, consistent abstractions and structure at its core and to put those characteristics into the product, if they aren’t already there in the code delivered by the team. The person dedicated to do so ought to be free to operate, without second guessing or being undermined. Both music and film, in contrast, highly respect the keepers of clarity, continuity and structure.
More important than the role, however, is the practice of creating clarity and structure and it’s presence in methods of software development. It shouldn’t be omitted, unless what you’re creating demands that it is. This is true for all of the roles discussed in this article. Who it is doesn’t matter as much as what they do should be a part of making great software products.
Finalising
When a feature film or record album are ready, there is one last step in the development process. The finished film or album is finalised. In music, that task falls to the mastering engineer, who checks the product for technical compliance and sound quality. They ensure the final product sounds perfect, with no obvious flaws or inconsistencies between individual tracks. In feature films, the studio generally has the final say on the finished edit. Everything is checked for quality and for a logical and pleasing unfolding of the story being told. Finalising is a last sanity check, before the product is released to the public.
In software product development, this process either doesn’t have a close analogue, or else it’s done in a fragmentary way. QA, DevOps and InfoSec often check a release candidate before it goes out, but nobody is empowered to change anything. Instead, they raise rework tickets against the developers. In film and music, the finalisers have the power to make the changes necessary to ensure the product is perfect on their own. They do not need to refer back to the director, music producer, actors and musicians, in most cases.
Finalising is a distinct skill that requires a different mindset to that of a regular developer. They are looking for different flaws than a developer would and they have skills in fixing those flaws, independently of the originator of the work. In software development, this would mean a specialist developer with the ability to rework source code and release candidate integration, without involving the original code developers. What they would be looking for and correcting are the seemingly minor ripples that make a product look and feel less than complete.
Music and movie production relies on finalising because their product cannot be modified easily in the field. After release, the product remains invariant for all time and stands as a permanent monument to the artists involved. Software products, in contrast, are often chased out the door, with patches and revisions issued seemingly all the time. This creates a moving target for users, who are never sure if they have the right version and who have to take time out from using the product for its intended purpose to apply updates and fixes. Customers are beginning to get increasingly impatient with the intrusion.
Continuous delivery avoids the user-disrupting updates, but the user is never quite sure if the product will work the same way as it did yesterday and they struggle to become aware of any improvements made, thereby rendering them invisible and consequently useless. While continuous delivery might make it possible to get early and frequent user feedback, it denies the need to add final polish to the product before putting in front of customers. That’s a pity.
Producers of digital musical and visual content finalise because you never get a second chance to make a first impression. Software producers should consider this.
Novelty Beats Competency
Competency is a pre-requisite for any practice that forms part of your method, but there is an exception. When the method you follow, to produce a piece of music or a film, incorporates a novel process or produces an interesting result, then the competency level doesn’t have to be so high, simply because there are no other examples of the craft to compare with. If you are the first person to use a harmonizer on a bagpipe recording, for example, then it doesn’t have to be very good, so long as it’s unique. The fact that it has been incorporated at all, in the work, is enough to make it valuable, even if you are learning as you are doing (and truthfully, who really isn’t?). In short, doing something innovative allows you to do so with less competency than doing something routine and expected.
The implications for software product development are that if the product does something unavailable elsewhere, or you have incorporated a new practice into the method (such as a better way to take design work into code development, for example), then anything you do is an improvement over nothing, even if the new practice isn’t fully worked out yet, or not at the same level of proficiency as other practices. Innovations have a lower threshold of acceptability and viability, in a product, than the stuff everybody else already provides. This is yet another reason why innovations in software products are so valuable.
Defect Backlog
When you make a film or a record album, there is no defect backlog. Tickets are never raised in Jira, for the attention of the original artists in a subsequent iteration. Defects don’t boomerang back to the musicians and actors. As noted earlier, there is no defect backlog because the product gets finalised — perfected (as close as is possible at the time) so that it can stand the test of time, in the wild. If you think it needs a tweak here or there, too late because it has already been released. The people that care most about the final quality of the product fixed the deficiencies as they found them, prior to release.
Software products are rarely finalised, so every release results in a growing backlog of minor defects that disrupt the flow of creating the next new product. They become nuisance issues that developers must attend to, which they somewhat resent having to fix, because they seem trivial. It’s also a drain on margins, as fixing an issue with something already sold incurs an opportunity cost — it takes time away from making the next thing you could sell. To the customer, however, these defects, discovered in the field, represent a reduction in the quality of the product they bought and detract from the user experience. They feel let down and short changed. It damages the brand. Yes, the feedback from users can be valuable in guiding how to develop the next thing, but the cost is reputational damage. Users want a product of the quality they’ve come to expect from films and records. They do not enjoy reporting defects and then having to wait, to see if and when the issue will be corrected for them.
When you ship a minimum viable product (or preferably, an optimal viable product), this does not mean that what you have completed is allowed to be flawed. The things that are finished should be properly finished and finalised, so that they don’t raise defects in the field. The product might not yet have all the functionality envisaged and the overall complexion of the product can be subject to radical change, but what you’ve done so far ought to be fit for release forever, with no corrective patches.
Technical Debt
In film and music, there is no technical debt, per se, because even though formats come and go and technology makes better products possible with each passing year, the old products were finalised and finished. If anything is reused, it’s generally the finished master that gets reprocessed. Source materials are rarely reworked, so dealing with the built-in technical debt of obsolete formats or technologies doesn’t arise. What this means is that when the film or music producers start on the next product, they can start with a clean slate and an open choice of which technologies to rely on. They don’t have to unpick existing mixes or edits to make new ones. There is no compromise of having to make the new work encumbered by old technologies and formats, or else trying to bridge the quality of the new to the old. Even if an album is remastered or a film re-released as the director’s cut, it’s usually only the finished, finalised master work that is being modified, so the scope of the technical debt is limited to dealing with technical quality and consistency of older finalised artefacts.
Software product development, in contrast, trips itself over by either having to support and work with obsolete technologies, long after it makes business sense to do so, or else having to somehow glue new and old technologies together in a kludgey way. Refactoring is hard, because the software was not written in a modular enough way. Dragging the old technical debt around with the new product is like trying to sail fast with an anchor bumping along the bottom. Forward progress takes much more energy and time than it ought to.
The film and music equivalent, in software development, would be to instead take finished, finalised modules of software and wrap them in an API, so that the internals of the old functionality continue to work as before, untouched, but with the new parts of the product realised with the appropriate technology and architectural choices for the present day. If an old piece of software functionality really needs extensive surgery, its analogous to re-recording the performance made by the original musician or actor, to correct a line or a note. It will never sit seamlessly within the old work, because people age and mature, gaining new skills and changing their artistic delivery. The better choice is to abandon any temptation to rework the old code using old technologies and instead, recreate it anew, with up to date thinking and techniques, creating the functionality you really require. Software reuse is only valuable if you don’t have to do very much to the code you’re reusing. Otherwise, consider it scrapware.
When teams complain that they have a large backlog of technical debt, what they mean is that there are a lot of software components they’re continuing to maintain and modify, which should have been scrapped and recreated using modern tools and techniques, long ago.
The Cutting Room Floor
Movie directors and music producers frequently capture many takes of any given performance, keeping more material than they really need, so they have options over what to include in the final work. In feature films, there can be two hundred times more footage captured than will be used in the final edit. Film and music producers do this so that they can choose from alternatives, selecting the takes that work best in the context of the finished product. They think about capturing the performance from different angles, with different composition of the frame and with different lighting. In music, the musician may try recording the same part with different guitars, microphone placements or effects, for example. Sometimes, the take that works best in the overall finished product isn’t the best individual performance. It’s usually the one that contextualises most comfortably that is kept.
To make the final product be as good as it can be, producers and directors often discard perfectly good scenes entirely. The performances, no matter how perfectly executed by the artists, wind up on the cutting room floor. If the contribution doesn’t serve the story or the theme of the record, then it’s ruthlessly excised.
In software product development, every piece of code written seems to ship. There are no alternatives to choose between and its rare indeed that a chunk of perfectly good code doesn’t make the final cut. Everything written gets used. Wouldn’t software products be better if there was scope to make more software alternatives than you need, choosing the best solution further downstream in the production process? You could choose the solution that works best in context, even if it wasn’t the best piece of code. It ought to be possible to throw away inferior solutions, or solutions that don’t sit well with the rest of the product.
Being able to choose from alternative solutions would let the integration optimise on user experience, for example. Who is to say that the programmer’s first approach is their best approach? Their third attempt might be much cleaner, easier to understand, cheaper to maintain and work better than their first go. Shouldn’t they, as coding performers, have the same opportunity as musicians and actors to warm up and give their very best work on a later take?
Similarly, if a feature isn’t all that great, it ought to be easy to take the feature out completely, before the product is finalised. Just because it was written doesn’t make it sacred. A product manager could adjudicate, with input from other stakeholders, whether what the developers made is worthy of inclusion in the product at all. If it falls short, then excluding it ought to be possible, simply by not integrating the code into the release candidate. Taking stuff out is a great test of modularity and coupling, too. When developers have the expectation that features and code can and will be removed, for the betterment of the finished product, they will code accordingly.
Promoting and Evangelising
One of the things that software product development organisations often struggle with is developing “product sense” in their programmers. Product sense is the ability to create solutions that have a high likelihood of acceptance and success in the market just by knowing the right thing to do. You only attain product sense by spending a lot of time with customers, learning to do what they do with your software.
Organisations also find it hard to get developers to speak directly with customers, to empathise with what their users are trying to accomplish and to code solutions that address the real jobs to be done. Music and film doesn’t have this problem, quite so much, because of their method of introducing newly-released products.
When a new product is released, actors and musicians go on the road to promote and evangelise it, meeting their audience face to face, doing interviews and talking about the product endlessly to the media. Having to explain (and sell) their work to the paying public is a great way to have them focus, professionally, on doing future work that people will accept and admire.
Sending developers on the road like this, to promote a new software product, would be a good thing to do. It would get them off the endless coding treadmill, for a while, and let them experience reactions to their work, first hand. Marketing could develop collateral and marketing materials directly from the authoritative sources (the people that wrote the code) simply by recording what the developers say in these scenarios and the clarity that developers try to put into their solutions could be shared with potential users. It’s great brand-building stuff, but more importantly, a great opportunity for professional development.
Does Method Matter?
I hope you are convinced, by now, from the above examples, that there is a lot missing from the standard software product development methods currently in vogue and that the method you choose is crucial. When a creative guiding intelligence embarks on a software product development of their vision, the method itself will significantly influence what can be accomplished. Look at any room full of portrait painters and you’ll see multiple methods for making a painting, but each one constrains the gamut of possible outcomes according to the method they choose to follow. Not every painting method can produce any painting. The method makes the art. It will determine what you will pay attention to and what you will ignore. Today, software development methods, including most agile frameworks, ignore important aspects of making a great consumer product. It’s no wonder their products are so heavily criticised by their customers so often.
Essentialized or not, there are practices missing from the popular software development methods and there ought to be myriad methods to choose from, many of which you create yourself, because you’re innovating.
Method choice and the collection of practices applied within that method is a creative decision. We have no expectation that two record producers (or musical artists) making their record will use the same approach. We have a similar lack of expectation that two film directors will approach the task of making a film using the exact same method. Why, then, does the software development industry continually search for the one true and righteous software development method to end all software development methods?
Also, nobody ever said “Gee, I must go and see that movie, or listen to that record. I heard it came in under budget and on time.” (with sincerest apologies for mangling the quote, to film director Billy Wilder)
Honestly, which of your customers has ever bought your software solely on the basis of you having done the same?
Closing Thoughts
What Essentializing does for software product development is create a library of modularised practices, which scale. It provides adaptive ways of working and active guidance to practitioners. This common language has universalised progress indicators and provided a way to drive measurable performance improvements. Sadly, these metrics are usually about maximising extractive profiteering, wringing the very most out of developers for the least cost, or giving the customer as little as you can get away with, calling it a minimum viable product. There are better things to optimise on and film and music still do (e.g. emotional impact, entertainment value).
Does music production and movie making need this kind of rigorous formality to describe what they do, or are their performance metrics more about audience satisfaction and artistic merit? Isn’t it all about how it makes an audience feel? How would that be measured, other than by public acceptance and critical acclaim? Isn’t making people happy and helping people live better lives the point of software-based product design too?
Software development methods apparently exclude equivalents of practices that film and music production find essential. Software products are all the poorer for these omissions and neglects. It’s to the detriment of everybody that these practices (i.e. their analogues) are not part of any software development methodology. Adding and essentializing the equivalents of these practices and the methods that use them is probably worth doing.
Finally, fixed, one-size-fits-all software development methods are a mistake, because the methods and practices that you choose, along with the performance metrics you associate with them, are crucial creative choices, which drastically affect what your product can become. What you throw away is as important as what you keep.
Product development can learn a lot from music production and movie making.
About the author
Michael Topic is a freelance Product Manager with over thirty years experience delivering products that didn’t exist before. He welcomes contract enquiries to define new, competitive products, design them and deliver them. His speciality is software-based products.
Disclaimer
The opinions offered in this article are intended to describe common scenarios that sometimes occur in general product management practice. They are in no way intended to be read as referring to any particular employer, past or present.
