Be Skeptical of Standard Methods

Standards can be a real time-saver when used appropriately. Adhering to the right collection of standards for a project can save a lot of blood, sweat, and tears. But the wrong application of standards can have as bad of an effect on a project as the right way can be good, resulting in wasted effort, delayed projects, and general frustration. The difference between these situations often boils down to experience and judgement.

Of course, 'standards' is a loaded term. We could be talking about anything from communication protocols to development processes or anything in between. There are industry standards that are in widespread use and proprietary standards that don't even extend beyond a single department or team within a company.

Some standards provide great value because they establish a way of doing things that all interested parties can depend on to work with reliable results. Other standards stifle creativity by restricting possibilities too much or they create obscene amounts of useless work by trying to be everything for every possible situation.

In general, industry standard communication protocols are known quantities, and it's more or less straightforward to choose the ones you need for a given project. While it is certainly possible to make bad decisions in this arena, they're not going to be on the order of rolling your own protocol for connecting to a web server or transferring files across a network (I hope). Those are not the types of standards we're going to worry about here.

On the other hand, an industry standard development process is an oxymoron. There is no such animal. Even the various well-documented Agile methods are implemented in wildly different ways from one company to the next, and those differences matter. Companies can choose between Scrum, Extreme Programming, Lean Software Development, Kanban, or many other variants. They can mix and match different aspects of many of these methods. The possible variations are as numerous as the number of companies that use Agile methods. As a result, it's quite easy to claim to use Agile methods while doing nothing of the sort.

Regardless of what kind of development process a company uses, the main purpose of a standard process is to avoid as many mistakes as possible, whether those mistakes are software bugs, broken features, or deployment errors. It may seem that junior developers would need this kind of structure to help them avoid the innumerable pitfalls that are strewn throughout the development process, but as they gain experience, all of the processes, protocols, and reviews become extra overhead that provides less and less benefit. For senior developers all of this structure confines thinking and saps creativity.

My previous company designed ASICs, and on any given project we had a series of reviews that gated each phase of the development process. (It's much harder to practice Agile methods in ASIC design because each release includes a six to eight week wait to get real silicon back to validate, but we did work towards Agile methods in many ways.) As part of each review, the team needed to run through an extensive checklist of design issues and confirm each item as either verified correct or not applicable to the design.

Some of these checklist items were good design practices that were worth remembering, but they would have been more useful as guides throughout that phase of design than when we were trying to pass the gate into the next phase. Other items made no sense at all because there was no context for them. Some team somewhere in the company was bitten by some issue at sometime in the past, and this checklist item was meant to make sure that never happened again. The problem was that no one else had any idea what it meant or why it had earned a spot on the checklist.

Once useless items start creeping onto company-wide review checklists, they degrade the validity of the entire checklist. Developers get confused about which items apply to the design they're working on, and they start glossing over more and more items until they start rubber stamping the whole thing. Then the review becomes a tedious exercise in hand waving and rationalization that results in endless developer frustration and destructive management skepticism.

The original intent of these checklists was good, but as soon as they started trying to cover too much, they stopped being a tool that developers could use effectively and became an annoyance that developers dealt with tangentially to the project. Instead of enabling good development practices, the checklist creators tried to dictate those practices and micro-manage the development process.

A flexible and efficient development process will provide primitives, not solutions, much like a RISC processor is more flexible and efficient when compared to a CISC processor. A CISC processor tries to be all things for all programs by including many complex instructions that do specific high-level tasks like procedure calls or list sorting. The added complexity ends up wasting silicon area on instructions that are hardly ever used in most programs.

On the other hand, RISC processors attempt to provide a much smaller set of simple instructions that are widely applicable and more easily optimized by compilers. The extra silicon area freed up from not having complex instructions can be spent on things that improve the efficiency of most of the remaining instructions, like caching, pipelining, and branch prediction.

RISC processors provide primitives, not solutions. It's up to the software developer and the compiler to most effectively use those primitives to build the best solution. Passing that control up the stack where there's more contextual information results in cleaner, faster programs.

A standard development process is similar in that the context of any particular project can't be known when the process was created, and the people that create the process may not even understand which methods will make projects more successful. The development team knows the context, and they need to use the process in the context of their project. Smart developers will be able to use a flexible development process to build great software. An overbearing process filled with mindless checklists, inane procedures, and verbose coding guidelines will only slow them down and reduce the quality of their output.

As for the junior developers that are normally the justification for detailed development processes, company standard processes will not save them from making mistakes. Making mistakes is how they learn. A bunch of formal processes are not going to magically create expert developers. They need to gain real project experience and deal with setbacks and slip-ups before they will have internalized the knowledge and skills that make up a senior developer. Having a good mentor will go much further towards this goal than instituting standard development methods and expecting good practices to rub off on them.

Great developers have great judgement. They know what to do in a wide variety of situations to create amazing software with a minimum expenditure of effort. This kind of judgement cannot be captured in lists or guidelines, procedures or protocols because every situation is different; every software program is unique. Standards cannot possibly encompass the diversity of thought and experience that goes into software development, nor should we want them to. While standards certainly have their place and can be useful guides, standardization cannot create greatness. It can only achieve consistent mediocrity. It takes good judgement and creativity to elevate that mediocrity to greatness.