Catalysoft   Turtle
home products articles about us contact us

Recent Articles

What's Good About Clojure?

Clojure is a relatively new language to appear on the Java Virtual Machine (JVM), although it draws on very mature roots in the form of the LISP langu ...

Should You Care About Requirements Engineering?

Recently, I (Adil) was invited to participate in a one day seminar on the subject of Requirements Engineering. Whilst I have no direct experience of t ...

Tips for Setting Up Your First Business Website

To attract all potential customers to your business you need a presence on the web. The problem is that if you haven't set up a website before, you p ...

What's Good about LISP?

LISP is a general-purpose programming language and is the second-oldest programming language still in use, but how much do you know about it? Did you ...

Open Source Tools for Developers: Why They Matter

From a developer's point of view use of open-source tools has advantages beyond the obvious economic ones. With the open-source database MySQL in mind ...

Practising Best Practises in Your Software Development Process

Discuss this article >>>

Overview

Why is so much software developed chaotically? Why are best practises ignored? In this article, I first explain the traditional view of why you need a software development process. This is, in itself, no big surprise. I therefore question why so much software is still developed chaotically, either without a process or with disregard to best practises. I suggest The Knowing-Doing Gap as a possible cause.

What is a Software Development Process?

It cannot be stated too often that if you are developing software, you need a process. By process, I mean a guideline description of a repeatable procedure that both describes and prescribes the way that your organisation develops software. It describes the process because the process that works best for your organisation is different to the process that works best for another organisation, so the process says something about how your organisation works. It prescribes the way that your organisation develops software because it is the process that will be applied to the development of future projects.

It is not sufficient just to say that the process is simply

Requirements->
  Design->
    Implementation->
      Testing->
        Documentation->
          Maintenance

as this leaves far too many questions open to interpretation by customers, managers, and developers alike. For example, what kinds of requirements are gathered in the first phase? How detailed is this analysis? Does the design phase wait until the requirements phase is completed before it starts? How is the usability of the end-product ensured? How is the assessment of usability represented in the phases given above?

Of course, every organisation will have its own answers to these and other questions. The answers to such questions should be recorded in some sort of a document, say a process manual. Then, if a potential customer asks the (almost inevitable) question "How do you develop software?", then you can demonstrate not only that you have thought about it, but that you have a well documented, mature process that works for you.

Different organisations have different requirements for their process, so there is not one "correct" process. That's why, in this document, I can't tell you what your process should be. In a sense, it doesn't really matter what your process is, as long as you are confident that it works for you. To be more precise, all the stakeholders in the process need to be confident that it will help them to achieve their goals. In bespoke development projects, the customer needs to respect that this process is the one that best suits your organisation, people and products. The managers in your organisation need to believe that it works for them, so that they continually refer to the process and do not allow the project to be diverted. The developers also need to believe in the process so that they can work together as a team, respecting the boundaries between different parts of the process and the implications of the process on their roles in the overall development.

A process can be lightweight to offer flexibility and minimal overhead to a small development team, or it can be more heavyweight to provide firmer expectations of the way that the project will be run.

Why You Need A Process

Your process is your guide through a landscape of highly complex, interdependent activities and sub-activities. During the lifetime of a project, there will be many tensions and pressures that try to divert your attentions and efforts from developing good quality, usable, software that is delivered in a timely fashion. However, keeping your eye on the process should help you to focus attention on what is important.

Lastly, but critically, the process must be maintained. A software development process is not a recipe that, once written down, is never questioned or changed. It is a guideline that incorporates much of the knowledge from previous projects. Maintaining your process is your opportunity to feed back the successful, or otherwise, experiences of projects into the future projects of your organisation. An organisation that cultivates, respects, and maintains its software development process is a learning organisation.

Why a Learning Organisation is Not Enough

It would be easy for me to stop here and say "If you already have a process, well done, you're on the right track. If you don't have a process, go and get one, otherwise you're doomed." The truth is, most developers already know the mantra, so I wouldn't be saying anything new if I stopped there. So this really begs the following question:

If developers already know the importance of a process, why do so many of them develop software so chaotically?

By 'chaotically', I simply mean in an ad hoc fashion, without following a process. If we think about this a little more, we realise that developers fall into the following three categories:

  • those that operate without a process;
  • those that subscribe (and adhere) to a process;
  • those that know their process but do not implement it, or only partially implement it.

Which of those categories are you in? Given the mountain of available literature on programming best practices and how to manage a successful software project, yet simultaneously the number of stories of software projects that have overrun and failed, I conjecture that most developers fall into the last of the three categories. That is:

Software professionals know what is best to do to develop software, but they don't (always) do it!

Let's think of some examples. Do you always comment your code so that the next programmer can understand it? Do you document your program designs? Do you document all the decisions that you make, with justifications as to why you made the decision that you did? Do you build your project's latest source code frequently to uncover bugs early? Do you analyse what went wrong when aspects of the project were not successful? Do you communicate lessons learned to other members of the team, group, department, or organisation? Do you document surprises that occurred during the project, and feed your ideas of how to prepare for such surprises back into the process? Do you fix errors in your code when you discover them or do you sometimes 'bury them under the carpet' in the hope that they do not recur?

OK, so none of us is perfect. But I believe that developers (and others involved in software projects) consistently underachieve because they don't do what they know to be the best practice.

The Knowing-Doing Gap

The phenomenon of practitioners not doing what they know to be best practice is not peculiar to the software industry. In their book of the same name, Jeffrey Pfeffer and Robert Sutton call it 'The Knowing-Doing Gap', and cite many instances taken from a wide range of industries. They argue that the most important impedance to progress and competitive advantage is not ignorance, but the inability to turn knowledge into action.

So what prevents us from turning knowledge into action? Well, sometimes the cause is a basic self-deception - a belief that a pre-cursor to action is the action itself. For example, there is a big difference between talking about doing something and actually doing it. Other times, you need to add procedural structure to your knowledge to successfully translate from knowledge into action. For example, it's no good just having the goal of losing weight, and knowing that you need to plan your diet - you need to attend to your diet as part of your daily routine.

Pfeffer and Sutton cite the following as the main causes of the Knowing-Doing Gap:

  • Substituting talk for action - Talking about what should be done, writing plans, and analysing data to help make decisions are all useful precursors to action, but are no substitute for action itself.
  • Using Memory as a substitute for thinking - People often do what has always been done without thinking. The organisational memory in the form of company customs, standard procedures, or personal experience of problems solved in the past are often misapplied to a new problem.
  • Fear prevents acting on knowledge - In some situations, people might know what action they should take, but the fear of its consequences prevents them from doing it. The action might be complex, or simply new, different and unpredictable. The resulting fear is a fear of risk, of making mistakes, discovering errors and providing an imperfect end product. There might also be fear that the competition could get ahead in the meantime, a fear of the reactions of others, or a fear of making a fool of oneself. In the face of such fears, the safest and most 'rational' action can be inaction.
  • Measurement obstructs good judgement - Decisions are made based on what is measured. If inappropriate things are being measured or the choice of what is measured is biased, then the resulting statistics will cloud good judgement.

Bridging the Knowing-Doing Gap in Software Development

Having explained the causes of the 'Knowing-Doing Gap', let us now consider some situations when it might arise during software development, and how the effects can be mitigated.

We all know that devising a project plan is not the same as executing it; knowing how to solve a programming problem is not the same as solving it; and reading about a new technology or algorithm is not the same as using or implementing it. These are all necessary pre-cursors to action, but are no substitute for the actions themselves. If the action is important, then make it a formal part of your software development process and, as part of that process, ensure that it actually happens.

Software development is not like a factory assembly line, in which every component is, and should be, just the same as the previous one. As developers, we strive to achieve a certain level of repeatability, since this makes the process much more predictable, but slightly different problems and problem contexts often demand different solutions. You should therefore identify and challenge your assumptions early in the software life-cycle, so that the implications of lazy thinking can be addressed before it becomes too late. You should also encourage peer reviews of code as a part of your process, so that ground-level assumptions and practices do not go unchallenged. At a strategic level, changes in the market-place can also make it dangerous to use memory as a substitute for thinking. Organisations need to be agile and react to changes in the market-place to maintain their market share.

How often have you been scared to change code because you are afraid of the consequences? You need to make a change, but maybe you don't completely understand the code and the guy that wrote it has left the organisation. Your fear is preventing you from achieving the best result. To avoid such a situation, encourage joint ownership of designs and code in your team, so that many developers can maintain the code base. And when it comes to the crunch, be brave! (…but make a back-up!)

It is common knowledge that what gets measured gets done, and what is not measured gets ignored. Therefore in the very act of measuring anything at all, there are obligations to your organisation. You must make sure that what you are measuring reflects and contributes to the goals of your organisation, as well as reflecting the importance of both long-term and short-term goals. So, for example, measuring the number of lines of code written by developers is a bad measure of productivity, bearing little relation to project progress and no relation to the fitness-for-purpose of the end-product. Also, what does it mean to say that a task is 90% complete, when the last 10% always seems to take as long as the first 90%?

To conclude, note that, perversely, the phenomenon of substituting talk for action also applies to other activities, such as attendance at lectures, where students might learn the 'what' but not the 'how', and reading articles, which might leave the reader 'knowing' more, but rarely changes working patterns. Of course, this article is no exception, so many thanks for taking the time to read it, but I should surely be surprised if it changes your software development practises.

Discuss this article >>>


Simon White