Category: cognitive bias

Small ‘a’ agile

June 19, 2024 /

It’s not the rituals, but the reason you do them that matters

Sailing ship in a storm
Image by Andy from Pixabay

Some history

The term agile as it’s used these days in software development came out of the publication of the Agile Manifesto in 2001. A group of experienced software developers came together and wrote some principles on a whiteboard. Most of these folks are now regarded as luminaries in the software business.

Why did they write the manifesto?

Before this most people worked on the assumption that software was an engineering discipline. If you think how most engineers work they have tightly controlled processes and books that say what the best approach is to building a bridge or a bend in the road. If you think about how complex projects like building ships are done then the emphasis was first to do all of the analysis (requirements), validate it somehow (check the designs), and then write the software (build the ship). The process of discovery was done in the analysis phase, then it was designed, and the software was almost supposed to write itself. Think create architect’s plans and then build the building, along with strict processes to make it happen. Even then it never goes smoothly.

The problem is that software is only 60 or so years old as a discipline, and has only been practiced by large numbers of people for maybe 45. We don’t have all of the knowledge that several hundred years of proper engineering gives us. Most of the time we’re still trying to work things out. We’re still inventing new programming languages and frameworks to find the best way to solve certain kinds of problem, there is no right way to do things in software. There is no best practice, just knowing what worked last time and building up experience to tackle common problems. As far as platforms go the internet itself isn’t actually that old, modern JavaScript frameworks are very new, in fact, and also constantly changing as people fix mistakes and bad thinking. We often discover better ways of doing things by writing the software itself. Sometimes faults in the specifications come up when we try to build things and we realise they’re full of contradictions.

Writing software isn’t the same as traditional engineering, despite people often being called engineers. It’s much more like a craft that you learn from experience, it may seem like something programmers do each on their own but getting good at it requires feedback. There are no recipes, if there were then programmers wouldn’t be needed. You could automate it. A recipe is an algorithm and can be automated if you have the right kind of tools. This is what the AI pushers claim can happen, but who then validates the code is correct and isn’t going to kill anyone, a different AI?

Additionally the engineering-style methods were often extremely bureaucratic as a function of the big companies that developed them. You often find that people criticise what is called the waterfall method that rigidly went from analysis to design to build to implementation, but in fact that characterisation is unfair to the person who first commented on it. He did put feedback loops in between the various stages, it’s just that the people who picked up the idea didn’t see them as being necessary, hence all of the bureaucracy around managing change. Highly controlled processes are necessary when you’re building military or medical software, there needs to be a high level of accountability and traceability if lives are at risk, but the other 99% of us can be a little more laissez faire.

All of the separation between the stages in the process that came from the bureaucratic approach also meant that the people who understood the problem from a business perspective were often very far away from the people who were trying to write the software to deal with it. This meant that we hit the big problem we always hit with software, or any complex endeavour:

All the problems you encounter are, at the base, communication problems.

  • A bug is a product of misunderstanding
  • Incorrect functionality is a misunderstanding
  • Going over budget is usually due to a lack of knowledge, assuming the budget was set correctly by people who knew what they were doing.

So the quest for agility, which is where we should be paying attention, is ultimately a quest to remove barriers to understanding and communication. With all of the things in that list, you should always be trying to identify how the problem arose and how to stop it happening next time. You don’t need to inspect quality after you’ve shipped it, you need to know it works for the people you wrote it for before it even leaves the gate.

The core of the agile manifesto

On the website it says

We have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Some thoughts on the core

The first is actually quite against what passes for an agile approach these days. We have more tools and processes than we can shake a stick at. This is because they’re something that experts can sell to the rest of us. There are a few core concepts I like, and the rest is window dressing to make money for gurus.

The second working software point has been consistently misunderstood – it doesn’t mean no documentation, it means enough documentation to get you started, explain any gotchas, and the approach to any complex processing you’ve identified. I’ve become extremely tired of arriving on a project to find the only documentation is the code and the code was written by blind squirrels on acid who stopped doing tests because it’s faster. Save me from this lazy bullshit.

Customer collaboration is contained in the idea of having a business owner, but also someone to bounce ideas off. Far too often I’ve had to deal with the vague needs expressed in a Jira ticket that contains a couple of sentences and the person you need to clarify things with is too busy. Don’t let this happen to you. It also helps you work out what’s important to the people you’re writing the software for.

Responding to change means being willing to stop doing what you thought was important and instead work on what is important. It’s very hard to do if you’ve got embroiled in the sunk cost fallacy. It also means you need to regularly review what you’re doing and be really honest about what needs to happen. It’s also why you need to concentrate on being effective rather than efficient. Heading in the right direction is way more important than heading in any old direction as fast as you can because it feels good. Sometimes you have to slow down and do a lot of thinking to speed up.

The birth of methods and people selling consultancy

The drive to bring what’s needed in the software to the front and boost understanding of the needs of your customers is why we have the rituals we do in things like Scrum. The rituals alone are irrelevant. The understanding they generate, and the documentation that should be part of the process, are what’s important. Mechanically doing things because it’s in the calendar is not a recipe for success. Knowing why things are done and making sure everyone has the cognitive space to engage properly with them is more important than the rituals themselves.

The most complete version of an agile approach is Extreme Programming (XP). The best book I’ve ever read on XP is The Art of Agile Development. It contains all of the practices and explains the pros and cons of using them.

XP was seen as expensive and difficult to implement because it needs heavy commitment to get it working properly, plus a real understanding of how to do work with other people. It’s still the gold standard, in my opinion. It’s also very developer and development focussed so harder for non-technical management to understand.

Several other ways of trying to encapsulate this need to communicate often and well were thought up, but the one that eventually won out was Scrum. Scrum is relatively cheap to implement and only has a few core ideas:

  • Sprints
  • Daily standup
  • Sprint planning
  • Retrospectives
  • Demo
  • The business owner

There’s a few ancillary things like burn down charts, but that’s the core. XP came from the developers trying to make things work better and it has TDD and pairing as part of the approach, where they’re usually add ons for Scrum. XP’s roles are also much more flexible, which makes it harder to sell the recipe to a willing audience because there isn’t one.

The project management iron triangle turned into a square

Historically there were three parts to managing a project:

  • Cost
  • Time
  • Quality

It used to be said you can pay attention to any two of these but not all three. One of the things that is an accidental product of the adoption of agile ideas was that people realised there is another leg to this, scope. If you manage the scope well you can deliver on all three of the other legs, as long as you’re willing not to deliver everything all at once. It’s also why the methods use short iterations, it means that you can discover what scope matters without having to redo all of the mountain of analysis doing the entire thing upfront would require.

There’s also some other things to do with scope, for example doing the most risky things first, because if they can’t be done the rest of the project is impossible. Sometimes you need to do the least risky to help you get all of your testing and deployment infrastructure working. The point is all decisions about the approach should be deliberate and not because a book, or course, or coach, said so.

Capital A

In the early days people used to say they were doing Agile. A lot of us used to find this a bit worrying. It seemed that it was something you could turn on and off. It also fitted into the whole thing about doing the rituals and following the rules, without understanding why they were there.

Small a

Instead the push is about being agile. Making it the way you are, and having organisational behaviours that improve what you have. You may choose not to do certain things that are canon because of your circumstances. The book on XP mentioned above says that you should do all of the things it recommends for several months before dropping any of them or you will never understand their value properly. Despite my saying you shouldn’t take things just from a book I think it’s good advice.

I used to call myself a recovering agile practitioner and be very anti things like regular stand up meetings if they weren’t needed. Doing things deliberately is the way. Doing things because everyone else is doing them is not.

Steady cadence

One of the goals is to have a steady rhythm to the work, so it gets done but without lots of stress. Again, short iterations support this, you can see things being built. Not having to deliver something for six months is a recipe for disaster because humans are bad at thinking that far ahead and it’s very hard to track things over a long period.

Stress from poor planning engages the fight or flight mechanisms of the body. The energy is moved away from your brain so you can deal with trouble. You’re instantly stupider than you were. Taking time to think things through and not rushing and forgetting things ends up with a far better outcomes. The silly mantra you see in job specs, where it’s said be able to cope in a high stress, high pressure environment when that means you’re going to be acting stupid and going along with bad ideas because you’re exhausted is silly.

Estimation and expectations

There is a great tension in software development that often goes unacknowledged. You may remember the quote

There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.

Traditional businesses invest and expect a return on that investment. A well functioning business in a stable market is like turning a handle. You put an amount of money in, turn the handle, and out comes more money than went in if the business is built correctly. Of course this can stop working if the environment the business works in changes dramatically. Just look at what Covid did to people-facing businesses.

Software development is inherently risky. You’re answering a question when you write it, and the answer may be much more complex than you thought it was. For example, a business may want to invest in getting to a new market. If you’re used to thinking in the first mode you may assume that building software is turning the handle and giving you the new market automatically. Nothing could be further from the truth. The process of developing the software uncovers how you might connect with customers in your new market. Most of the work is discovering how to create a turnable handle, and the software is actually a minor part of this. Unless you are replacing or copying something well understood it will take a lot of effort to make things work for the business. Then the people using the software need to know how to use it to make the handle turn, this is not trivial.

In the early days of NASA they realised that flying to the Moon was something that had never been done before, so you can’t estimate how long things will take. This is where the concept of milestones came from. You say at a high level when you’d like something to exist, and then you can see how far along you are compared to what you expected. When you’re working with unknown unknowns it’s always a guess, and great care is needed to manage expectations. I always add about 30% to every budget and it always gets used on things you didn’t anticipate when the project began. This seems to be a rule that I’ve never managed to break, unknown unknowns will always bite you so be ready. Also put the contingency on the whole project instead of padding each task you’ve identified because it will confuse everything and you won’t have any idea how much contingency is left.

You have a tension between the people who do the budgets and want things to happen by certain dates etc. and the reality of exploring the unknown and drawing a map, which may prove to be only a partial one, as you go along. Again this is why short iterations are needed so it’s easy to spot if things are going awry so we can negotiate with each other to see what can be achieved by flexing the scope or doing something else that’s more profitable.

Retrospectives – get ahead of the crowd

The word quality has been mentioned a few times here. One of the things people often drop in the rush to get things done is retrospectives. Without them, you are never going to improve except accidentally. If you want to create software with no bugs you need to look at your processes, and listen to the whole team. I was trained in a quality management system called zero defects in my first job after graduating in 1986. The concept is simple, instead of allowing things to be released that you know aren’t right make the time and effort to make sure they are. Then you need to hold retrospectives so that the learning that went into fixing it the first time stays learned. This is also related to double loop learning.

Conclusions – set yourself free

The main idea behind this article was to give an outline of the history of the agile movement and where it came from. As someone doing a difficult thing, such as writing software, you are not obliged to “do” any of the behaviours and prescriptions from the agile movement. Instead, think about how to get the best results for you. If you can’t get these ideas working in a small team, creating the steady cadence, and being consistent you will need to work out why. Adding new processes or documents or rules will not help. You need critical thinking and an understanding of why things are done to make it work. You also need to work with the whole team, there is no ivory tower way of imposing true agility from on high, or by manipulation of spreadsheets. It’s a communication and people problem, so you need to communicate with people.

Survivor bias: an experiment

August 4, 2014 /

I write a simple computer program that randomly selects one of two outcomes. I send out letters telling people I can pick stocks and shares that will go up. About 50% of the people I write to think I know what I’m doing and think they might buy subscriptions to my stock picking service.

I do the same thing with the 50% that got the right answer, each time making an ever smaller group think I’m a stock picking genius.

Eventually I will run out of people, but right up to the last two it will appear to them that I really knew what I was doing, when I was effectively tossing a coin.

In the mean time quite a few of them may have subscribed to my stock picking service. Good for me, not so much for them.

The people at the end of this chain of probabilities will think that they are kings of the world, when they are only survivors of a simple process that could have picked anyone from the original group of stock buyers.

I recently attended a virtual course on complexity and discovered the fun simulation language Net Logo. For historical reasons the little actors displayed on the screen are called turtles. I could just as easily build a program that has a population that halves every turn. Is the lone turtle left blinking on the screen at the end a special turtle, did it somehow avoid the grim reaper against the odds? No, obviously not.

If that turtle was instead a person? The story it had to tell would be a story the rest of us would want to emulate because it was a survivor. This is why the biographies of the heroic entrepreneurs are often not that surprising. This is why their opinions quite often don’t differ a lot from those of others in the same cohort.

I used to work for Oracle and read the unauthorised biography of Larry Ellison – it’s an entertaining read and certainly not very complimentary. One of the things that comes through is luck, Oracle came close to going over several times, literally so close that a single large order from Japan saved it. I was talking about Richard Branson with someone recently. If I remember it right he originally had the idea of opening his record stores near tube stations late at night so people travelling home could buy records, he also had an aunt who lent him the money to open these stores. Great idea, but he didn’t have to go to a bank and get laughed at. Again, when Steve Jobs early Apple got some venture capital so they could make things happen – whoa, things started happening! (Thanks to Tim Spencer for this one).

There is an unknown population of other people and businesses that didn’t make the cut, or that stayed small service companies that are still around but not mega corporations. Probably 99.99% of them. Jobs himself acknowledged this with the famous analogy that the things joining the dots together are only apparent when you look back and make a story of them. In my last post I talked about pareidolia, which is the human tendency to invent patterns where none exist. We can all find stories up like this; if I hadn’t answered a job advert in the Independent I wouldn’t have met my wife, my kids wouldn’t exist and, like, wow man (sarcasm off). We make a causal chain of events but forget the massive part that chance plays in what happens to us. If the University had placed the ad in a different paper on a different day, who knows?

Survivor bias makes the winners’ stories compelling, but there are another 9999 (or many more) stories of other people we never hear. Remember this, next time someone tells you that you must emulate this or that hero of theirs. Success needs an element of luck, and the same person may not be lucky twice. Napoleon used to ask of a general is he lucky? He was no fool. We all love 37 signals’ (now Bascamps’) story, but there was luck there. Lots of people have tried their formula without getting what they have, or even losing everything.

This isn’t meant to sound like a doom and gloom principle, far from it, what it is saying is you need to find your own way that works for you. You also need to take that opportunity when it falls into your lap and do something with it. But you aren’t any more special than anyone else. It doesn’t matter. What matters is being clear about what you want and sticking to it.

Generic is the wrong answer

July 27, 2014 /

Dear reader, cast your mind back to the time when you first wanted to put some information on a website.

You had to trust the crusty programmer types, and if you wanted changes they wouldn’t necessarily be able to do them at any great speed. This was fine in the stone age, around the year 2000 or probably earlier. Then, one day, the technology started to catch up with the aspirations and everything had to be done at internet speed. So crusty developed less crusty tools that let things happen quite a bit quicker.

We suddenly had content that people created for themselves. Never to be outdone the constant search for simple generic solutions to be everything meant that the companies that made the very expensive PC based tools for document management had a go at creating something. Some of their original big corporate customers kept on with them. These systems don’t work for the internet, except when you are creating some kind of library, which is the metaphor they started with.

The more considered approach was to treat managing content like it was a solved problem. The content management system (CMS) was born (or at least became more mainstream) and allowed you to manage your catalogue in one place so you could, you know, sell your stuff to people that wanted it.

Then we fell victim to the illusion of best practice. Problems are in a space where they seem to be the same, so we (software engineers, academics and other people who should know better) develop generic solutions. I mean, content is just abstract stuff that we let those tricky user types create so they can pay us. We can just pull something open source or off the shelf and say off you go to them and get back to important stuff like arguing over which 20 year old text editor is best.

But here’s the thing. The pattern user -> catalogue -> put catalog on the interwebs in front of customers is the same, but the user / business / enterprise is most definitely not. Most (not all) open source CMS were created for a small community to share info – they do that really well. However, a commercial company that sells white goods does not have even remotely the same needs as one that sells online holidays where the stock is a multi lingual database of hotels they have a relationship with. They need content management, the thing, but not in the same way, and the OSS solution is probably completely wrong. Also – you don’t want your site to look exactly like your competitors or you are competing on price, which is suicide.

Human beings are subject to pareidolia. Seeing patterns that aren’t there in the data. This is also related to clustering, confirmation and congruence biases. When it meant running away from the wind in the grass because you thought it was a lion that was ok. Now we look for levers to fix and change things that often don’t exist. Eric Reiss calls this robots in the Lean Startup. He cites the case when the US manufacturers couldn’t work out how the Japanese were getting such massive productivity gains so they went to see and saw robots. After a robot installing frenzy the US was still far behind. The Japanese spent a lot of energy understanding how they did things first and put the automation in where it made sense, gradually, and always testing the results, for their specific needs. Good practice, but hard to emulate without a complete culture change. Pareidolia (and the other biases) put robots inside processes that were still broken. In knowledge work this is even easier because adopting the latest fad is only the cost of a couple of books and some (more) training, until your organisation fails catastrophically. Danger, Will Robinson.

Best practice only works in very simple environments where all of the problems are completely understood. Buying a CMS from someone assumes that they understand your needs properly. They probably don’t. It’s dangerous.

Good practice means the using the skills of analysing the problem properly and divining the principles you should follow. A company should control its own content; after all that’s what it’s selling. This means layout, search, everything. However you might use a markup language and a standard group of elements for layout to make it easy for non-web heads to create that content. Find levers for the easy stuff, for the things that do have generic solutions that work.

Startups don’t want to spend money on stuff like a CMS that is tailored to their needs and will tend to go for the OSS solution. I think that they need to start small and find out what those needs are, just like everything in the lean approach. So don’t buy one, and don’t slavishly pull in an OSS one that will bite you and turn out to be an expensive mistake. Create something as simple as possible that meets your needs, then you will know what the needs are and where the pain is. Then you can use good practice to get to where you want to be. Then you can find the metaphor that works for you, and let your competitors try and shoe horn someone else’s into places they don’t work.

Best practice implies generic – generic means cheapest and also (face it) BORING.

Good practice is searching for the right metaphors and being patient enough to wait for them to appear. And, of course, those metaphors are the ones that work for you, might not be right for someone else.

Why I hate todo list apps

July 5, 2014 /

I recently had a twitter discussion about why I don’t like using Basecamp.

It’s a bit weird, I love Rails and it originated from the need to create a stable framework to create Basecamp. DHH and his colleagues were generous enough to share it with the rest of us and I’ve been using it for years to great effect. I admire DHH and the guys at the company and have read their books. But I hate their product. Using it makes me feel depressed.

I can’t see the flexibility others see anywhere. They are a poster child for great design and easy UI and it’s pretty dull and obvious, maybe that’s what the fuss is about. I’ve used it off and on for years and it doesn’t seem to have substantively changed in all that time, yet there are allegedly changes and improvements happening to it constantly. It’s also boring, even more boring than the bazillion sites you see built on top of Twitter Bootstrap – because they usually have some metaphor the coders are exploring and bootstrap’s a quick way to make it presentable while they experiment.

But the main thing is I can’t stand todo list apps.

This comes out of the way I work and think. I will write todo lists on paper and cross them off, but for anything bigger than just me (and up to 5 items) I need to visualise it and look at the flow and commitment of work. You can’t do this with static text-based lists without doing a lot of counting in your head. As far as I can work out you can’t create a different visualisation (in Basecamp) where you have a private list taken from other lists of things you care about today in the order that matters to you without creating another list that isn’t linked to the original tasks so you’d have to close everything twice. It hasn’t changed in years.

In the physical world you can do this with sticky notes or file cards and a bit of blu tac to put each piece of work into something that can be visualised and moved about. It doesn’t have to be complex, but nicking ideas from scrum and kanban around allocating points for pieces of work and limiting your work in progress (WIP) you can come up with something that lets you look at your pipeline and your commitments so you can start to have grown up conversations with other people about priorities and schedules. Limiting your WIP also gives you a far better chance of actually getting something worthwhile done. This is a big win over boring lists, but harder to understand at first.

Todo lists are, I think, a great tool for linear thinking and controlling.*

The metaphor lists give you is the linear instruction manual, codes, all you can do is change their priority and create more items. They also suffer from being unbounded, so you can just keep adding to them and never finish. There’s a very interesting time management book by Mark Forster called Do it Tomorrow. In it I found the idea of a closed list. This is a list you don’t add to and work on until it’s done – he’s discovered this makes you more productive and also you start tackling the tasks you really don’t want to do because you can’t close the list off until you’ve done them. This is a great antidote to procrastination. It’s also limiting your WIP. In fact it’s a different take on the same idea as a personal productivity booster.

So, in essence, for me the list metaphor stifles your ability to think and visualise without a lot of effort breaking out of the mental straightjackets it puts you in. There are better metaphors that allow you to easily move work about and discuss it properly with stake holders. So use them a little for small tasks, but for the big stuff – use bigger metaphors. I also think gantt charts cause similar cognitive blindness when you use them for anything other than checking if you are still on track.

* Aside: I was going to use the phrase command and control. The original meaning of this was simply that military operations have a designated officer who is responsible for seeing the mission through. How that officer does it is not defined. Some systems thinkers, for example John Seddon, use this term to mean enterprises where the people doing the work are not expected to show any initiative and be told what to do all the time so everything costs a fortune and is badly done by people who could do a much better job if left alone to do it properly. Military operations don’t work like this, people are trusted to do what their training tells them so they can achieve their objectives, see commander’s intent. Seddon’s meaning seems to have triumphed in systems thinking debates, however.