Daniel Kahneman: Noise – the Flaw in Human Judgment

On 21 September, our friends at DenkProducties organized a hybrid masterclass with the iconic psychologist Daniel Kahneman. During the course, we dove deeper into the concept of noise. What is it? Why should we care about it? And maybe most importantly; how can we get rid of it?

Here, we’ll share with you some answers to these issues based on the masterclass.

What Is Noise?

If you are familiar with Daniel’s work, you probably know that in the bestselling book, Thinking Fast and Slow, he talks about biases—the predictable errors in judgment and decision-making. Noise, on the other hand, refers to the unpredictable variation in judgment and decision-making.

“The word noise, as we use it, is really borrowed from the theory of measurement. When you’re measuring, say the length of a line with a very fine ruler, and you measure it multiple times, you’re not going to get the same result. Every measurement is going to be slightly different from the others. And if you know the exact length of the line, then you can look at any observation and see it has an error. Sometimes positive, sometimes negative, depending on randomness. The average of these errors is called a bias, and the variability of these errors is noise. Both bias and noise contribute to errors. But even if on average the measurements are precise it clearly makes a difference that some measurements are way higher than the truth and others are lower than the truth. That variability is undesirable, and that variability is noise.”

To demonstrate noise in the context of work, he told us about a real-life case on noise that was the most shocking to him: the judicial system in the United States. Daniel explained, that through their studies they found that different judges give very different sentences for the same crime. The beliefs, attitudes, and even the mood of the judges affect the decisions they make. “The defendant that is going to court is really facing a lottery. The lottery being who is the judge, what are their beliefs and attitudes, and what is their mood on that particular day. The result of the lottery can be years in prison—plus or minus. This is completely unfair and unacceptable.”

Why Should We Care about Noise?

Daniel highlighted, that what he and his colleagues found through research, is that “the amount of noise and how it affects organizations is way larger than we actually think”.

In some jobs, there’s automatically less noise as there are strict rules and guidelines that limit subjective judgment. However, there are many jobs where individual judgments are required and often guided by informal experience and general principles. These are the kind of jobs where the amount of noise is high.

At the masterclass, Daniel talked about the example of an insurance company—the organization where he actually began his study on noise. There, he carried out an audit (now called noise audit) to find out the amount of variability in the judgment of underwriters.

First, they had executives prepare cases for underwriters; realistic and similar cases that those underwriters work on every day. Then, they asked about 50 underwriters to set a dollar premium for a particular case. Finally, they looked at the difference between two random underwriters to see what is the difference between their answers in percentages. The executives in the company were expecting around 10% difference.

Can you guess what was the true difference?

“It turned out that the answer was 50%—five times larger than those people expected. This is what motivated the book noise. Nobody in the company was aware that they had a noise problem, but in fact the noise problem was acute.”

You probably see now why you should care. Because noise means problems: unfair treatment, loss of time, and loss of money, to mention a few.

How Can We Get Rid of Noise?

Over the course, Daniel gave us a few practical pieces of advice on how to tackle the noise problem in your organization.

First of all, the good news is that according to Daniel, “you can eliminate noise completely”. Whereas biases are impossible to get rid of, noise is actually something that you can affect. Also, it’s important to know that noise is much easier to measure than bias.

So how to get rid of noise? Here are 3 things that Daniel mentioned.

1. Conduct a Noise Audit

This is where you should begin: acknowledge that you have noise and conduct a noise audit. “If you have multiple people in the same role, conducting a noise audit is the first step. If you are willing to be exposed to the amount of noise that you observe—which will almost certainly be more than you’re expecting—then you may want to do something about it.”

2. Break Down Problems

Daniel suggested that in order to solve problems, we need to break them down. You need to have attributes through which you look at each option to solving the problem independently. “We have a slogan: options in decision-making are like candidates. When you have a complex decision and you have multiple options, you can treat them like candidates. You can look at their attributes and try to evaluate the attributes independently of each other. Then, you can try to delay intuition and delay the final choice.”

3. Take a Lot of Judgement

“Noise is eliminated by taking a lot of judgment, and it’s reduced substantially by even taking two judgments. That is one thing that you can do: two heads are better than one. And we know how they are better: they don’t reduce bias but they do reduce noise.” So, simply, when you have a complex decision to make, take more people in to weigh in on the options.