Intuition revisited

In our environment where data and mental processing is more plentiful, is intuition less valuable? I argue the opposite is true, although I have changed my view on the value of intution over time.

When analysis is cheap, the scarce thing is knowing which questions are worth asking and where to spend your energy.

What is intuition?

"Intuition is nothing more and nothing less than recognition." - Herbert Simon

In Simon's definition, a situation provides a cue, the cue unlocks something stored in memory, and that memory provides the answer. For me, this takes the mystery out. Intuition is pattern recognition built from exposure, so it can only be as good as the patterns we have actually lived through.

Where it struggles

Intuition is weakest around large numbers, rare events and probabilities. A good example is in a 1992 study, groups asked what they would pay to save 2,000, 20,000 or 200,000 birds gave average answers of $80, $78 and $88 (Desvousges et al.), i.e. they had a feeling, but it was not well callibrated to the large numbers. And in Thinking fast and slow, there was a section on highly unlikely events being "either ignored or overweighted".

The conclusion from behavioural research is that skilled intuition needs a regular environment and plenty of practice with feedback. Rare events, give us very little practice.

But it's not precise

The accusation in some circles has been that intuition is unreliable because it lacks precision. We have all seen the increasing rate that AI now outperforms clinicians in reading scans and detecting abnormalities.

But aren't we now outsourcing more of our judgement to black boxes ourselves? The Bank of England and FCA found that alost all UK financial services firms use AI, with nearly half having only a "partial understanding" of it.

A model can at least be tested against outcomes. But if we accept one opaque tool because it has been tested, the fair question for intuition is the same: has it been tested, in this environment, through honest feedback?

Where it adds to complex decisions

For me, intuition adds most in three ways.

1) Adding dimensions. Every investment pitch distils a business into three or four dimensions, e.g. for a credit deal: leverage, interest cover, sponsor quality and asset cover and lots of redundant and flimsy information besides.

A seasoned investor's intuition is a feeling about whether enough information is present to be high conviction on something and which information to discard. In this sense intuition is a weighing and concluding tool.

2) Killing projects quickly. Some projects stack up in the abstract yet still should not be pursued. The dilution of focus or pull against the strategy is the key intution. Michael Porter famous for Porter's 5 forces, argued that strategy means also choosing "what not to do" (Harvard Business Review, 1996). An experienced investor often senses the poor fit early, and a quick "no" saves weeks of resource on a doomed project.

3) Prompting further investigation. Kahneman and Klein describe neonatal nurses who spotted life-threatening infections before blood tests came back positive, and at first could not say how. Investors know the equivalent: the logic is sound, yet the story does not sit right. That feeling is is a reason to dig further.

Lived, not downloaded

Where does that feeling and sense come from? I think largely from seeing things not work as they "should". Robin Greenwood and Stefan Nagel, using fund manager age as a proxy for experience, found that around the dot-com peak only younger managers chased technology stocks.

I believe we are in a period we will look back on as frothy, but it is hard to relay to someone who has not lived through a hype cycle that one day rather than talking about not enough capacity, we may be talking about excess capacity.

When we outsource our thinking we are also narrowing our sources. There is a lot of salesy or doomerish content in the training data of many LLMs. Well thought through, balanced testimony is much rarer. Wisdom is becoming rarer even though intelligence might be advancing.

So what?

• Intuition is recognition. It can only be as good as the patterns we have lived through, and it is weakest around large numbers, rare events and probabilities.

• Both AI and Intution are black box. We are relying more on black boxes we only partly understand. The test for both is the same: has it been tested, in this environment, with feedback?

• The sense comes from lived experience. It comes from having seen things not work as they "should", and that is hard to pass on to someone who has never seen a hype cycle turn.

• As we outsource our thinking, we narrow our sources. Intelligence may be advancing, but wisdom is becoming rarer.

• The key is that intuition is valuable if it helps you add dimensions to a problem, kill a resource-intensive project quickly or prompt further investigation.

Next week's blog will be "The Washington Consensus revisited". As always get the blog delivered directly to your inbox on Home | Deciders | for mental fitness | change your mind.