The Wisdom of Crowds

Google's search algorithm. Wikipedia's vast knowledge repository. Linux, the operating system powering most of the internet. What unites these world-changing innovations? None sprang from a lone genius toiling in isolation. Each represents the culmination of countless contributions from diverse minds - testament to the extraordinary power of collective intelligence.

Yet for every Wikipedia success story, we've witnessed spectacular failures of crowd judgment - from market bubbles to political polarisation. How can the same mechanism produce both brilliant innovation and catastrophic error? This paradox sits at the heart of our exploration in this piece.

I've been thinking about this tension recently whilst watching the unfolding spectacle of social media discourse around complex global issues. The sheer confidence with which absolute positions are taken, the dismissal of nuance, and the formation of echo chambers - it's enough to make one question whether there's any wisdom in crowds at all. And it’s on both sides of the spectrum; the left is as vehement as the right has ever been. 

But dismissing collective intelligence entirely would be throwing the baby out with the bathwater. Instead, I want to explore a more nuanced question: Under what conditions does crowd wisdom emerge, and how can we strategically harness it?

This isn't merely academic musing. For organisations, governments, and individuals navigating our complex world, understanding the mechanics of collective intelligence offers a powerful strategic advantage. Whether you're developing products, forming policy, or making investment decisions, knowing when to trust the crowd - and when to run in the opposite direction - could be your most valuable skill.

The Collective Intelligence Paradox

Sir Francis Galton, a Victorian polymath and statistician, discovered something remarkable at a country fair in Plymouth in 1906. Observing a weight-guessing competition for an ox, he collected 787 guesses from fairgoers - butchers, farmers, and random passersby. When he averaged these guesses, the result was 1,197 pounds - just one pound off the actual weight of 1,198 pounds. The crowd, collectively, knew the answer despite no individual getting it exactly right.

This anecdote beautifully illustrates what mathematician and philosopher Nicolas de Condorcet had theorised over a century earlier in his ‘Jury Theorem’. Condorcet posited that a group making independent judgments will, under the right conditions, collectively make better decisions than even its most skilled individual members.

Yet this same mechanism that produces such impressive accuracy can rapidly deteriorate into madness. We need only look at the South Sea Bubble of 1720, where speculation drove share prices of the South Sea Company to unsustainable heights before crashing spectacularly - ruining countless British investors, including Sir Isaac Newton, who famously lamented, "I can calculate the motion of heavenly bodies, but not the madness of people".

So what explains this paradox? Scottish philosopher David Hume, often overlooked in discussions of crowd behaviour, offered valuable insights in his Treatise of Human Nature (another one of the classics we talked about in A-Level RS classrooms!). Hume observed that human judgment is frequently guided not by reason but by ‘sympathy’ - our tendency to adopt the opinions and emotions of those around us. This sympathy, while essential for social cohesion, creates dangerous feedback loops in collective judgment.

This tension between wisdom and madness forms what complex systems theorist Scott Page terms "the diversity prediction theorem". The theorem mathematically demonstrates that collective accuracy depends on both individual accuracy and cognitive diversity. When diversity collapses - when people begin thinking alike - the wisdom evaporates. In simple terms, it’s the echo chamber that creates the narrow ideas. 

The strategic implication is profound: diversity isn't just nice to have; it's mathematically essential for good judgement. This isn't diversity for appearance's sake but genuine cognitive diversity - different ways of thinking, different models of the world, different heuristics for solving problems.

The collective intelligence paradox suggests that crowds contain both extraordinary wisdom and dangerous folly. The strategic challenge lies in designing systems that amplify the former while mitigating the latter. To do this effectively, we must understand the underlying mechanisms that make crowds smart - or devastatingly wrong.

The Mechanics of Crowd Wisdom

James Surowiecki's The Wisdom of Crowds popularised the idea that under the right conditions, groups can be remarkably intelligent. But these conditions are precise and often fragile. Three elements in particular determine whether collective intelligence emerges or dissolves: diversity, independence, and decentralisation.

Diversity ensures a rich pool of perspectives and problem-solving approaches. Independence prevents the cascade of influence that leads to groupthink. Decentralisation allows local knowledge to inform the collective judgment.

“Groups are only smart when there is a balance between the information that everyone in the group shares and the information that each of the members of the group holds privately. It's the combination of all those pieces of independent information, some of them right, some of the wrong, that keeps the group wise.” James Surowiecki

When these elements combine, something extraordinary happens: errors in individual judgment tend to cancel each other out, while accurate insights accumulate. It's akin to what statisticians call the "law of large numbers" - with sufficiently diverse inputs, we approach a more accurate output.

This mechanism operates brilliantly in everyday scenarios we've all encountered, like how Premier League football odds work. Bookmakers don't set these based on a single expert's judgement but on the collective betting patterns of thousands of punters. When the odds shift dramatically before a match, it often signals inside information has reached the market - the crowd has detected something the experts missed. This is why betting markets frequently predict match outcomes more accurately than individual pundits, however knowledgeable they might be.

We see similar principles at work in practical business settings. When Tesco's forecasting team predicts demand for Christmas products, they don't rely solely on last year's data or a single analyst's projection. Instead, they combine inputs from store managers, supply chain experts, and regional buyers - each with unique local knowledge - to create more accurate stocking plans. This approach helped them navigate the particularly challenging 2022 holiday season despite inflation and supply chain disruptions. The diversity of perspectives meant they spotted regional variations in consumer behaviour that would have been missed by centralised planning.

However, these conditions are fragile and easily disrupted. When diversity collapses or independence is compromised, the wisdom evaporates with alarming speed. Social influence can create information cascades - where people disregard their own knowledge in favour of following what appears to be the consensus.

We've all been in that meeting where everyone nods along with the dominant voice, despite our private reservations. This isn't just annoying corporate theatre - it's a classic demonstration of conformity bias wrecking the potential wisdom of the group. I observed this recently whilst consulting for a tech firm where the MD’s initial reaction to a product proposal essentially shut down all further debate. No one wanted to be the dissenting voice. The result was a product launch that failed spectacularly because obvious flaws went unchallenged.

This same mechanism operates in our schools, colleges and universities. When I taught A-Level philosophy, I noticed how one confident student's answer could sway the whole class, even when that student was gloriously wrong. I started having students write down their thoughts before discussion, preserving their independent thinking before the inevitable social influence took hold. The quality of subsequent debates improved dramatically.

We can look at social media for perhaps the most vivid example. We've all seen how quickly Twitter (or X, or whatever Elon's calling it this week) can form a consensus that's wildly disconnected from reality. Remember the certainty with which armchair experts declared the pandemic over, multiple times? Or the absolute conviction that various football managers should be sacked, only for them to go on and win championships? The echo chamber effect isn't just annoying – it actively destroys the cognitive diversity needed for genuine collective wisdom.

These examples highlight something crucial: when independence collapses, crowd wisdom evaporates. This is why thoughtful organisations build safeguards into their decision-making processes. A brilliant example comes from one of the academies I worked with. Rather than having the head teacher evaluate lesson quality alone, they use a panel approach involving teachers from different departments, student representatives, and occasionally external educators. Each observes independently before comparing notes. This structured approach consistently identifies both strengths and improvement areas that any single observer would miss. It might seem overkill but I reckon it’s a better option than the nonsense that OfSTED currently proffer!

The strategic principle here isn't complicated: crowd wisdom requires independence before conformity kicks in. It's about capturing diverse perspectives before the inevitable social dynamics start working to erode them. Capturing that diversity isn't a fluffy nice-to-have; it's a mathematical requirement for better decisions.

Learning from the Ancients (Without the History Lecture)

While we sometimes act as if collective intelligence is a Silicon Valley discovery, humans have been tackling this problem for millennia. And they've developed some surprisingly sophisticated approaches we'd do well to learn from.

One example is the jury system we inherited from medieval England. The requirement for unanimous verdicts might seem inefficient, but it serves a crucial purpose: it forces deliberation to continue until all perspectives are truly heard. This protection against majority steamrolling recognises something fundamental about group wisdom - it's not just about aggregating opinions but ensuring genuine consideration of all viewpoints.

This insight wasn't limited to England. Ancient Celtic communities in Britain used a fascinating approach to dispute resolution through their Brehon system, which I learned about recently. Rather than having a single authority figure decide cases, matters were deliberated by the community, with special attention paid to those with relevant experience. A boundary dispute might involve farmers familiar with the land in question; a family matter would include elders who had navigated similar situations. This natural expertise mapping prevented the folly of having generalists make specialist decisions. I like this a lot. 

‍

These historical methods highlight something we often forget in our modern obsession with efficiency: wisdom takes time to emerge. When I work with school leadership teams on difficult decisions, the most common mistake I see is rushing to judgment before diverse perspectives have been properly explored. The ancient approach of ‘sitting with’ a problem rather than immediately solving it often produces significantly better outcomes.

Even ancient warfare contains relevant lessons. Roman military councils required officers to speak in reverse order of rank, preventing junior officers from simply echoing their superiors. The modern RAF has repurposed this approach in their mission debriefs, with junior officers speaking first to ensure candour before senior voices dominate the conversation.

What these examples demonstrate isn't just historical curiosity. They show that humans have intuitively understood the mechanics of collective intelligence for thousands of years, developing practical methods to preserve the conditions that make crowds wise. The core insight runs through all these approaches: diversity must be actively protected, not just passively assumed.

This isn't about blindly copying ancient methods though. It's about recognising that the fundamental dynamics of human groups haven't changed, even if our technologies have. Whether we're gathering in a village square or on a Zoom call, the same social forces threaten to collapse cognitive diversity into conformity. The ancients developed practical methods to counter these forces, and we'd be fools not to learn from them.

When the Herd Stampedes: The Dark Side of Crowds

For every example of crowd wisdom, there's a balancing case study in crowd madness. I'm not just talking about financial bubbles or Twitter pile-ons, though these are certainly relevant. I'm talking about the systematic ways that groups can make worse decisions than individuals, despite having more information at their disposal.

I saw this dynamic play out painfully during the early pandemic response in several organisations I worked with. In one notable case, a company that prided itself on collaborative decision-making became utterly paralysed as conflicting information circulated. Rather than producing wisdom, their collective process produced anxiety and inaction. Meanwhile, a smaller organisation with a clear decision-making structure was able to move quickly and effectively, protecting both their people and their business. Bureaucracy became a burden. By the way, the etymology of the word bureaucracy is fascinating. Yuval Noah Harari explains it in Nexus but I will throw it in here:

“Bureaucracy literally means rule by writing desk. The term was invented in eighteenth-century France, when the typical official sat next to a writing desk with drawers - a bureau.?' At the heart of the bureaucratic order, then, is the drawer. Bureaucracy seeks to solve the retrieval problem by dividing the world into drawers, and knowing which document goes into which drawer. The principle remains the same regardless of whether the document is placed into a drawer, a shelf, a basket, a jar, a computer folder or any other receptacle: divide and rule. Divide the world into containers, and keep the containers separate so the documents don't get mixed up. This principle, however, comes with a price. Instead of focusing on understanding the world as it is, bureaucracy is often busy imposing a new and artificial order on the world. Bureaucrats begin by inventing various drawers, which are intersubjective realities that don't necessarily correspond to any objective divisions in the world. The bureaucrats then try to force the world to fit into these drawers, and if the fit isn't very good, the bureaucrats push harder.” Yuval Noah Harari

‍

The key difference between the organisations I worked with wasn't the quality of people or information – it was the structure of their decision processes. The collaborative organisation had no mechanism for filtering signal from noise, no way to prevent the anxiety of a few from contaminating the judgment of many. They had a crowd, but not the conditions for crowd wisdom.

This points to something Nassim Nicholas Taleb has observed in his work on antifragility: systems that work well under normal conditions often fail catastrophically under stress. Many crowd-based decision systems have exactly this property - they function beautifully when the stakes are low and emotions are calm, but collapse precisely when we need them most.

"Optionality is the property of asymmetric upside (preferably unlimited) with correspondingly limited downside (preferably tiny)." Nassim Nicholas Taleb

We see this in schools during crises too. The usually effective collaborative approach to behaviour management can quickly break down when facing a genuinely challenging student or parent. Without clear structures for decision-making during stress, the wisdom of the teacher collective gives way to conflicting approaches that make the situation worse, not better. Social media amplifies these failure modes to industrial scale. The very platforms that could theoretically harness collective intelligence are designed in ways that systematically undermine it. When engagement trumps accuracy as the core metric, we shouldn't be surprised when the crowd becomes a mob rather than a mind.

This connects to what economist Charles Goodhart noted decades ago: 

"When a measure becomes a target, it ceases to be a good measure." Charles Goodhart 

When crowds know their opinions will influence outcomes, the incentive for honest judgment gets overwhelmed by other motivations - from status signalling to tribal loyalty.

I've observed this in school governing bodies when discussing contentious issues like budget cuts. Rather than offering their genuine best judgment, governors often position themselves to avoid blame if things go wrong. The result? Decisions optimised for defensibility rather than effectiveness - a problem Philip Tetlock termed ‘the accountability paradox’.

The strategic implication is sobering: crowd wisdom isn't something that happens naturally or automatically. It requires careful cultivation and protection. Subtle changes in how questions are framed or options presented can dramatically shift collective judgments.

Understanding these failure modes isn't just academic navel-gazing. It's essential knowledge for anyone hoping to harness collective intelligence effectively. Whether you're designing decision processes for your team, gathering input on strategic options, or trying to extract signal from social media noise, awareness of how crowds go wrong is the first step toward making them go right.

Building Better Crowds

Having explored both the promise and pitfalls of collective intelligence, let's get practical. How do we actually design systems that amplify we-wisdom while suppressing mob-madness?

The answer starts with structure. Unstructured crowds rarely produce wisdom; carefully structured ones often do. This isn't about complex algorithms or elaborate voting systems - sometimes the most effective approaches are refreshingly straightforward.

Take the "Think-Pair-Share" technique I've used countless times in classrooms and boardrooms alike. Rather than jumping straight to group discussion, participants first develop their own thoughts independently (think), then discuss with just one other person (pair), before bringing ideas to the full group (share). This simple structure preserves independence in the crucial early stages, preventing social cascades from undermining diversity.

M&S has used a version of this approach for product development decisions. Before reviewing new clothing lines as a group, buyers and designers record their independent assessments. This prevents their experts from inadvertently influencing each other and has reportedly improved their hit rate on seasonal collections.

Another straightforward approach is the "silent brainstorm." Instead of the traditional verbal brainstorming (which research consistently shows produces fewer and lower quality ideas than individual work), participants write ideas independently before sharing them. When I introduced this method to a struggling charity's strategy session, the quality and quantity of ideas improved dramatically, with the quieter voices suddenly contributing some of the most innovative thinking.

Blind assessment offers another powerful tool. I worked with a school that was struggling with teacher recruitment. They switched to anonymising applications before the first review, removing names, universities, and other potentially biasing information. The result was a significantly more diverse shortlist and ultimately better hires. The same principle works for evaluating ideas - judge the thought, not the thinker.

For more complex decisions, the "pre-mortem" technique developed by psychologist Gary Klein (which I have talked about LOTS of times) provides a structured way to harness collective foresight. Unlike a post-mortem that analyses why something failed after the fact, a pre-mortem asks participants to imagine a future where the decision has failed spectacularly, then work backwards to identify what went wrong. I've used this with school leadership teams making major curriculum changes, businesses planning a new marketing campaign and individuals making big life-decisions, and it consistently surfaces risks and weaknesses that optimistic group discussion would miss.

Beyond these specific techniques, effective crowd wisdom requires attention to three critical elements:

First, Cognitive Diversity must be actively cultivated. This means deliberately including people with different mental models and problem-solving approaches. It's not just about demographic diversity (though that helps), but about how people think. When helping a tech company revamp their product development process, I insisted they include customer service staff alongside their developers and designers. The resulting insights into real-world usage problems transformed their approach. For more ideas on this, I highly recommend Rebel Ideas by Matthew Syed. 

Second, Independence must be protected through process design. The Royal Society's voting procedure for new fellows requires members to cast votes before any discussion occurs, preserving independent judgment. This simple rule dramatically reduces the influence cascades that undermine collective wisdom. I've adapted this approach for marketing campaigns with excellent results. I’ve also seen it done well when shortlisting candidates in recruitment situations. 

Third, Aggregation Mechanisms must suit the specific context. For numerical estimates (like projected costs or timelines), simple averaging often works beautifully. For complex qualitative judgments, more nuanced approaches are needed. When helping a multi-academy trust evaluate potential school partners, we developed a structured assessment framework that preserved the richness of different perspectives while enabling meaningful comparison. For more on this, check out The End of Average. 

Perhaps most importantly, we must ensure all relevant voices are included. When particular perspectives are systematically excluded or downweighted, the crowd becomes less wise, regardless of its size. I've seen this problem in school parents' forums where working parents - unable to attend daytime meetings - had no input into decisions affecting their children.

Building better crowds isn't about blind faith in collective judgement or dismissing it entirely. It's about understanding the specific conditions under which collective intelligence emerges, and then deliberately engineering those conditions. When we do this well, the results can be remarkable - better decisions, more creative solutions, and outcomes that no individual, however brilliant, could have reached alone.

Key Takeaways

1. Design for Diversity, Not Just Size: Cognitive diversity - different ways of thinking about problems - is mathematically essential for crowd wisdom. Actively recruit varied mental models and problem-solving approaches rather than assuming diversity will happen naturally.

2. Structure Independence Before Collaboration: Collect individual judgements before group discussion to prevent information cascades and groupthink. The sequence matters: independence first, then aggregation, then deliberation.

3. Match the Aggregation Method to the Problem: Simple averages work for numerical estimates, but complex decisions require more sophisticated approaches. Consider whether your challenge needs prediction markets, Delphi methods, or other specialised aggregation techniques.

4. Beware Status and Reputation Effects: They collapse independence and reduce diversity. Use blind assessment protocols and anonymous contribution methods to ensure ideas are evaluated on merit rather than source.

5. Create Accountability for Process, Not Just Outcomes: Reward people for how they contribute to collective intelligence, not just for being right. This encourages information sharing and honest signalling of confidence levels.

6. Remember That Crowds Are Tools, Not Oracles: Collective intelligence works for specific types of problems under specific conditions. Know when to trust the crowd, when to trust experts, and when to trust neither.

The strategic use of collective intelligence represents one of our most powerful and underutilised tools for navigating complexity. Neither blindly trusting crowds nor dismissing them is wise; instead, we must understand the specific conditions under which collective wisdom emerges.

As Aristotle noted millennia ago, 

"The many, of whom each individual is not a good man, when they meet together may be better than the few good, if regarded not individually but collectively." Aristotle

This insight has been rediscovered and refined throughout history, from the Iroquois Great Law of Peace to modern prediction markets.

The strategic imperative is clear: in a world of increasing complexity, no individual mind - however brilliant - can match the potential wisdom of properly structured collective intelligence. The organisations, governments, and individuals who master the art of building better crowds will hold a significant advantage in the decades ahead.

Perhaps the final word should go to the 12th-century Persian poet and philosopher Farid ud-Din Attar. In his allegorical poem The Conference of the Birds, thirty birds search for the mythical Simorgh, only to discover at the end of their journey that they themselves, collectively, are the Simorgh. The wisdom they sought externally was emergent from their collective journey.

So it is with our modern quest for wisdom in decision-making. The intelligence we seek doesn't reside in any individual mind, however expert, nor in unstructured crowds, however large. It emerges, under the right conditions, from our collective cognitive diversity - a strategic resource of immeasurable value in navigating our complex world.