Ditch the ‘Experts’

Grading pundits and prognosticators: More famous = less accurate.

By Geoffrey Colvin
January 30, 2006: 1:28 PM EST

(FORTUNE Magazine) – You have been a world-class sap for years. Why?
For listening to the economic and political forecasts of experts. We in
the media have been irresponsible fools for reporting those forecasts.
And the experts themselves? Delusional egomaniacs–and maybe even con

I didn’t always think this way. But I’ve been reading a book that
marshals powerful evidence to make this case. For all of us in the
world of business, economics, and capital markets–a world that often
turns on the judgments of experts–the question is whether we’re brave
enough to face these uncomfortable facts.

The book is Expert Political Judgment: How Good Is It? How Can We
Know? by Philip E. Tetlock, a professor at the University of California
at Berkeley. It summarizes the results of a truly amazing research
project: Over seven years Tetlock got a wide range of experts and
nonexperts to answer carefully constructed questions about the
likelihood of specific future events. He ended up with a staggering
82,361 forecasts, expressed in quantifiable form and thus able to be
analyzed deeply. His definition of “political judgment” included plenty
of topics that you and I would call economic, such as government
spending and national economic performance.

Tetlock then cranked all those numbers through every kind of
statistical thresher, flail, and grinder you can imagine, and the
result was clear: Experts don’t actually exist. Specifically, experts
were no better than nonexperts at predicting the future. They weren’t
even as good as computer programs that merely extrapolate the past. The
best experts could not explain more than 20% of the variability in
outcomes, but crude algorithms could explain 25% to 30%, and
sophisticated algorithms could explain 47%. Consider what this means.
On all sorts of questions you care about–Where will the Dow be in two
years? Will the federal deficit balloon as baby-boomers retire?–your
judgment is as good as the experts’. Not almost as good. Every bit as

Which is not to say that experts are no different from you and me.
They’re very different. For example, they’re much more confident in
their predictions than nonexperts are, though they obviously have no
reason to be. For example, the members of the American Political
Science Association predicted in August 2000 that a Gore victory was a
slam dunk.

Experts can also give far more reasons for their predictions than
nonexperts can. Their vast erudition lets them explain at daunting
length why something will or won’t happen. Not that all those reasons
make the forecasts one bit better.

The question that screams out from the data is why the world keeps
believing that “experts” exist at all. In large part, the answer is
human nature. We desperately want to believe the world is not just a
big game of dice, that things happen for good reasons and wise people
can figure it all out. It may not be so; a school of researchers known
as radical skeptics presents impressive evidence that the world is
totally random, or at least that we humans are eternally unable to
figure it out. But most of us can’t bear to believe that, so we cling
to the notion of experts.

Another part of the answer is especially troubling for the media.
The awfulness of Tetlock’s experts was almost uniform whether they had
doctorates or bachelor’s degrees, lots of experience or little, access
to classified data or none. He found but one consistent differentiator:
fame. The more famous the experts, the worse they performed. And of
course it’s those of us in TV, radio, newspapers, magazines, and on the
web who bestow that fame. We’re now reaching a deeper answer to the
question of why experts persist. The media like experts, the more
confident the better, though Tetlock found that more confident means
less reliable. The media like them because you, the consumers of media,
like them. Experts like to appear in the media because it pays.
Tetlock’s conclusion: “The three principals–authoritative-sounding
experts, the ratings-conscious media, and the attentive public–may
thus be locked in a symbiotic triangle.”

It’s never too late for a good New Year’s resolution, so I propose
that in the coming year, whenever any of us read or hear an expert
opinion, or publish one, or issue one, we ask ourselves whether Tetlock
might be onto something with this quietly withering conclusion about
all of us in that triangle: “It is tempting to say they need each other
too much to terminate a relationship merely because it is based on an

FEEDBACK [email protected]



Danna Vale’s Muslim population prediction…

bird flu/ housing boom

astrology based:






Age experts deliver their predictions for the 2005 AFL grand final

Sydney 25 points

West Coast by 15 points

West Coast’s midfield depth and talent is superior to Sydney’s.
The Eagles can rectify their front problems after their last
meeting with the Swans.
Norm Smith Medal: Dean Cox (WC).

Sydney by 10 points

Sydney is the in-form team, winning 12 of its past 15, West
Coast losing four of its past eight. Winning a final last week at
the MCG will hold the Swans in good stead.
Norm Smith Medal: Brett Kirk (Sydney).

Sydney by 13 points

The Swans have excellent form and play the MCG extremely well.
The last time the sides met, the Eagles’ midfield starred, yet they
were still lucky winners. But what worries is the Eagles’ inability
to reproduce the home form on the road.
Norm Smith Medal: Brett Kirk (Sydney).

Sydney by 18 points

The Swans have grown from a team into a unit, capable of
applying and withstanding grand final pressure. West Coast’s
midfield will have a monopoly, but who will kick the goals?
Norm Smith Medal: Adam Goodes (Sydney).

Sydney by 15 points

The Swans might not have the best individuals, but they have the
best team. Sydney could well struggle to match the Eagles’ midfield
but its cluster of options up forward is much better, provided they
attack and push the ball inside 50 often enough.
Norm Smith Medal: Tadhg Kennelly (Sydney)

West Coast by 28 points

The Eagles might have fixed up their scoring problems, and
certainly have the edge in midfield, where they dominated Sydney
three weeks ago. Class will prevail.
Norm Smith Medal: Andrew Embley (WC).

Sydney by 17 points

With Hall in, the Swans’ forward line is too multi-dimensional.
The Swans’ midfield will try to stifle West Coast, although expect
the Eagles’ runners to still get plenty of the ball but their
forward line to be unable to take advantage.
Norm Smith Medal: Chris Judd (WC).

West Coast by 18 points

West Coast has an edge in freshness and in class around the
ball. In the qualifying final three weeks ago, the ultimate
difference was the ability of Judd and others to somehow break into
the open from the stifling packs.
Norm Smith Medal: Chris Judd (WC).

West Coast by 9 points

It is hard to go past Sydney for the hard, shutdown,
“finals-type” football it plays. But games are so often won in the
midfield, and West Coast’s class there, both tall and small, will
be the difference in one that could go either way.
Norm Smith Medal: Brett Kirk (Sydney).

Sydney by 22 points

The warrior Swans have the defensive skills to shut down West
Coast’s midfield and with the superior forward line – Hall,
O’Loughlin, O’Keefe and Goodes – should be able to do plenty of
Norm Smith Medal: Brett Kirk (Sydney).

West Coast by 16 points

Almost too close to call but the Swans’ tough finals campaign
may catch up with them. What may also work in West Coast’s favour
is that players who endure a tribunal battle in grand final week
historically have little influence on the game.
Norm Smith Medal: Chris Judd (WC).

Sydney by 6 points

The game will be a contest of styles. The Eagles are the runners
who need space, the Swans the crampers with a preference for scrums
and defensive pressure, but with a more efficient attack and a
superior defence.
Norm Smith Medal: Tadhg Kennelly (Sydney).

Sydney by 18 points

With Hall free to play, the Swans have what West Coast doesn’t:
a focal point up front. That, combined with the wave of sentiment
behind them, should get them over the line.
Norm Smith Medal: Brett Kirk (Sydney).

Sydney by 22 points

Grand finals are won by settled sides, and the Swans have been
just that over the past two months. They have the far more
dangerous forward line, a resilient defence and so, like few sides,
can afford to fall a tiny bit short in the middle and still
Norm Smith Medal: Jared Crouch (Sydney).

Sydney by 18 points

The Swans have not conceded more than 69 points in any final so
far. They have allowed the opposition only 26 goals in the three
matches preceding the grand final. This speaks of a solidarity that
is unlikely to break down in the biggest game of all.
Norm Smith Medal: Brett Kirk (Sydney).

Sydney by 14 points

The Swans can contain West Coast and have enough fighting
momentum to get them over the line. Hall has had a big week and now
it’s time for him to perform on the big stage.
Norm Smith Medal: Barry Hall (Sydney).

Sydney by 21 points

In this most unpredictable of seasons, Sydney will prove the
surprise packet of 2005, shutting down the Eagles’ midfield on a
ground it has flourished on. Barry at one end and Hall at the other
should star. Goodes and O’Loughlin are due big performances.
Norm Smith Medal: Brett Kirk (Sydney).