tip off

Report Whacking: your how-to guide on demolishing rubbish reports

You too can demolish “independent reports” with this handy guide to the scams used by the consultants and PR agencies who write them (the media isn’t very good at it).

I seem to spend a large amount of my time in this job explaining in detail why a particular report produced by a particular organisation is particularly problematic.

But it’s increasingly a Canute-like effort. The tide of stupid is rising too fast. Every day, someone somewhere produces a report not with the intent of adding to the total of human knowledge about the objective state of the world (pffft, whatever “objective” is) but in order to influence policy in someone’s interests rather than the public interest, or to publicise themselves. And an ever-more-harried and under-resourced mainstream media lacks the resources to properly analyse these reports: there are only so many Ross Gittinses and Peter Martins to go around, ready to call bullshit on the stuff being served up by special interests and self-promoters.

So I thought I’d put together a brief guide to how you, too, can take up the fight … erm, OK, the bucket, against the tide of stupid. Not that you need my advice, as mere possession of a basic scepticism is probably sufficient, but nonetheless, this is what I do when confronted with a new report freshly dropped on a newspaper journalist and dutifully written up as presented in the executive summary or accompanying press release.

1. Who wrote the report, for whom, and why? Consultants do not write reports for free, indeed they charge a great deal for them, particularly if they contain economic modelling. The bulk of reports are commissioned by interested parties — industry bodies and companies, usually — with the goal of influencing debate in their own interests. And consultants provide exactly what is expected of them: independent modelling is never independent. The other type of report is the self-promotional one, in which a consultancy aims to generate publicity for itself by releasing a report with a nice grab for the media, or to stimulate demand for its own products. A great example was the recent report on music piracy by consultancy firm Musicmetric, which cleverly generated headlines in several different countries each suggesting downloaders in each of those countries were the world’s worst, to maximise publicity.

2. What is their record? Have they predicted the end of the mining boom for a decade? (Did they predict Tony Abbott would turn the Liberals into a “reactionary rump”?) Has the industry or company commissioning the report issued several different reports over time saying entirely different things?

3. What is their methodology? Have they bothered to provide one? A report that doesn’t explain in detail its methodology, or which is simply a press release for a report that isn’t publicly available, can be binned immediately. Next, is it based on a survey or poll? If so, what were the questions? Did the questions make sense? Did they include, for example, defining being a victim of cybercrime as getting a PC virus or being subject to “online harassment”? What was the sample size? Have they used a survey of 68 business executives to judge an entire economy, as the World Economic Forum did in its competitiveness report? Was the survey response weighted to ensure it accurately reflected the relevant population? How did the authors take into account the way people’s political beliefs influence the way they view the world, such that Liberal voters currently think the Australian economy is a disaster area and Labor voters think it’s Eden?

4. If the methodology is based on modelling, what were the assumptions? What was the “business as usual” model? Did the alternate model accurately reflect what it is purported to represent? Or was it like the Access Economics carbon pricing report commissioned by the Baillieu government that conveniently forgot to include carbon price compensation? Does the report simply apply overseas modelling to Australia and assume we’re exactly the same? Or compare Australia to irrelevant and misleading overseas examples (AKA the Economist Intelligence Unit scam)?

5. Does the report use the three most regular tricks in the book? Time and again the same tricks pop up, faithful friends who never go away no matter how much they’re mocked and derided:

  • The fictitious jobs scam is to compare the industry employment outcomes from the modelled scenario with the business-as-usual scenario and declare that a particular policy will “cost thousands of jobs,” as though thousands of workers will be pinkslipped tomorrow, even when the particular industry will in fact grow.
  • The time travel scam is to compare the GDP outcomes from the modelled scenario with the GDP outcome of the business-as-usual scenario and claim that a policy will “shrink the economy by 5%“ as though we’ll immediately enter a prolonged recession rather than have a minutely slower growth path over decades. This was a trick that first came to prominence as ABARE’s contribution to the Howard government’s war against climate science under Brian Fisher.
  • And the third is the reverse magic pudding: to assume that money lost from one industry entirely vanishes from the economy, rather than being spent in another industry, supporting jobs and economic growth there rather than in the industry funding the report.

6. Has Major Events Mathematics been employed? A special branch of applied mathematics, Major Event Maths is studied closely in consultancies working for governments, and sporting bodies, and deals with the remarkable properties of Major Event Numbers, which do not comply with ordinary laws of maths but instead have an additional dimension of multiplication and a capacity to erase negative symbols. Thus, for example, a grand prix that costs Victoria $50 million somehow is estimated to generate $30 million in benefits. Or FIFA announces that 29 billion people have watched a soccer tournament on a planet with only 7 billion people.

7. Do the results actually make sense? Or are they so overstated that they can’t possibly be correct? Like the losses from filesharing that, extended pro rata, would be greater than the entire content industry revenue, or how smokers on average spend an hour every day outside workplaces smoking? Or the way cybercrime cost Australians more than burglary and assaults combined?

With this handy cut-out-and-keep guide, hours of fun are yours to be had playing along at home in the fun game of Report Whacking.

20
  • 1
    David Griffin
    Posted Wednesday, 26 September 2012 at 1:22 pm | Permalink

    Did you coin the “reverse magic pudding” bernard, because I’m goong to use that!

  • 2
    Dajopa
    Posted Wednesday, 26 September 2012 at 1:42 pm | Permalink

    In the words of Mozorov, it is just marketing masquerading as theory

  • 3
    paddy
    Posted Wednesday, 26 September 2012 at 1:48 pm | Permalink

    Worth the read for your startlingly good analysis of “Major Events Mathematics” Bernard.
    Now if you could just come up with a fool proof methodology for applying it to everyday events….We’d be talking a Nobel prize for Economics. (Get writing)

  • 4
    Bill Hilliger
    Posted Wednesday, 26 September 2012 at 2:26 pm | Permalink

    Good article Bernard. Many feasibility reports and studies do use the three most regular tricks you describe. Nowadays these reports and studies are nothing more than just PLEASABILITY studies and reports with particular outcomes to satisfy the paying client.

  • 5
    Posted Wednesday, 26 September 2012 at 2:45 pm | Permalink

    Excellent article Bernard - thanks.

    I also like when Big Coal claims that a certain policy will “cost jobs” and provide a large figure like 15,000. Jobs that never existed and probably never will.

    Fighting this tide of stupid certainly seems like Trivial Pursuit.

  • 6
    Come On Carlton
    Posted Wednesday, 26 September 2012 at 3:11 pm | Permalink

    If it’s ” whacking ” day?.. please whack slowly and then whack faster.

  • 7
    klewso
    Posted Wednesday, 26 September 2012 at 3:17 pm | Permalink

    So that 1 could be “Who commissioned the report and what predetermined path do they want to justify”?

    Isn’t “Economics - the science of predicting the past”?

  • 8
    John Bennetts
    Posted Wednesday, 26 September 2012 at 5:47 pm | Permalink

    Major Event Mathematics has a branch called Traffic Prediction Mathematics.

    Every major capital city in Australia, if not the universe, has an example which has either cost the public a heap or bankrupted its proponents or both, depending on contractual clauses.

    It’s never the fault of the traffic study - those perverse drivers just fail repeatedly to join the conga-line towards the toll gates. This failure is 100% certain but also, by the laws of TPM, absolutely unexpected and unpredictable.

  • 9
    Steve777
    Posted Wednesday, 26 September 2012 at 6:12 pm | Permalink

    Is that the ‘Carbon tax’ wrecking ball in the illustration, working its way through the Australian economy?

  • 10
    Steve777
    Posted Wednesday, 26 September 2012 at 6:19 pm | Permalink

    A very simple test of a claim being made in a report or elsewhere is to do a rough check on any numbers quoted, with the help of Google. It will often reveal that the claim or finding is ridiculous. A recent example was the Club industry’s estimated cost of $5 billion for pre commitment technology. With about 200,000 machines in Australia that is about $25,000 for each machine, which is the approximate cost of a new one. The claim was clearly rubbish because it seemed to assume that every poker machine had to be replaced. In the case of Carbon Pricing, given that power costs were to increase by about 10%, any attribution of a cost increase of more than a few percent to carbon pricing is suspicious. It could happen but should be challenged.

  • 11
    AR
    Posted Wednesday, 26 September 2012 at 9:07 pm | Permalink

    Shorter BK, no Cicero he but well worth his salt, - cui bono?

  • 12
    David Hand
    Posted Thursday, 27 September 2012 at 12:05 am | Permalink

    Bernard,
    I await your analysis of the IPCC reports as a worked example of your handy guide.

  • 13
    David Hand
    Posted Thursday, 27 September 2012 at 12:06 am | Permalink

    Bernard,
    I await your an al ysis of the IPCC reports as a worked example of your handy guide.

  • 14
    Harper Colin
    Posted Thursday, 27 September 2012 at 8:18 am | Permalink

    no Cicero he but well worth his salt,

    Beware of Mark ( Abbott ) Antony.

  • 15
    Jim Bob
    Posted Thursday, 27 September 2012 at 5:21 pm | Permalink

    Nice one Bernard - I particularly liked the ‘Major Event Mathematics’. Scaretistics is another term I think we should start using.

  • 16
    izatso?
    Posted Thursday, 27 September 2012 at 8:41 pm | Permalink

    David Hand. David Hand. You are Twice the Twit you ever were. Bernard allows you a guide for your own use, but no, you are lazier than a 2 dollar miner, YOU OAF.

  • 17
    izatso?
    Posted Thursday, 27 September 2012 at 8:44 pm | Permalink

    David Hand. David Hand. You are Twice the Twit you ever were. Bernard allows you a guide for your own use, but no, you are lazier than a 2 dollar miner, YOU OAF

  • 18
    MJPC
    Posted Friday, 28 September 2012 at 7:47 am | Permalink

    Bernard,

    Excellent article that should be read by everyone in Australia (but alas will not be).
    Reference 1. “Who pays the piper, calls the tune”. The bane of the working class are consultants writing reports that the management wants to read ands support but the ordinary worker has no input into.
    I’m surprised you didn’t take a hatchet job to electroal polling, if any reports needed a thorough de-bunking it would be those. How can 1200 persons interviewed accurately access the voting/political intentions of millions of voters?

  • 19
    Peter Logan
    Posted Saturday, 29 September 2012 at 10:27 am | Permalink

    Thank you Bernard.
    A recent example of the gullibility of the Melbourne Herald Sun was when they (repeatedly) printed findings of a report on the branding value of Melbourne’s grand prix by consultants Formula Money.
    Last year, the same consultancy published a report on the brand value of Melbourne’s exposure at the 2011 Australian grand prix. At the top, it had the race winner’s Red Bull brand at $14.5m, while Melbourne’s branding was only $262,000. Now, with Formula 1 for sale and Melbourne’s contract up for consideration, the Herald Sun tells us the same consultants, Formula Money have revalued the exposure of Melbourne in 2011 at a staggering $216 million. In terms of consultants for hire, this may be a world record in hyperbole.
    I doubt whether the Herald Sun has sighted the report but I sent copy of the former $262,000 report to the editor and am happy to provide it to anyone in the media. With a taxpayer funded $11m advertising budget swinging on the grand prix, the Herald Sun editor refused to correct the articles. He offered me letter to the editor but it’s not The Age, where the letters page actually has some weight.
    It all comes down to the public’s right to know the truth – they just paid $57m to cover the operating loss and contribute millions more in hidden subsidies every year – but the Auditor-General’s peer reviewed cost- benefit analysis revealing the economic loss is not given credence by the Herald Sun.
    Peter Logan
    Save Albert Park Inc media
    0412697074

  • 20
    dazza
    Posted Wednesday, 17 October 2012 at 3:27 pm | Permalink

    Good one B.K. Tell me, why do I keep thinking “The Australian” when I read your article?

Womens Agenda

loading...

Smart Company

loading...

StartupSmart

loading...

Property Observer

loading...