Menu lock


Sep 26, 2012

Report Whacking: your how-to guide on demolishing rubbish reports

You too can demolish "independent reports" with this handy guide to the scams used by the consultants and PR agencies who write them (the media isn't very good at it).

Bernard Keane — Politics Editor

Bernard Keane

Politics Editor

I seem to spend a large amount of my time in this job explaining in detail why a particular report produced by a particular organisation is particularly problematic.

But it’s increasingly a Canute-like effort. The tide of stupid is rising too fast. Every day, someone somewhere produces a report not with the intent of adding to the total of human knowledge about the objective state of the world (pffft, whatever “objective” is) but in order to influence policy in someone’s interests rather than the public interest, or to publicise themselves. And an ever-more-harried and under-resourced mainstream media lacks the resources to properly analyse these reports: there are only so many Ross Gittinses and Peter Martins to go around, ready to call bullshit on the stuff being served up by special interests and self-promoters.

So I thought I’d put together a brief guide to how you, too, can take up the fight … erm, OK, the bucket, against the tide of stupid. Not that you need my advice, as mere possession of a basic scepticism is probably sufficient, but nonetheless, this is what I do when confronted with a new report freshly dropped on a newspaper journalist and dutifully written up as presented in the executive summary or accompanying press release.

1. Who wrote the report, for whom, and why? Consultants do not write reports for free, indeed they charge a great deal for them, particularly if they contain economic modelling. The bulk of reports are commissioned by interested parties — industry bodies and companies, usually — with the goal of influencing debate in their own interests. And consultants provide exactly what is expected of them: independent modelling is never independent. The other type of report is the self-promotional one, in which a consultancy aims to generate publicity for itself by releasing a report with a nice grab for the media, or to stimulate demand for its own products. A great example was the recent report on music piracy by consultancy firm Musicmetric, which cleverly generated headlines in several different countries each suggesting downloaders in each of those countries were the world’s worst, to maximise publicity.

2. What is their record? Have they predicted the end of the mining boom for a decade? (Did they predict Tony Abbott would turn the Liberals into a “reactionary rump“?) Has the industry or company commissioning the report issued several different reports over time saying entirely different things?

3. What is their methodology? Have they bothered to provide one? A report that doesn’t explain in detail its methodology, or which is simply a press release for a report that isn’t publicly available, can be binned immediately. Next, is it based on a survey or poll? If so, what were the questions? Did the questions make sense? Did they include, for example, defining being a victim of cybercrime as getting a PC virus or being subject to “online harassment“? What was the sample size? Have they used a survey of 68 business executives to judge an entire economy, as the World Economic Forum did in its competitiveness report? Was the survey response weighted to ensure it accurately reflected the relevant population? How did the authors take into account the way people’s political beliefs influence the way they view the world, such that Liberal voters currently think the Australian economy is a disaster area and Labor voters think it’s Eden?

4. If the methodology is based on modelling, what were the assumptions? What was the “business as usual” model? Did the alternate model accurately reflect what it is purported to represent? Or was it like the Access Economics carbon pricing report commissioned by the Baillieu government that conveniently forgot to include carbon price compensation? Does the report simply apply overseas modelling to Australia and assume we’re exactly the same? Or compare Australia to irrelevant and misleading overseas examples (AKA the Economist Intelligence Unit scam)?

5. Does the report use the three most regular tricks in the book? Time and again the same tricks pop up, faithful friends who never go away no matter how much they’re mocked and derided:

  • The fictitious jobs scam is to compare the industry employment outcomes from the modelled scenario with the business-as-usual scenario and declare that a particular policy will “cost thousands of jobs,” as though thousands of workers will be pinkslipped tomorrow, even when the particular industry will in fact grow.
  • The time travel scam is to compare the GDP outcomes from the modelled scenario with the GDP outcome of the business-as-usual scenario and claim that a policy will “shrink the economy by 5%” as though we’ll immediately enter a prolonged recession rather than have a minutely slower growth path over decades. This was a trick that first came to prominence as ABARE’s contribution to the Howard government’s war against climate science under Brian Fisher.
  • And the third is the reverse magic pudding: to assume that money lost from one industry entirely vanishes from the economy, rather than being spent in another industry, supporting jobs and economic growth there rather than in the industry funding the report.

6. Has Major Events Mathematics been employed? A special branch of applied mathematics, Major Event Maths is studied closely in consultancies working for governments, and sporting bodies, and deals with the remarkable properties of Major Event Numbers, which do not comply with ordinary laws of maths but instead have an additional dimension of multiplication and a capacity to erase negative symbols. Thus, for example, a grand prix that costs Victoria $50 million somehow is estimated to generate $30 million in benefits. Or FIFA announces that 29 billion people have watched a soccer tournament on a planet with only 7 billion people.

7. Do the results actually make sense? Or are they so overstated that they can’t possibly be correct? Like the losses from filesharing that, extended pro rata, would be greater than the entire content industry revenue, or how smokers on average spend an hour every day outside workplaces smoking? Or the way cybercrime cost Australians more than burglary and assaults combined?

With this handy cut-out-and-keep guide, hours of fun are yours to be had playing along at home in the fun game of Report Whacking.


Leave a comment

20 thoughts on “Report Whacking: your how-to guide on demolishing rubbish reports

  1. AR

    Shorter BK, no Cicero he but well worth his salt, – cui bono?

  2. David Hand

    I await your analysis of the IPCC reports as a worked example of your handy guide.

  3. David Hand

    I await your an al ysis of the IPCC reports as a worked example of your handy guide.

  4. Harper Colin

    no Cicero he but well worth his salt,

    Beware of Mark ( Abbott ) Antony.

  5. Jim Bob

    Nice one Bernard – I particularly liked the ‘Major Event Mathematics’. Scaretistics is another term I think we should start using.

  6. izatso?

    David Hand. David Hand. You are Twice the Twit you ever were. Bernard allows you a guide for your own use, but no, you are lazier than a 2 dollar miner, YOU OAF.

  7. izatso?

    David Hand. David Hand. You are Twice the Twit you ever were. Bernard allows you a guide for your own use, but no, you are lazier than a 2 dollar miner, YOU OAF

  8. MJPC


    Excellent article that should be read by everyone in Australia (but alas will not be).
    Reference 1. “Who pays the piper, calls the tune”. The bane of the working class are consultants writing reports that the management wants to read ands support but the ordinary worker has no input into.
    I’m surprised you didn’t take a hatchet job to electroal polling, if any reports needed a thorough de-bunking it would be those. How can 1200 persons interviewed accurately access the voting/political intentions of millions of voters?

  9. Peter Logan

    Thank you Bernard.
    A recent example of the gullibility of the Melbourne Herald Sun was when they (repeatedly) printed findings of a report on the branding value of Melbourne’s grand prix by consultants Formula Money.
    Last year, the same consultancy published a report on the brand value of Melbourne’s exposure at the 2011 Australian grand prix. At the top, it had the race winner’s Red Bull brand at $14.5m, while Melbourne’s branding was only $262,000. Now, with Formula 1 for sale and Melbourne’s contract up for consideration, the Herald Sun tells us the same consultants, Formula Money have revalued the exposure of Melbourne in 2011 at a staggering $216 million. In terms of consultants for hire, this may be a world record in hyperbole.
    I doubt whether the Herald Sun has sighted the report but I sent copy of the former $262,000 report to the editor and am happy to provide it to anyone in the media. With a taxpayer funded $11m advertising budget swinging on the grand prix, the Herald Sun editor refused to correct the articles. He offered me letter to the editor but it’s not The Age, where the letters page actually has some weight.
    It all comes down to the public’s right to know the truth – they just paid $57m to cover the operating loss and contribute millions more in hidden subsidies every year – but the Auditor-General’s peer reviewed cost- benefit analysis revealing the economic loss is not given credence by the Herald Sun.
    Peter Logan
    Save Albert Park Inc media

  10. dazza

    Good one B.K. Tell me, why do I keep thinking “The Australian” when I read your article?