Sep 26, 2012

Report Whacking: your how-to guide on demolishing rubbish reports

You too can demolish "independent reports" with this handy guide to the scams used by the consultants and PR agencies who write them (the media isn't very good at it).

Bernard Keane — Politics editor

Bernard Keane

Politics editor

I seem to spend a large amount of my time in this job explaining in detail why a particular report produced by a particular organisation is particularly problematic. But it's increasingly a Canute-like effort. The tide of stupid is rising too fast. Every day, someone somewhere produces a report not with the intent of adding to the total of human knowledge about the objective state of the world (pffft, whatever "objective" is) but in order to influence policy in someone's interests rather than the public interest, or to publicise themselves. And an ever-more-harried and under-resourced mainstream media lacks the resources to properly analyse these reports: there are only so many Ross Gittinses and Peter Martins to go around, ready to call bullshit on the stuff being served up by special interests and self-promoters. So I thought I'd put together a brief guide to how you, too, can take up the fight … erm, OK, the bucket, against the tide of stupid. Not that you need my advice, as mere possession of a basic scepticism is probably sufficient, but nonetheless, this is what I do when confronted with a new report freshly dropped on a newspaper journalist and dutifully written up as presented in the executive summary or accompanying press release. 1. Who wrote the report, for whom, and why? Consultants do not write reports for free, indeed they charge a great deal for them, particularly if they contain economic modelling. The bulk of reports are commissioned by interested parties -- industry bodies and companies, usually -- with the goal of influencing debate in their own interests. And consultants provide exactly what is expected of them: independent modelling is never independent. The other type of report is the self-promotional one, in which a consultancy aims to generate publicity for itself by releasing a report with a nice grab for the media, or to stimulate demand for its own products. A great example was the recent report on music piracy by consultancy firm Musicmetric, which cleverly generated headlines in several different countries each suggesting downloaders in each of those countries were the world’s worst, to maximise publicity. 2. What is their record? Have they predicted the end of the mining boom for a decade? (Did they predict Tony Abbott would turn the Liberals into a "reactionary rump"?) Has the industry or company commissioning the report issued several different reports over time saying entirely different things? 3. What is their methodology? Have they bothered to provide one? A report that doesn’t explain in detail its methodology, or which is simply a press release for a report that isn’t publicly available, can be binned immediately. Next, is it based on a survey or poll? If so, what were the questions? Did the questions make sense? Did they include, for example, defining being a victim of cybercrime as getting a PC virus or being subject to "online harassment"? What was the sample size? Have they used a survey of 68 business executives to judge an entire economy, as the World Economic Forum did in its competitiveness report? Was the survey response weighted to ensure it accurately reflected the relevant population? How did the authors take into account the way people's political beliefs influence the way they view the world, such that Liberal voters currently think the Australian economy is a disaster area and Labor voters think it's Eden? 4. If the methodology is based on modelling, what were the assumptions? What was the "business as usual" model? Did the alternate model accurately reflect what it is purported to represent? Or was it like the Access Economics carbon pricing report commissioned by the Baillieu government that conveniently forgot to include carbon price compensation? Does the report simply apply overseas modelling to Australia and assume we’re exactly the same? Or compare Australia to irrelevant and misleading overseas examples (AKA the Economist Intelligence Unit scam)? 5. Does the report use the three most regular tricks in the book? Time and again the same tricks pop up, faithful friends who never go away no matter how much they’re mocked and derided:
  • The fictitious jobs scam is to compare the industry employment outcomes from the modelled scenario with the business-as-usual scenario and declare that a particular policy will "cost thousands of jobs," as though thousands of workers will be pinkslipped tomorrow, even when the particular industry will in fact grow.
  • The time travel scam is to compare the GDP outcomes from the modelled scenario with the GDP outcome of the business-as-usual scenario and claim that a policy will "shrink the economy by 5%" as though we’ll immediately enter a prolonged recession rather than have a minutely slower growth path over decades. This was a trick that first came to prominence as ABARE’s contribution to the Howard government’s war against climate science under Brian Fisher.
  • And the third is the reverse magic pudding: to assume that money lost from one industry entirely vanishes from the economy, rather than being spent in another industry, supporting jobs and economic growth there rather than in the industry funding the report.
6. Has Major Events Mathematics been employed? A special branch of applied mathematics, Major Event Maths is studied closely in consultancies working for governments, and sporting bodies, and deals with the remarkable properties of Major Event Numbers, which do not comply with ordinary laws of maths but instead have an additional dimension of multiplication and a capacity to erase negative symbols. Thus, for example, a grand prix that costs Victoria $50 million somehow is estimated to generate $30 million in benefits. Or FIFA announces that 29 billion people have watched a soccer tournament on a planet with only 7 billion people. 7. Do the results actually make sense? Or are they so overstated that they can’t possibly be correct? Like the losses from filesharing that, extended pro rata, would be greater than the entire content industry revenue, or how smokers on average spend an hour every day outside workplaces smoking? Or the way cybercrime cost Australians more than burglary and assaults combined? With this handy cut-out-and-keep guide, hours of fun are yours to be had playing along at home in the fun game of Report Whacking.

Free Trial

You've hit members-only content.

Sign up for a FREE 21-day trial to keep reading and get the best of Crikey straight to your inbox

By starting a free trial, you agree to accept Crikey’s terms and conditions


Leave a comment

20 thoughts on “Report Whacking: your how-to guide on demolishing rubbish reports

  1. David Griffin

    Did you coin the “reverse magic pudding” bernard, because I’m goong to use that!

  2. Dajopa

    In the words of Mozorov, it is just marketing masquerading as theory

  3. paddy

    Worth the read for your startlingly good analysis of “Major Events Mathematics” Bernard.
    Now if you could just come up with a fool proof methodology for applying it to everyday events….We’d be talking a Nobel prize for Economics. (Get writing)

  4. Bill Hilliger

    Good article Bernard. Many feasibility reports and studies do use the three most regular tricks you describe. Nowadays these reports and studies are nothing more than just PLEASABILITY studies and reports with particular outcomes to satisfy the paying client.

  5. Coaltopia

    Excellent article Bernard – thanks.

    I also like when Big Coal claims that a certain policy will “cost jobs” and provide a large figure like 15,000. Jobs that never existed and probably never will.

    Fighting this tide of stupid certainly seems like Trivial Pursuit.

  6. Come On Carlton

    If it’s ” whacking ” day?.. please whack slowly and then whack faster.

  7. klewso

    So that 1 could be “Who commissioned the report and what predetermined path do they want to justify”?

    [Isn’t “Economics – the science of predicting the past”?]

  8. John Bennetts

    Major Event Mathematics has a branch called Traffic Prediction Mathematics.

    Every major capital city in Australia, if not the universe, has an example which has either cost the public a heap or bankrupted its proponents or both, depending on contractual clauses.

    It’s never the fault of the traffic study – those perverse drivers just fail repeatedly to join the conga-line towards the toll gates. This failure is 100% certain but also, by the laws of TPM, absolutely unexpected and unpredictable.

  9. Steve777

    Is that the ‘Carbon tax’ wrecking ball in the illustration, working its way through the Australian economy?

  10. Steve777

    A very simple test of a claim being made in a report or elsewhere is to do a rough check on any numbers quoted, with the help of Google. It will often reveal that the claim or finding is ridiculous. A recent example was the Club industry’s estimated cost of $5 billion for pre commitment technology. With about 200,000 machines in Australia that is about $25,000 for each machine, which is the approximate cost of a new one. The claim was clearly rubbish because it seemed to assume that every poker machine had to be replaced. In the case of Carbon Pricing, given that power costs were to increase by about 10%, any attribution of a cost increase of more than a few percent to carbon pricing is suspicious. It could happen but should be challenged.

Share this article with a friend

Just fill out the fields below and we'll send your friend a link to this article along with a message from you.

Your details

Your friend's details