Pick your cherries while ye may

The nature of scientific evidence has been under scrutiny this week with the coinciding of three important news stories. Simon Singh was in court for the appeal against the ruling last year by Judge Eady that he libelled the British Chiropractors Association - an action that the presiding judge described as "baffling" (story here) - while the House of Commons Science and Technology Committee delivered a damning verdict upon the funding of homeopathy by the NHS (here). Finally, and a little late in the day, the General Chiropractic Council fessed up the evidence for the efficacy of its treatments, a decidedly mixed bag (here).

What unites these three stories is they are examples of the reliance on cherry-picked evidence by the less-than-scientific, in order to bolster their credentials. In the realms of scientific practice and statistical validity, cherry-picking is one of the big sins; put simply it means ignoring all the evidence that doesn't support your hypothesis. So, if you conduct 100 trials on a new drug and 95 show it performs no better than the current treatment, but five show some improved outcomes, then to present these latter five trials as evidence supporting your drug's efficacy is to do A Bad Thing. Depending on what you are trying to prove, at best you are being unfair, and at worst committing fraud, though that doesn't stop it from being widely practised, and is surely the basis of 1,001 advertising campaigns.

And it also was the basis of an unusual claim by Inspector Roger Bartlett of the Devon and Cornwall Police this week of divine intervention to account for an improvement in crime statistics in his manor (story here). According to Inspector Bartlett the power of prayer by local christian groups has lead to a direct improvement in clear-up rates in the Barnstaple area, and similar decrease in number of serious road accidents in north Devon. Impressively, it would seem Insp Bartlett asked a group of local Christians to pray for a specific reduction in road accident deaths in the area the year before the number of incidents fell by 67%. Rather less impressively, he then continues to list random statistical improvements that he retrospectively attributes to prayer, which rather weakens his case.

But rather than give his notion the fisking it deserves, I wondered if I could try the same thing? Of course I can, for while it seems the good Lord is spending a lot of time helping out the motorists of Devon, he's been taking his eye off the deserving but weak-hearted. In 2006 a double-blinded RCT on the power of prayer was performed Harvard Medical School upon those recovering from coronary artery bypass graft (CABG) surgery, with some allocated to receive unctions and some not, and those two groups further divided, with half being told they were being prayed for and the rest not.

The conclusion? "Intercessory prayer itself had no effect on complication-free recovery from CABG, but certainty of receiving intercessory prayer was associated with a higher incidence of complications." In other words, knowing you were being prayed for actually increased the chances of you getting worse, not better.

This then leads me to conclude that, while God clearly cares enough about the commitment of sin (crime) to help reduce it, he's not so keen on the love and compassion bit if you have a dicky ticker. Sounds a bit of an Old Testament sort of chap to me, which leaves me to assume the entire basis of Christianity is flawed, and therefore Judaism is the one true faith. QED. Suddenly things become enormously simple if you just notice what you want to see, and select your evidence accordingly, or look for patterns where none exist.

Meanwhile do be sure to drive carefully in north Devon, if you find yourself in that neck of the woods. God's made an impressive start, but wouldn't want you to ruin his hard work by demonstrating free will and driving off a cliff - he'd rather you wait for that heart attack to strike. And poor Inspector Bartlett would have to explain how his number went back up again without defaming Jehovah.