Just spotted an interesting letter in last week’s Lancet discussing selective reporting of clinical trials. This may sound like something you’ve heard many times before (eg here) but in the Lancet letter the authors describe what happenned to trials for which the original protocols were posted on Lancet’s own website.
Evidence regarding selective reporting has been around for a while (eg the 2004 study mentioned above pulled out a set of protocols approved by Danish ethics committees, and matched them up with corresponding results publications. The authors showed that 62% of trials had “at least 1 primary outcome that was changed, introduced, or omitted”). However, at the moment very few trial protocols are ever published; the decision to publish or not is entirely voluntary, and very few journals offer the service. The Lancet offers protocol review – although the page is a bit hard to find), as do many of the BioMed Central journals.
In their letter to the Lancet, Al-Marzouki and colleagues describe how, for the set of trials they examined, unreported or switched primary and secondary outcomes were common, with no explanation for why outcomes were included, left out, or changed between the original plan and final report. What’s amazing here is that this phenomenon occurs despite the fact that the protocol has been made public at an early stage, entirely voluntarily, and that triallists are presumably aware that as a result their eventual papers might be compared against the original analysis plans.
PLoS Medicine doesn’t offer protocol review, but we do require authors to send us copies of the trial protocol when they submit their papers. We, our reviewers and academic editors find it’s an incredibly tough job to compare submitted trial reports against protocols, but do occasionally find discrepancies. Al-Marzouki and colleagues suggest that a universal standard for protocols might simplify the process – but we’re not there yet. The Good Clinical Practice guidelines provide a format – but only for trials done by drug or device companies to gain regulatory approval. Sadly it seems that in order to carry out a thorough scientific evaluation of a submitted trial, someone — either an editor, reviewer, or careful reader — has to …. actually read through the entire protocol… plus the report of course. Surely technology can give us something better than this?