So, Tamiflu has been proven to be a complete waste of our money and naughty old Big Pharma had the Government’s trousers down by being economical with the actualité.
Well, you’d think so unless you spent the last week in a coma; according to all reports it appears the manufacturer was cherry-picking favourable data from trials on a drug that’s no more effective than paracetamol and that actually causes harm. Really? That’s what they said in the press conference.
But this story has many nuances, unstated biases and other flaws that make it far more complex and interesting than the reporting might have you believe. And this isn’t just another rant about shite science reporting. Well, not quite so much of a rant as usual. Let’s start with the paper that caused all the fuss and look at the data it presents. Overall it accepts Tamiflu reduces duration of flu symptoms in adults by about 2/3rds of a day and in children by just over a day (not exactly a stellar reduction) but in terms of prevention it reduces the risk of symptomatic influenza in individuals by 55% and in households by 80%. It had no effect on hospital admission rates when used for influenza prevention and had no effect on asymptomatic influenza transmission.
This paper adds little to current knowledge of Tamiflu’s side effects; it is generally well tolerated but does reduce “cardiac body system events” – which is unsurprising as infection = inflammation which is pro-thrombotic: that is, it increases risk of heart attacks, strokes etc.
All in all, this paper does give evidence that Tamiflu has a benefit when treating influenza but the benefit is small. It also shows Tamiflu is of benefit in the prevention of influenza, especially in households. It also says there is no evidence to suggest that Tamiflu increases psychiatric side effects – the opposite of what one of the authors told the BBC.
So, what’s the real story?
This story raised a number of red flags for me from the beginning. I’m not an epidemiologist but the way this has been reported and the dramatis personae had me looking a lot further than the abstract. And all is not what it may seem. But to get there we need some scene setting. [Clinicians and those familiar with flu virology can skip a para or three here – the juicy bits come in ‘So What Does It All Mean…’]
Tamiflu Basics
Tamiflu is a neuraminidase inhibitor. WTF? OK. In order to infect us, flu needs a way to get in to a host cell and a way for the daughter viruses to get out. The flu virus has stuff called haemagglutinins on its surface which enable it to stick to cells in our airway so it can infect them. Once the virus has infected a host cell and – like all viruses – hijacked it to manufacture copies of itself, something else on the flu virus coat called neuraminidase helps the daughter viruses escape from the host cell so the cycle can start all over again.
So, the H enables flu virus to invade a host cell, the N helps it get out again – and we use the flavour of the ‘H’ and the ‘N’ to describe a particular strain – H1N1 or H3N5 for example.
Tamiflu inhibits this neuraminidase stuff thus making it much more difficult for daughter flu viruses to escape an infected host cell, and so slows down the infection. So, Tamiflu is not a flu vaccine – it’s not designed to prevent you getting flu like a vaccine would but you may well get a far less serious illness because the drug will slow the infection down – so your immune system has a chance to get stuck in. It’s not perfect. But it’s not useless either. Bear with me because that’s important later.
(Incidentally, most antibiotics work in a similar way against bacteria in that they slow down an infection so your immune system can get stuck in; very few antibiotics actually kill bacteria outright. But flu is a virus so antibiotics are ineffective against it – so antivirals like Tamiflu have to work in very different ways to antibiotics).
Data, Deception, Dogma
There is a phenomenon called publication bias – that negative studies are less likely to be published than positive ones. This is a known bias and has been a focus of the Alltrials campaign – that manufacturers should be forced to publish all their data, not just the favourable stuff. Ben Goldacre wrote a book about it and the BMJ has been very vocal in getting all the data for Tamiflu and another neuraminidase inhibitor called Relenza into the public domain. Roche finally relented, resulting in the BMJ paper that is now being widely reported.
The Cochrane Collaboration
This organisation is named after Archie Cochrane, one of the fathers of modern epidemiology – the notion that rather than rely on anecdote or “we’ve always done it this way” any intervention should be properly studied and the data analysed rigorously. The Cochrane Collaboration analyses randomised controlled trial (RCT) data to help clinicians make informed decisions. The pinnacle of any study is the randomised, placebo-controlled, double-blind trial – where half the people get the drug or treatment, half get the placebo (usually far more sophisticated than a sugar pill). The key point is neither the patient nor the people administering the treatment know who has had what. That way you try to eliminate bias and can see if the intervention works better than placebo. Allegedly.
The problem is it can be difficult to design a perfect trial with a big enough data set (number of participants) and free of unintentional bias. So this is where systematic review and meta-analysis comes in – where you take a bunch of similar studies, lump them together and see what it tells you (apologies to epidemiologists and statisticians for the gross oversimplification but I’m a microbiologist – what do you expect?).
This all sounds fine but can also be deeply flawed if not done properly: the risk is that if you take a bunch of dog turd studies and put them together, you just get a massive pile of shit, it doesn’t automatically turn into a work of truth and beauty. And many of us (me included) are not good enough at stats or epidemiology to tell if the result is a silk purse or a sow’s ear.
But there are exceptions.
For example Cochrane did a meta-analysis of one paper suggesting acupuncture (proven to be no better than placebo) is effective against mumps. This has zero scientific plausibility and how can you do a meta-analysis of a single paper? Pure Tooth Fairy Science. They also continually suggest homeopathy is worthy of further study based on analysing trials of the efficacy of something that not only doesn’t work, it cannot work. Their paper suggesting Oscillococcinum may work is a classic.
So, these sorts of analyses can be of use when you understand how they work and what the limitations are. A big limitation is that unless you also take into account basic plausibility – which the Cochrane Collaboration doesn’t – you get Tooth Fairy Science.
And they are notoriously wishy-washy; were the Cochrane Collaboration to investigate a murder scene they would insist equal weight be given to the idea that the victim backed onto the knife a dozen times themselves.
Cochrane and Flu
Enter Dr Tom Jefferson. He is the lead author of this paper and heads the Cochrane Acute Respiratory Infections group. He has drawn a lot of criticism in the past for his worship of the randomised controlled trial as being the only standard for judging any medical intervention, his antipathy to the flu vaccine and his appearance on shows promoting quackery and woo – where he made a twat of himself on very many levels.
I would suggest he reads Smith and Pell’s landmark paper, also published by the BMJ, arguing that while RCTs have their place, medicine is more complex than that. (Email me for a full version of that paper, it’s brilliant).
The second author, Peter Doshi, is not an epidemiologist and to get an idea of this man’s pedigree I suggest you read this excellent post by Reuben over at The Poxes Blog.
So What Does It All Mean?
OK. So we have authors who have views at variance with just about all the other experts in the field (they describe flu as ‘benign’, FFS) and who have been publicly decrying Roche not publishing all their data. The publisher is a respectable journal that has also been calling for Roche and others to publish all the data and have publicly and repeatedly kicked Roche in the Jacobs about that data for some years.
Quite simply, these people all have a dog in the fight.
But data is data is data, right?
Wrong. If you interrogate the data for long enough, they will confess to anything you want.
When you do an analysis on a bunch of data from several studies you need to decide which studies you include. And you have to state the criteria for what qualifies. So, in this case they only studied RCTs on people who were otherwise well or had a chronic (as opposed to acute) illness and they excluded anyone with an immunosuppressive disease.
Errrr…. but people in hospital with acute illness and those with a less than tip-top immune function are precisely the sort of people flu will affect most seriously – and often kill directly or by making other conditions worse – and also those most likely to benefit from Tamiflu (I’ll return to that in a minute).
But when you dig deep you also find:
“Because of discrepancies between published and unpublished reports of the same trials, we decided to include only those trials for which we had unabridged clinical study reports.”
They also “ignored published trials” due to risk of bias!
They ignored all published trials? WTF? Overall this is an incredibly strict set of inclusion criteria and – to someone more cynical than I – might even seem designed to exclude any study that could show where Tamiflu has benefit. The overall effect this has on the result I’ll leave to epidemiologists but it seems to me this is not a scholarly review of the data; it’s a political piece with a veneer of science over it to bamboozle any lay reader who will just go by the headlines and the abstract. Which is exactly what the journalists all did. They uncritically regurgitated the Press Release and repeated the Jefferson / Doshi spin that Tamiflu is a worthless drug and a huge waste of public funds.
The whole purpose of Cochrane is to shine a light that will guide clinical decisions – but here the data is being used as a drunk uses a lamp post: for support rather than illumination.
Flu Reality
I have several issues with some basic stuff they get very, very wrong. They claim flu isn’t serious. It is. It kills directly. It kills indirectly. It makes other illnesses worse. (I will do a future post on the flu virus as it is relentlessly fascinating – if you can’t wait, contact me and I’ll send you a rather spiffing flu poster wot I wrote for some healthcare clients recently).
But the key point is they exclude from the study most of the people who will benefit from Tamiflu – people with acute illnesses in hospital where flu kills 25-30% of those who catch it.
Interestingly the BMJ also published a review of ‘real world data’ in the same issue that shows Tamiflu is effective. The difference is this is based on ‘observational’ studies, not RCTs. Like RCTs, observational studies have flaws but they are ‘real life’ – but results from these studies don’t count for Dr Jefferson’s group cos they’re not an RCT. I think it worthy of note this paper on precisely the same issue has had three parts of FA coverage as it doesn’t have the political baggage that comes with a long struggle to make ‘Big Pharma’ ‘fess up plus the public purse being ‘wasted’. And it gives the wrong answer…
Was the £500m wasted?
Yes. But only because there wasn’t a pandemic. And predicting history is easy. It’s the future that’s tricky. Don’t let anyone tell you H1N1 wasn’t pretty bad. It had real potential to turn nasty and still does. Also pandemic flu and seasonal flu are very different animals. We continually expect a repeat of 1919 – where more people were killed than in both World Wars combined in a single season. In that situation neuraminidase inhibitors would be pretty useful. Not perfect, but what else is out there?
Should the Government continue to stockpile neuraminidase inhibitors?
Absolutely yes. Until a universal flu vaccine comes out – which will hopefully be in the next decade. Tamiflu is not perfect and neither are the influenza vaccines. But that doesn’t make them 100% useless. Far from it. Which even this latest paper shows despite its authors’ apparent best efforts to include only data that – curiously – happens to suit biases they have been publicly called on in the past.
I’m not against full publication of trials’ data nor am I suggesting Tamiflu is a wonder drug. And were Roche a bit naughty? I don’t know. But I do know this paper is not a work of scholarship so far as I understand it. Its flaws are so evident in terms of basic science to a non-epidemiologist like me I can’t wait for a proper one to kebab it good and proper.
What’s that noise, Dr Jefferson? Archie Cochrane spinning in his grave, perhaps?
I wouldn’t be surprised if by the next decade the new vaccine discovered is an off spring of paracetamol and tamiflu = paramiflu
By which time the NHS would have reached its shelf life and a new way to defraud the insurance companies emerges
Been some interesting stuff going on with highly conserved CD8 epitopes in some inf A strains (and a bunch of other viruses) but a team at Stanford are now targeting a highly conserved area on the shaft of the haemagglutinin – hence my throwaway line re universal flu vaccine.
If they can get a decent antibody response to that it’ll be easier to produce than a T-cell vaccine – but all the nutters will claim influenza isn’t serious and the vaccine causes autism….
It all boils down to ££$$££
And whilst over time the amount of novel new treatment reduces, the only thing left to do is to re-invent it. This allows the renewal of the patent rights.
The tweak usually is the third carbon along the chain and then have another cockroach review to say; it now has a enhanced targeting mechanism
We’d welcome postpublication peer review of The BMJ’s articles on Tamiflu and Relenza, via bmj.com Rapid Responses (eletters).
Since 1998 The BMJ has published nearly 100,000 Rapid Responses to articles in The BMJ, many of them substantive. Yet the recent articles on neuraminidase inhibitors have had only a handful of Rapid Responses.
Why the silence?
Trish Groves
Head of Research, The BMJ
tgroves@bmj.com
Twitter/trished
Competing interests: I chair The BMJ’s weekly research committee meetings, including the one where we accepted the Cochrane reviews on neuraminidase inhibitors
“If you disagree with our findings, or if you want to carry out your own analysis or just want to see what around 150,000 pages of data look like, they are one click away”
(http://dx.doi.org/10.5061/dryad.77471).
Carl Heneghan, Tom Jefferson, Peter Doshi
http://www.bmj.com/content/348/bmj.g2545?tab=responses
I’m sure the analysis of those data included in the review is spot-on and would withstand scrutiny from those far better qualified to comment than a simple microbiologist.
As I say above – inter alia – it’s the criteria for inclusion that I find most interesting and worthy of comment.
But I must also admit I have neither the skill nor the inclination to take up the kind invitation to analyse 150,000 pages of data. But I take great comfort this is a shortcoming shared by the authors, who state:
“The main limitation of our study is our relative inexperience in dealing with large quantities of information and our lack of familiarity with certain trial documents such as blank case report forms. A further limitation of our review is that the methods we have developed to assess and summarise information from clinical study reports may not apply to non-industry trials (which may not be reported in clinical study reports).”
Also, in the interests of openness I have no conflicts of interest: I have never received a penny from a pharmaceutical company, I do not work for a competitor of the BMJ or any other interested parties and – for the avoidance of doubt – I have never contributed to any Cochrane Collaboration review or been associated with Cochrane etc etc.
I have no dog in the fight here and I only make this point to follow @Trish’s laudable precedent – and I’m sure anyone else who takes time to comment will do likewise if it’s relevant.
Again, very sincere thanks for all the comments. That anyone would take time so to do I find humbling and bewildering in equal measure.
While I also have no desire to take up Trish Groves’ and Peter Doshi’s rhetorical offer to review hundreds of thousands of pages of CSRs (Unlike Doshi et al, I don’t have salary funding to spend the next 6-8 months doing that), I also found the process troubling.
Having made Tamiflu the poster child for the AllTrials campaign, what were the chances that Cochrane would then undercut its own position by issuing a review stating that it came to conclusions essentially identical to those reported by the Roche analysis?
The close connection between BMJ and Cochrane in pursuing the data release, and the steady stream of editorial content critical of Roche and Tamiflu over the last two years, creates serious questions about the peer review process as well.
Ideally, the data once acquired would have been reviewed by an independent group and published in a journal without ties to the controversy.
Historically, the role of medical journals has been to provide a neutral forum for the discussion of scientific and medical data. BMJ in recent years has become highly politicized, and its editors increasingly combative and dismissive of dissent from what they have defined as political correctness. Its a different approach, I guess time will tell whether it is successful and whether it leads to better medicine.
John,
“The poster child for the AllTrials campaign” – thank you for this.
There’s another well-informed commentary which provides more wood to your fire with some interesting links to reviews of the data that are a little less ~ahem~ ‘focussed’ than Jefferson, Doshi et al:
http://www.socialsciencespace.com/2014/04/tamiflu-and-the-ethics-of-the-british-medical-journal/
I’ll try to stick to the microbiology; current neuraminidase inhibitors aren’t perfect. We all know that. How imperfect? A pandemic would be the ultimate test but I agree a proper estimate can only come if ALL the available data is now properly analysed by someone untainted by all this – and then published in a high-impact journal other than BMJ. And using a methodology without the huge gaps that even a simple microbiologist such as I can drive a coach and horses through.
And other commentators have also noted BMJ has been digging itself deeper into a hole on this issue – one wonders when they might stop.
Anyway, the data (linked to in the link above and elsewhere) shows that people will die unnecessarily in a pandemic if neuraminidase inhibitors are not readily available. (Unless something better turns up in the meantime or the then current seasonal vaccine is effective against the pandemic strain and available in abundance – which is incredibly unlikely for many self-evident reasons.)
So one wonders not just how on earth an academic journal such as BMJ could allow itself to get so mired in this. The aspect that really beggars belief is adopting a position where an increase in preventable morbidity and mortality from a pandemic – inevitable without rapid access to neuraminidase inhibitors – is a price worth paying because winning a war of words with Roche is apparently just too important.