[HSF] More on the Traysylol saga
Ben Bidstrup
benjamin.bidstrup at bigpond.com
Mon Oct 9 12:38:08 EDT 2006
Ani,
It is all about statistics. To examine the data,
you need the right tools. Herein lies the
difficulty. How do you compare groups that have
had either one drug (aprotinin, cyclokapron or
amicar ) or a combination of them, or no therapy?
As we have discussed, the indications for each of
these can vary from surgeon to surgeon, from
hospital to hospital and from country to country.
Not all drugs were available in all countries.
McSPI is a huge database and has a wealth of
information that can be gleaned from it. This
requires it to be analyzed in the appropriate
manner There are methods to do this. Logistic
regression and propensity analysis are possible
ways. These methods depend in part on use of
collected data only.
If an author uses complex statistics, will the
reader have the knowledge or patience to go
through it in detail and dissect the methodology?
In this case many of the HS Forum particpants did
not even go past the abstract. They also took
note of what was published on the front pages of
the WSJ and NY Times. Some took at at face value
and said this is a bad drug, I will stop using
it. Some said this is a bad study and threw it in
the bin and got on with the job. Some were more
analytical and tried to make sense of the data
and methods. They realized the limitations of the
whole 'study' which after all was a retrospective
data dredging exercise which failed to apply
basic principles.
Tea has mentioned cause and effect. An RCT is
said to be the only way to determine this. By
controlling treatment allocation to a
'homogeneous' group, the unknown differences in
the study population are said to even out and not
be an influence on the outcomes. An observational
study can indicate an association which can then
be tested in an RCT or better designed
observational study.
Look at the August edition of JTCVS. There is a
wealth of discussion about surgical studies and
the reporting of them. The CONSORT statement is
about RCTs but much of this can apply to any
study.
>Dear Dr Levinson
>
>I am not sure statistics has much to do with
>this one. The statistics in the NEJM Aprotinin
>paper to my knowledge has not been a subject of
>criticism. The problem with Mangano's paper is
>one of flawed methodology and not flawed
>statistics. Aside from the methodological flaws
>in the paper, they make the mistake of
>publishing a paper that uses the wrong method
>(retrospective study rather than RCT) to assess
>safety profile of a drug and they allowed the
>authors, as you said, to make inflammatory
>conclusions. The debate about statistics,
>P-values etc does not really apply to this
>paper. The dilemmas you mention are things that
>should come up in discussion and interpretation
>of data.
>
>I see no problem too with surgeons publishing in
>NEJM - we would all rather see our landmark
>papers published there than in any surgical
>journal. The problem we are dealing with is a
>biased editorial board with their own agenda. It
>would be very easy for them to seek a surgical
>review of the manuscript (and I suspect they
>did).
>
>I do not think our cardiothoracic journals are
>any better - most of the flaws we see in the
>NEJM we also see in the surgical journals and
>certainly we see far more poorly designed,
>flawed methodology, and 'inappropriately
>published' articles in our journals than are in
>the NEJM. In our specialty journals, who you are
>or where you work can guarantee you to get any
>manuscript published regardless of content. The
>difference though is that people (including non
>surgeons) regard NEJM as holy grail so NEJM has
>a greater responsibility to ensure they do not
>mislead the public compared to specialist
>journals.
>
>Ani Anyanwu
>
>
>----- Original Message -----
> From: Mark Levinson, MD<mailto:mmlevinson at hsforum.com>
> To: OpenHeart-L at lists.hsforum.com<mailto:OpenHeart-L at lists.hsforum.com>
> Sent: Sunday, October 08, 2006 1:45 AM
> Subject: Re: [HSF] More on the Traysylol saga
>
>
>
> On Oct 5, 2006, at 3:21 AM, Ben Bidstrup wrote:
>
> > The power that Journal Editors have is such that they must be
> > answerable for their actions. ........ Now if a statistician who
> > has such an exalted post cannot indicate the weaknesses in this
> > study, then he should withdraw his paper and offer an apology, to
> > all readers of the journal, to the public and to patients who
> > suffered as a result of the non use of aprotinin. He should also be
> > joined in any action.
>
> Ben:
>
> I appreciate your comments. We have discussed the role of
> statistical consultation for each manuscript submitted to the HSF
> journal during our Board meetings. Although this seems like the
> answer for blocking papers with bad science, there are problems.
> First of all, not all statisticians agree. There are many tests
> that can be applied to the same data, of which the conclusions of
> each test may be different. In some cases, one statistical
> consultant will claim a test is not even valid for the data, while
> the other insists it is the test of choice. Who can arbitrate
> these differences?
>
> Then there is the consistent dilmena of what to do with a statistical
> result. Does a p value of 0.05 mean that the study has proved
> cause and effect?
>
> I agree that statistical analysis is an improvement over the era
> where there was no statistics. However, there are misused as
> well. One of the problems is that now everyone intrinsically trusts
> the final conclusions of the statistical analysis. If a p value
> is found, then readers believe it establishes truth, and even worse
> "cause and effect". Some things that could not be related are
> forced into a relationship by the presence of a statistical result.
>
> For example, if you were to perform a study about the mortality rate
> of intubation, you will find the hospital mortality is higher for
> intubated patients than non-intubated patients. Does this mean
> that intubation caused their deaths? Even if multivariant analysis
> shows that intubation is still associated with a higher mortality
> rate, this does *not* establish cause and effect. We all know that
> the cause of death is the disease process which ends up with
> intubation and the tube is not the cause. Cause and efffect is
> only established be eliminating each variable and repeating the
> experiment until the observation goes away. Since this is
> impossible in patient care situations, we utilize less perfect
> methods initially developed in the behavioral sciences and poorly
> adapted to our needs in surgery.
>
> Recently there was a paper discussing the falacies of statistical
> analysis and centering on the t-test (the most common test used in
> reporting differences between groups). I was amazed at how limited
> the scope of the t-test actually is. After reading this paper, I
> realized that almost all of the t-tests we use are based on incorrect
> methods or assumptions. So, we are standing on soft ground with
> even the most basic statistical analysis and we must understand this....
>
>
> > Readers of the HSF who feel so inclined, should express their
> > dismay at this turn of events and send an open email to the editors
> > of the Journal and to Mr David Armstrong,
>(David.Armstrong at wsj.com<mailto:David.Armstrong at wsj.com>)
> > the author of the article in the Wall Street Journal (Hal's
> > favourite medical source). Furthermore, you should encourage your
> > library to cancel their subscription to the NEJM. The Massachusetts
> > Medical Society should also be contacted.
> >
>
> Personally, the main failure of the NEJM is that they published a
> paper on a surgical subject when they are a medical journal. They
> are no experts in surgical sciences and should have referred the
> author to another journal. They are not set up to evaluate
> articles on surgical subspecialties. The results analyzed were
> surgical endpoints. I do not know who the reviewers were, but I
> doubt any surgeon, and particularly any cardiac surgeon, reviewed
> this paper prior to publication. Also, whoever reviewed this paper
> exceeded the normal tolerance for personal diatribes in the
> discussion section. Inflammatory statements made in the
> discussion section of the Mangano paper should have raised a flag and
> caused a secondary review, or be returned to the author for
> deletion. It is my opinion that if the same paper has been
> submitted to the HSF board, it would have been severely criticized or
> rejected outright. But the HSF has cardiothoracic surgeons on its
> Board while the NEJM does not. And of course Mangano chose not to
> submit his paper to the Annals, JTCVS or another respected
> cardiothoracic journal. I suspect he knew that such an audience
> would have no problem finding the weaknesses and rejecting the paper
> apriori.
>
> Just my opinion....
>
>
> Mark Levinson, M.D.
> Founder, Editor-in-Chief
> The Heart Surgery Forum®
> URL: http://www.hsforum.com<http://www.hsforum.com/>
> Email: mmlevinson at hsforum.com<mailto:mmlevinson at hsforum.com>
>
>
> _______________________________________________
> OpenHeart-L mailing list
>
> Send postings to:
> OpenHeart-L at lists.hsforum.com<mailto:OpenHeart-L at lists.hsforum.com>
>
> To UNSUBSCRIBE, to CHANGE email address, or to view archives:
>
>http://mmp.cjp.com/mailman/listinfo/openheart-l<http://mmp.cjp.com/mailman/listinfo/openheart-l>
>
> All messages transmitted by the OpenHeart-L are subject to the policies and
> disclaimers posted at:
> http://www.hsforum.com/listdisclaim<http://www.hsforum.com/listdisclaim>
> -----------------------------------------
>_______________________________________________
>OpenHeart-L mailing list
>
>Send postings to:
> OpenHeart-L at lists.hsforum.com
>
>To UNSUBSCRIBE, to CHANGE email address, or to view archives:
>http://mmp.cjp.com/mailman/listinfo/openheart-l
>
>All messages transmitted by the OpenHeart-L are subject to the policies and
>disclaimers posted at:
>http://www.hsforum.com/listdisclaim
>-----------------------------------------
--
Ben Bidstrup FRACS FRCSEd FEBCTS
Consultant Cardiothoracic Surgeon
More information about the OpenHeart-L
mailing list