[HSF] More on the Traysylol saga
Michael Firstenberg
msfirst at gmail.com
Sun Oct 8 18:49:08 EDT 2006
A major problem that I have with all of this kind of stuff is the
statistics.
A very good friend of mine (a woman no less) who was a PhD researcher
used to say that "statistics are like a string bikini - what they
reveal is interesting, but what they hide is....." well lets keep
this a family show.
We are all smart people, and I would like to think of myself as not
to shabby myself, but the whole field of medical statistics has just
gotten out of control in how complicated it is. Yes, I read these
articles and usually skip over the statistics part of the methods - I
trust (yeah, big joke) that the editorial boards are doing their
homework and the techniques are proper - but who knows. Now with all
of this propensity scoring and whatever the trendy test du jour is
combined with whatever crafty manipulation of the hypothesis, data,
and/or conclusions (ad hoc analysis, sub-group hypothesis testing,
non-superiority testing, yadda yadda yadda), not to mention vague
definitions (anyone want to give a standard definition of 3+ MR?) -
this stuff has gotten out of control.
...am I missing something here?
michael
On Oct 8, 2006, at 9:11 AM, Ani Anyanwu wrote:
> Dear Dr Levinson
>
> I am not sure statistics has much to do with this one. The
> statistics in the NEJM Aprotinin paper to my knowledge has not been
> a subject of criticism. The problem with Mangano's paper is one of
> flawed methodology and not flawed statistics. Aside from the
> methodological flaws in the paper, they make the mistake of
> publishing a paper that uses the wrong method (retrospective study
> rather than RCT) to assess safety profile of a drug and they
> allowed the authors, as you said, to make inflammatory conclusions.
> The debate about statistics, P-values etc does not really apply to
> this paper. The dilemmas you mention are things that should come up
> in discussion and interpretation of data.
>
> I see no problem too with surgeons publishing in NEJM - we would
> all rather see our landmark papers published there than in any
> surgical journal. The problem we are dealing with is a biased
> editorial board with their own agenda. It would be very easy for
> them to seek a surgical review of the manuscript (and I suspect
> they did).
>
> I do not think our cardiothoracic journals are any better - most of
> the flaws we see in the NEJM we also see in the surgical journals
> and certainly we see far more poorly designed, flawed methodology,
> and 'inappropriately published' articles in our journals than are
> in the NEJM. In our specialty journals, who you are or where you
> work can guarantee you to get any manuscript published regardless
> of content. The difference though is that people (including non
> surgeons) regard NEJM as holy grail so NEJM has a greater
> responsibility to ensure they do not mislead the public compared
> to specialist journals.
>
> Ani Anyanwu
>
>
> ----- Original Message -----
> From: Mark Levinson, MD<mailto:mmlevinson at hsforum.com>
> To: OpenHeart-L at lists.hsforum.com<mailto:OpenHeart-
> L at lists.hsforum.com>
> Sent: Sunday, October 08, 2006 1:45 AM
> Subject: Re: [HSF] More on the Traysylol saga
>
>
>
> On Oct 5, 2006, at 3:21 AM, Ben Bidstrup wrote:
>
>> The power that Journal Editors have is such that they must be
>> answerable for their actions. ........ Now if a statistician who
>> has such an exalted post cannot indicate the weaknesses in this
>> study, then he should withdraw his paper and offer an apology, to
>> all readers of the journal, to the public and to patients who
>> suffered as a result of the non use of aprotinin. He should also be
>> joined in any action.
>
> Ben:
>
> I appreciate your comments. We have discussed the role of
> statistical consultation for each manuscript submitted to the HSF
> journal during our Board meetings. Although this seems like the
> answer for blocking papers with bad science, there are problems.
> First of all, not all statisticians agree. There are many tests
> that can be applied to the same data, of which the conclusions of
> each test may be different. In some cases, one statistical
> consultant will claim a test is not even valid for the data, while
> the other insists it is the test of choice. Who can arbitrate
> these differences?
>
> Then there is the consistent dilmena of what to do with a
> statistical
> result. Does a p value of 0.05 mean that the study has proved
> cause and effect?
>
> I agree that statistical analysis is an improvement over the era
> where there was no statistics. However, there are misused as
> well. One of the problems is that now everyone intrinsically
> trusts
> the final conclusions of the statistical analysis. If a p value
> is found, then readers believe it establishes truth, and even worse
> "cause and effect". Some things that could not be related are
> forced into a relationship by the presence of a statistical result.
>
> For example, if you were to perform a study about the mortality rate
> of intubation, you will find the hospital mortality is higher for
> intubated patients than non-intubated patients. Does this mean
> that intubation caused their deaths? Even if multivariant
> analysis
> shows that intubation is still associated with a higher mortality
> rate, this does *not* establish cause and effect. We all know that
> the cause of death is the disease process which ends up with
> intubation and the tube is not the cause. Cause and efffect is
> only established be eliminating each variable and repeating the
> experiment until the observation goes away. Since this is
> impossible in patient care situations, we utilize less perfect
> methods initially developed in the behavioral sciences and poorly
> adapted to our needs in surgery.
>
> Recently there was a paper discussing the falacies of statistical
> analysis and centering on the t-test (the most common test used in
> reporting differences between groups). I was amazed at how
> limited
> the scope of the t-test actually is. After reading this paper, I
> realized that almost all of the t-tests we use are based on
> incorrect
> methods or assumptions. So, we are standing on soft ground
> with
> even the most basic statistical analysis and we must understand
> this....
>
>
>> Readers of the HSF who feel so inclined, should express their
>> dismay at this turn of events and send an open email to the editors
>> of the Journal and to Mr David Armstrong,
>> (David.Armstrong at wsj.com<mailto:David.Armstrong at wsj.com>)
>> the author of the article in the Wall Street Journal (Hal's
>> favourite medical source). Furthermore, you should encourage your
>> library to cancel their subscription to the NEJM. The Massachusetts
>> Medical Society should also be contacted.
>>
>
> Personally, the main failure of the NEJM is that they published a
> paper on a surgical subject when they are a medical journal. They
> are no experts in surgical sciences and should have referred the
> author to another journal. They are not set up to evaluate
> articles on surgical subspecialties. The results analyzed were
> surgical endpoints. I do not know who the reviewers were, but I
> doubt any surgeon, and particularly any cardiac surgeon, reviewed
> this paper prior to publication. Also, whoever reviewed this
> paper
> exceeded the normal tolerance for personal diatribes in the
> discussion section. Inflammatory statements made in the
> discussion section of the Mangano paper should have raised a flag
> and
> caused a secondary review, or be returned to the author for
> deletion. It is my opinion that if the same paper has been
> submitted to the HSF board, it would have been severely
> criticized or
> rejected outright. But the HSF has cardiothoracic surgeons on its
> Board while the NEJM does not. And of course Mangano chose not to
> submit his paper to the Annals, JTCVS or another respected
> cardiothoracic journal. I suspect he knew that such an audience
> would have no problem finding the weaknesses and rejecting the paper
> apriori.
>
> Just my opinion....
>
>
> Mark Levinson, M.D.
> Founder, Editor-in-Chief
> The Heart Surgery Forum®
> URL: http://www.hsforum.com<http://www.hsforum.com/>
> Email: mmlevinson at hsforum.com<mailto:mmlevinson at hsforum.com>
>
>
> _______________________________________________
> OpenHeart-L mailing list
>
> Send postings to:
> OpenHeart-L at lists.hsforum.com<mailto:OpenHeart-L at lists.hsforum.com>
>
> To UNSUBSCRIBE, to CHANGE email address, or to view archives:
> http://mmp.cjp.com/mailman/listinfo/openheart-l<http://
> mmp.cjp.com/mailman/listinfo/openheart-l>
>
> All messages transmitted by the OpenHeart-L are subject to the
> policies and
> disclaimers posted at:
> http://www.hsforum.com/listdisclaim<http://www.hsforum.com/
> listdisclaim>
> -----------------------------------------
> _______________________________________________
> OpenHeart-L mailing list
>
> Send postings to:
> OpenHeart-L at lists.hsforum.com
>
> To UNSUBSCRIBE, to CHANGE email address, or to view archives:
> http://mmp.cjp.com/mailman/listinfo/openheart-l
>
> All messages transmitted by the OpenHeart-L are subject to the
> policies and
> disclaimers posted at:
> http://www.hsforum.com/listdisclaim
> -----------------------------------------
More information about the OpenHeart-L
mailing list