[HSF] More on the Traysylol saga

Sun Oct 8 10:11:37 EDT 2006

Dear Dr Levinson

I am not sure statistics has much to do with this one. The statistics in the NEJM Aprotinin paper to my knowledge has not been a subject of criticism. The problem with Mangano's paper is one of flawed methodology and not flawed statistics. Aside from the methodological flaws in the paper, they make the mistake of publishing a paper that uses the wrong method (retrospective study rather than RCT) to assess safety profile of a drug and they allowed the authors, as you said, to make inflammatory conclusions. The debate about statistics, P-values etc  does not really apply to this paper. The dilemmas you mention are things that should come up in discussion and interpretation of data.

I see no problem too with surgeons publishing in NEJM - we would all rather see our landmark papers published there than in any surgical journal. The problem we are dealing with is a biased editorial board with their own agenda. It would be very easy for them to seek a surgical review of the manuscript (and I suspect they did).

I do not think our cardiothoracic journals are any better - most of the flaws we see in the NEJM we also see in the surgical journals and certainly we see far more poorly designed, flawed methodology, and 'inappropriately published' articles in our journals than are in the NEJM. In our specialty journals, who you are or where you work can guarantee you to get any manuscript published regardless of content. The difference though is that people (including non surgeons) regard NEJM as holy grail so NEJM has a greater responsibility to ensure they do not mislead the public compared to  specialist journals.

Ani Anyanwu

----- Original Message ----- 
  From: Mark Levinson, MD<mailto:mmlevinson at hsforum.com> 
  To: OpenHeart-L at lists.hsforum.com<mailto:OpenHeart-L at lists.hsforum.com> 
  Sent: Sunday, October 08, 2006 1:45 AM
  Subject: Re: [HSF] More on the Traysylol saga

  On Oct 5, 2006, at 3:21 AM, Ben Bidstrup wrote:

  > The power that Journal  Editors have is such that they must be  
  > answerable for their actions. ........ Now if a statistician who  
  > has such an exalted post cannot indicate the weaknesses in this  
  > study, then he should withdraw his paper and offer an apology, to  
  > all readers of the journal, to the public and to patients who  
  > suffered as a result of the non use of aprotinin. He should also be  
  > joined in any action.

  Ben:

  I appreciate your comments.     We have discussed the role of  
  statistical consultation for each manuscript submitted to the HSF  
  journal during our Board meetings.    Although this seems like the  
  answer for  blocking papers with bad science, there are problems.     
  First of all, not all statisticians agree.    There are many tests  
  that can be applied to the same data, of which the conclusions of  
  each test may be different.    In some cases, one statistical  
  consultant will claim a test is not even valid for the data, while  
  the other insists it is the test of choice.    Who can arbitrate  
  these differences?

  Then there is the consistent dilmena of what to do with a statistical  
  result.    Does a p value of 0.05 mean that the study has proved  
  cause and effect?

  I agree that statistical analysis is an improvement over the era  
  where there was no statistics.   However, there are misused as  
  well.   One of the problems is that now everyone intrinsically trusts  
  the final conclusions of the statistical analysis.     If a p value  
  is found, then readers believe it establishes truth, and even worse  
  "cause and effect".    Some things that could not be related are  
  forced into a relationship by the presence of a statistical result.

  For example, if you were to perform a study about the mortality rate  
  of intubation, you will find the hospital  mortality is higher for  
  intubated patients than non-intubated patients.       Does this mean  
  that intubation caused their deaths?    Even if multivariant analysis  
  shows that intubation is still associated with a higher mortality  
  rate, this does *not* establish cause and effect.   We all know that  
  the cause of death is the disease process which ends up with  
  intubation and the tube is not the cause.       Cause and efffect is  
  only established be eliminating each variable and repeating the  
  experiment until the observation goes away.     Since this is  
  impossible in patient care situations, we utilize less perfect  
  methods initially developed in the behavioral sciences and poorly  
  adapted to our needs in surgery.

  Recently there was a paper discussing the falacies of statistical  
  analysis and centering on the t-test (the most common test used in  
  reporting differences between groups).    I was amazed at how limited  
  the scope of the t-test actually is.     After reading this paper, I  
  realized that almost all of the t-tests we use are based on incorrect  
  methods or assumptions.       So, we are standing on soft ground with  
  even the most basic statistical analysis and we must understand this....

  > Readers of the HSF who feel so inclined, should express their  
  > dismay at this turn of events and send an open email to the editors  
  > of the Journal and to Mr David Armstrong, (David.Armstrong at wsj.com<mailto:David.Armstrong at wsj.com>)  
  > the author of the article in the Wall Street Journal (Hal's  
  > favourite medical source). Furthermore, you should encourage your  
  > library to cancel their subscription to the NEJM. The Massachusetts  
  > Medical Society should also be contacted.
  >

  Personally, the main failure of the NEJM is that they published a  
  paper on a surgical subject when they are a medical journal.  They  
  are no experts in surgical sciences and should have referred the  
  author to another journal.    They are not set up to evaluate  
  articles on surgical subspecialties.      The results analyzed were  
  surgical endpoints.    I do not know who the reviewers were, but I  
  doubt any surgeon, and particularly any cardiac surgeon, reviewed  
  this paper prior to publication.    Also, whoever reviewed this paper  
  exceeded the normal tolerance for personal diatribes in the  
  discussion section.       Inflammatory statements made in the  
  discussion section of the Mangano paper should have raised a flag and  
  caused a secondary review, or be returned to the author for  
  deletion.       It is my opinion that if the same paper has been  
  submitted to the HSF board, it would have been severely criticized or  
  rejected outright.    But the HSF has cardiothoracic surgeons on its  
  Board while the NEJM does not.    And of course Mangano chose not to  
  submit his paper to the Annals, JTCVS or another respected  
  cardiothoracic journal.   I suspect he knew that such an audience  
  would have no problem finding the weaknesses and rejecting the paper  
  apriori.

  Just my opinion....

  Mark Levinson, M.D.
  Founder, Editor-in-Chief
  The Heart Surgery Forum®
  URL:   http://www.hsforum.com<http://www.hsforum.com/>
  Email:  mmlevinson at hsforum.com<mailto:mmlevinson at hsforum.com>

  _______________________________________________
  OpenHeart-L mailing list

  Send postings to:
   OpenHeart-L at lists.hsforum.com<mailto:OpenHeart-L at lists.hsforum.com>

  To UNSUBSCRIBE, to CHANGE email address, or to view archives:
  http://mmp.cjp.com/mailman/listinfo/openheart-l<http://mmp.cjp.com/mailman/listinfo/openheart-l>

  All messages transmitted by the OpenHeart-L are subject to the policies and 
  disclaimers posted at:
  http://www.hsforum.com/listdisclaim<http://www.hsforum.com/listdisclaim>
  -----------------------------------------