Taking A Leaf Out Of An Old Book

The Decameron of poor research

BMJ 2004; 329 doi: https://doi.org/10.1136/bmj.329.7480.1436 (Published 16 December 2004) Cite this as: BMJ 2004;329:1436
  1. Vance W Berger, mathematical statistician1,
  2. John P A Ioannidis, chairman (jioannid{at}cc.uoi.gr)2
  1. 1 Biometry Research Group, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA
  2. 2 Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece
  1. Correspondence to: J P A Ioannidis

    Most gracious readers, two boring researchers present a series of pitiful tales—covert and overt fraud, petty misdeeds, and misconceptions by honest and not so honest researchers, professors, industry sponsors, bureaucrats, and other legendary adventurers of modern science

    “Most gracious ladies and gentlemen, knowing that you are by nature pitiful, we know that in your judgment this work will seem to have a painful and sad origin.” It was not in the company of fair and elegant young ladies and pleasant knights in the Tuscan countryside that this imitation of Boccaccio originated. Simply, dismayed at the scandals of poor research, two boring researchers exchanged emails to narrate brief stories for consolation. We tried to beat each other in narrating the worst tale possible. Stories of cheating, betrayal, unrequited love, tricks, and opportunistic adventure—in brief, any story of bad research was eligible for the project.

    Let there be no mistake: we did not try to compete against the Florentine genius. Conversely, we were fully confident that we could write a very poor paper indeed. To avoid embarrassing people or institutions, heroes are named after the Decameron. After all, these tales have reached us from very different, truly international sources. For practical purposes they could have happened anywhere.

    First tale

    Federigo, a research fellow, pleads for help to defend Vancouver; yet, after a secret serious discussion on the deeper philosophical meaning of scientific authorship, he spontaneously applauds and kneels in admiration before his Master

    Aldobrandino was collaborating with Federigo, a young physician trained in world class institutions. After lots of work, tons of analyses, and several revisions, Aldobrandino sent the final manuscript to Federigo for his approval. In response, he got a confidential email. “Dear friend,” Federigo wrote, “this has been a wonderful collaboration. However, I have bad news. There are seven authors now. Messer Guglielmo, my director, is eating my flesh, but I don't want my career to end. Monna Nonna, my wife, needs papers to get a job. Peronella is directing the lab next door—we must be kind to our neighbours and build collaborations. Tedaldo, my young fellow, also must build his curriculum vitae. Lastly, Messer Guglielmo had another co-author to add. He forgot his name, but he will let me know tomorrow.” A postscript followed: “Have you seen my work at XXX? Many consider XXX the top university worldwide. In my best paper, of seven authors, three contributed nothing, and one had no clue of the project's existence until he received a reprint.”

    Scientifically speaking, Aldobrandino went bananas. Federigo apologised and pleaded for help to face the bloodsucking Messer Guglielmo. He promised that, with Aldobrandino at his side, he would stand firmly against his director in defence of the Vancouver authorship criteria. In this spirit, a meeting was arranged where Alobrandino and Federigo met with Messer Guglielmo two days later. Federigo asked to talk first. Making a request to talk when your director is present is indeed a valiant act, and Aldobrandino was pleased that his young colleague seemed so determined. However, instead of praising Vancouver, Federigo started praising his worthy director. He made a deep and moving confession, asked for forgiveness, and urged that his Master should be the senior author and sole arbitrator of authorship.

    Albobrandino understood that something must have happened the previous day that had changed the stance of his colleague; he asked his name to be removed from the manuscript and thanked them both. Much later the paper was published without his name.

    Second tale

    A completed non-randomised study by Chichibio and colleagues is rejected, but then it miraculously becomes randomised and gets published with full honours

    Chichibio submitted to a prestigious journal the manuscript he had written with countless others, as in the previous tale. One reviewer noticed that, although the title and abstract promised that the study was randomised, the methods section clearly revealed the contrary. This reviewer mentioned the discrepancy and later saw that this comment had, in fact, made it to the authors.

    The manuscript was rejected, but it was published eventually in another journal. Just as Rodney Danger-field swore that he would fix it so that his wife would never be on the couch with another man again, and then sold the couch, so did Chichibio and colleagues fix the problem. The word “randomised” still appeared prominently in the title and abstract. However, the authors were sufficiently considerate to readers to relieve them of the unnecessary trouble of noting that this was not a randomised trial. The contradictory wording from the methods had been appropriately erased. As for the discussion section, this also gracefully expounded the merits of this unique randomised clinical experiment. The medical literature was being enriched with yet another superb study without a section on limitations.

    Third tale

    Efigenia, a young physician in search of a faculty job, is forced by seasoned professors to repent the mortal sin of having been involved in evidence based medicine and considers becoming a nun instead

    Well, I doubt randomised trials mean much to many physicians worldwide. With solid clinical training and several publications in prestigious journals, Efigenia sought a junior clinical faculty job.

    The interviewing Professor lamented the fact that Efigenia was “working on controversial topics, randomised trials and even meta-analyses” and gave seasoned advice to the youngster: “Gone is your reputation as a physician.” Efigenia felt her interest in evidence based medicine was worse than filthy extramarital affairs. Well, even if Efigenia were to repent (an act which she certainly should do, at a minimum), the damage was probably irreversible. “It is so sad, you won't ever be able to think straight again after this exposure,” claimed the Professor, nodding his head. Nevertheless, it all ended in a friendly tone: “I am terribly sorry, we simply don't want mathematicians here.”

    Notwithstanding the first shocking disappointment, the second interview went better. “Seat please,” the Associate Professor said. “I looked at your CV.”

    “This is good news,” Efigenia thought.


    Embedded Image

    Portraiture of sublime research: lofty academics gracefully beguiled by idealistic sponsors (The Decameron (1916) by John William Waterhouse, held at the Lady Lever Art Gallery, Port Sunlight, Merseyside)

    Credit: BRIDGEMAN ART LIBRARY

    “So, I understand you have worked on, aaah…”the Associate Professor stopped trying to find the word. “Aaah, what's the name, aaah, yes, β analysis.”

    “Yes, of course,” Efigenia replied, thinking regretfully, that she had worked and wasted her life on α trials, β analysis, and other types of γ particles.

    The next she expected to hear was the Hamlet motto, “Get thee to a nunnery,” but let us stop here to avoid mixing Shakespearean and Italian literature.

    Fourth tale

    Beltramo, a regulatory agency statistician, in vain tries to prove his points; at the end, bad research is justly rewarded by regulatory approval and publication in a prestigious journal

    Who's afraid of evidence based medicine? At a regulatory meeting to discuss the approval of a drug, Beltramo, the regulatory agency statistician, pointed out that the singular pivotal randomised trial was badly flawed. Masking was forfeited, explained Beltramo. “Come on, we have seen worse,” was the reply he got.

    Extract from the English translation of The Decameron by J M Rigg (1903)*

    First day: Novel IV

    A monk lapses into a sin meriting the most severe punishment, justly censures the same fault in his abbot, and thus evades the penalty.…

    In the not very remote district of Lunigiana there flourished formerly a community of monks more numerous and holy than is there to be found to-day, among whom was a young brother, whose vigour and lustihood neither the fasts nor the vigils availed to subdue.

    One afternoon, while the rest of the confraternity slept, our young monk took a stroll around the church, which lay in a very sequestered spot, and chanced to espy a young and very beautiful girl, a daughter, perhaps, of one of the husbandmen of those parts, going through the fields and gathering herbs as she went. No sooner had he seen her than he was sharply assailed by carnal concupiscence, insomuch that he made up to and accosted her; and (she hearkening) little by little they came to an understanding, and unobserved by any entered his cell together.

    Now it so chanced that, while they fooled it within somewhat recklessly, he being overwrought with passion, the abbot awoke and passing slowly by the young monk's cell, heard the noise which they made within, and the better to distinguish the voices, came softly up to the door of the cell, and listening discovered that beyond all doubt there was a woman within. His first thought was to force the door open; but, changing his mind, he returned to his chamber and waited until the monk should come out.

    Allocation concealment was subverted, Beltramo continued, but the administrators asked for a dictionary: “Allocation concealment? What is this?”


    Embedded Image

    Frontispiece from a 17th century English translation of The Decameron (private collection)

    Several patients were randomised from the wrong stratum. Practically, the investigators had chosen at their will the patients “randomised” in each arm. What did the agency have to say for this? “We are late for lunch,” the chief officer replied.

    Eventually, showing its well founded and well funded respect for bad research, the regulatory agency approved the drug that was anxiously anticipated to make a big splash in the market. Only utopians would argue nowadays that the money is not more important than the science. However, a crust of scientific glory is not bad either. It is thus not surprising that a few months later, the trial found its way into the pages of a prestigious journal. The journal actually seemed to have done an excellent job in silencing the flaws of the study.

    Fifth tale

    Despite the reluctance of Cimone, the methodologist, a team of top researchers is rushing heroically to meet the abstract submission deadline of an international meeting—at all costs

    All trials have their problems. Data were still being collected when the principal investigator telephoned Cimone, the methodologist. The deadline for the international conference was imminent. Some centres had sent no data, others were uncertain whether they had even participated in the study, but then “Who cares about the data, just do it,” cried the principal investigator on the phone. Cimone declined to meet these high standards.

    Next morning, Lazzarino, a diligent clinician working with the principal investigator, showed Cimone the analyses he himself had run overnight. Cimone took a look. It soon became clear to him that all these analyses shared something in common: they were all wrong. After abusing all principles of statistics, Lazzarino had made the drug seem effective, despite it being completely ineffective. In vain Cimone tried to explain. Lazzarino remained unmoved. Cimone finally exploded, “We cannot mislead others and ourselves so blatantly.”

    Lazzarino responded quietly, “OK, let me ask. Have any others ever made such statistical mistakes in published papers, in good journals?”

    Cimone replied without second thought, “Sure, these are common major mistakes.”

    Lazzarino jumped from his seat: “If others have made these errors, why not us? We are fine, it's on its way.” He thanked Cimone kindly. “Sorry, I must run, the abstract is due by 5 pm.” The presentation made a great impression at the international conference.

    Sixth tale

    Alessandro, a statistician, fails to massage appropriately his mistress (the data) and insists on disseminating (rather than inseminating) the results of a study that would be unfavourable to the sponsor; there is no happy ending here

    Contrary to the previous story, Alessandro did perform an analysis of his data in exact accordance with what the study protocol had specified. The protocol was well thought, the study was well conducted, everything was in place. Analysing the data of a well designed and well conducted study is a superb, if uncommon, experience nowadays. Well, this analysis showed that the product developed by his employer was not, in fact, superior to the competitor, as had been hoped. Still, Alessandro reasoned, the facts are the facts.

    This position turned out to be unpopular. The Director of Alessandro's department told him to challenge this result. When Alessandro asked how to challenge it, he was told, with some exacerbation on the part of the Director (a more seasoned employee would have known, without having to ask), to play around until a model is found that proves “what we all know to be true.” Alessandro had a very short tenure with this particular institution.

    Seventh tale

    Guiscardo wastes his chance of a lifetime to become a star of the medical-industrial complex; it is only fair that he then disappears in vulgar anonymity

    What we should take as the key message from the previous story is that one must respect the powerful. Guiscardo was a young, hard working physician-scientist. He was pleased and flattered when Professor Tancred called. The school was organising a research conference, and Guiscardo was invited to give a plenary. Guiscardo proposed as a topic, “Medical errors and adverse effects of medical treatment.”

    After some silence, Tancred promised to “discuss this with the other organisers.” Several days later, he called back. A different topic would really be preferable. Guiscardo was pitifully stubborn.


    Embedded Image

    Giovanni Boccaccio. From the Villa Carducci series (c 1450) by Andrea del Castagno, held at the Uffizi, Florence, Italy

    Credit: UFFIZI, FLORENCE/BRIDGEMAN ART LIBRARY

    Then an even more powerful Professor called. The organising committee was unanimous. Influential journalists were expected; discussing medical errors was crazy in front of journalists—or even in front of physicians to be honest. More importantly, the school hoped to strengthen ties with the industry; side effects were a nuisance. Here was the deal. Guiscardo had also worked on infectious diseases; he could talk on “Introducing new antibiotics in the community” or anything on new drugs of his liking, a prime opportunity to show off. Guiscardo apologised for being unable to contribute to the conference.

    The meeting was a great success. Many journalists and drug reps attended, and even physicians were present. As for Guiscardo, he never again received a similar invitation and remained at best a totally obscure and controversial figure among his local peers.

    Eighth tale

    Having trashed Tedaldo, who is pleading for a sound study design, the sponsor of a study shows his open mind and inquisitive spirit in considering plans for an alternative analysis

    Unavoidably, research needs money and generates money. Tedaldo was involved in a meeting to discuss the possibility of using public money to fund a clinical study that would use a trick to ensure that it would get the politically correct result. Tedaldo disagreed about the plan, but he could only become annoying to the group. He then suggested a more appropriate approach to the study design and analysis.

    The sponsor insisted on the original plan, which would be more likely to yield the desirable conclusion. However, he was willing to consider also the analysis that Tedaldo was proposing, if the need were to arise unexpectedly. After all, he admitted after the study had been funded, sometimes one gets better results with methods that originally don't seem so friendly to the marketing needs. One needs to be open minded, and research always rewards the inquisitive spirit.

    Ninth tale

    Despite the annoying intervention of young Landolfo, the members of the board of directors of a prestigious agency tactfully and transparently award a juicy grant to themselves

    Landolfo was a board member in another public health government agency. Weirdly enough, he was young, unknown to the media, politically unconnected, and even had a research record. These major deficiencies clearly insulted the lofty non-meritocratic tradition of the agency. The presented proposal was one page long, showing the requested funding (a long number indeed), and a list of 20 investigators to be funded. These 20 investigators included only members of the board itself and their close buddies and academic subordinates.

    Carlo, a powerful lobbyist, was not expected at the board's meeting, but unfortunately he showed up. Paganino, the proposing investigator, prudently and quickly added in pencil the name of Carlo to the proposal. He then apologised to this “highly respected scientist” for this “unfortunate omission” that was being properly corrected on the spot. The correction made, everybody praised this “extremely well developed, worthy project of great national importance.”

    However, Landolfo argued that it is unfair to spend fortunes without any competition. He also requested, as a minimum, to see a protocol for the proposal. Paganino apologised that Landolfo was missing from the funding list but offered to correct this second “unfortunate omission” immediately, pencil at hand. Landolfo declined the offer and asked for a vote. The vote naturally turned out 8 to 1, with the money awarded without any request to see any protocol ever. Shortly thereafter, ungrateful Landolfo was deservedly relieved of his duties.

    Tenth tale

    Saladin, a superb academic leader, reflects on his successful career and on the difficult paths that lead laborious scientists to excellence; then he proceeds with his daily agenda to save the world

    It all boils down to running the (academic) show. That morning Saladin reflected on his very successful career and how he had nurtured the leadership of tomorrow. While other academic leaders had made professors among sons, nephews, and pet dogs, Saladin just gave away PhDs to his protégés, poor kids rewarded for fetching and carrying diligently for him.

    On one occasion, Gabriotto, an examiner, realised that the PhD candidate had performed practically no analyses at all. “Why, don't you know?” he was told, “This is ultra-modern research that overcomes the need for any analysis.” Gabriotto apologised for his ignorance. But then, everybody knew that the candidate had done nothing until two weeks ago—yet here was a completed thesis. Where did this come from? Gabriotto asked for the data records. “Records?” Saladin smiled, “Let's have a beer, my friend. Trust me, we have a big heart, who needs records?” Gabriotto also realised that the one piece of any worth in the thesis had actually been defended recently by another fellow. Whose work was it? “Take it easy, my friend,” Saladin replied, “We are all one big family.”

    Summary points

    We apologise that our attempt to describe variants of poor research is clearly a mess, noble ladies and gentlemen, but we cannot help it

    Five new authors appear on the day a manuscript is to be submitted

    A non-randomised study miraculously becomes randomised after its completion

    A young lady laments her onerous love affair with evidence based medicine

    Constant faith in bad research is rewarded

    Heroic investigators overcome all statistical obstacles to meet a submission deadline

    A statistician with a pure heart fails to massage appropriately his mistress (the data) and is dismissed

    A young researcher fails to become prime knight-defender of the medical-industrial complex and descends into gloomy anonymity

    A study sponsor proves his open and scientifically inquisitive mind in seeking ways to sell a product

    The pious members of a most honourable board of a governmental agency grant all the funding to themselves

    A valiant academic leader reflects on the difficult paths of academia and renews his uncompromising faith in research, provided it sells for a good price

    Gabriotto didn't feel like family. Yet, a touch of gentle family pressure (simply Gabriotto's private and public annihilation, no more than that) helped approve the thesis. Saladin regretted that Gabriotto could not understand that fetching and carrying for your Master makes for excellent scientific training. His skilled protégés would always get well paid jobs in circles where the virtue of servility is still highly valued.

    Other academic leaders had relied on religious faith. In the “martyrdom tenure track,” when Saladin was young, a professorship cost roughly 20 000 bent knees. The aspiring young faculty had an excellent chance of promotion after making 40 bent knees per mass in 500 masses, making sure he is always in front of the eyes of the pious members of the promotion committee. Yet faith has deteriorated since, so current junior faculty members should not dare think of promotion without a minimum of 100 000 bent knees to their superiors. Still others had exploited their solid devotion to specific political parties, nightclubs, football teams, or combinations thereof.

    Pitiful old style. Saladin could adjust his beliefs as requested. He befriended everyman, to undermine everybody blocking his way. His power was money and dexterity to avoid competition unless he knew the winner before the call for proposals. Only idiots apply for grants with 10% acceptance rate when funds are a telephone call away. Telephones beat super accelerators as research tools. Well, enough of that reverie, time to dial the first number for the day.

    Let us leave this academic leader do his serious business: the edification, progress, and overall advancement of mankind rest on his touch pad. And let us end these sad stories here, noble ladies and gentlemen. Forgive our insolence, “if in reading any of these tales you find any pleasure.”

    Footnotes

    • Contributors Both authors contributed tales and worked on the final text. VWB made the first contact, and JPAI had the idea of paying tribute to the great early Renaissance writer and is responsible for formulating the style in this miserable piece. We also sadly confess that the first meeting of the two plotting investigators did not happen in Santa Maria Novella in Florence or at some nice estate near the most beautiful city in the world (as we get no funding from the industry, so we could not afford this), but at a family restaurant at Rockville, MD, close to the NIH campus. The rest of the project was performed electronically from a distance. The quotations in the first and last sentences of this piece copy the original Decameron of Giovanni Boccaccio verbatim.

    • Competing interests None declared.

    View Abstract