Over the past few years, an international team of almost 200 psychologists has been trying to repeat a set of previously published experiments from its field, to see if it can get the same results. Despite its best efforts, the project, called Many Labs 2, has only succeeded in 14 out of 28 cases. Six years ago, that might have been shocking. Now it comes as expected (if still somewhat...
The inside story of how an Ivy League food scientist turned shoddy data into viral studies
But for years, Wansink’s inbox has been filled with chatter that, according to independent statisticians, is blatant p-hacking. “Pattern doesn’t look good,” Payne of New Mexico State wrote to Wansink and David Just, another Cornell professor, in April 2009, after what Payne called a “marathon” data-crunching session for an experiment about eating and TV-watching. “I also ran — i am not kidding —...
Sue the journal if it publishes an article critical of your research
Stanford University professor Mark Z. Jacobson has filed a lawsuit, demanding $10 million in damages, against the peer-reviewed scientific journal Proceedings of the National Academy of Sciences (PNAS) and a group of eminent scientists (Clack et al.) for their study showing that Jacobson made improper assumptions in order to claim that he had demonstrated U.S. energy could be provided exclusively...
Kahneman: “I placed too much faith in underpowered studies”
Kahneman responded: The argument is inescapable: Studies that are underpowered for the detection of plausible effects must occasionally return non-significant results even when the research hypothesis is true – the absence of these results is evidence that something is amiss in the published record. Furthermore, the existence of a substantial file-drawer effect undermines the two main tools that...
Is it easy to get data from another scholar?
I liked this very much. The sad thing is that it was obviously created by a medical researcher, not an educational researcher.
Psychology is broken
Now, a painstaking yearslong effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday in the journal Science, have confirmed...
Science isn’t broken
If you follow the headlines, your confidence in science may have taken a hit lately. Peer review? More like self-review. An investigation in November uncovered a scam in which researchers were rubber-stamping their own work, circumventing peer review at five high-profile publishers. Scientific journals? Not exactly a badge of legitimacy, given that the International Journal of Advanced Computer...
A scientific look at bad science
This is why replication is so vital, wouldn’t you agree Paul? A study in the Proceedings of the National Academy of Sciences reviewed 2,047 retractions of biomedical and life-sciences articles and found that just 21.3 percent stemmed from straightforward error, while 67.4 percent resulted from misconduct, including fraud or suspected fraud (43.4 percent) and plagiarism (9.8 percent)...
When researchers state goals for clinical trials in advance, success rates plunge
Around 2000, the U.S. government ordered researchers conducting clinical trials with federal money to announce ahead of time which medical question they were hoping to answer. Before then, 57 percent of large-budget trials for cardiovascular disease attributed a positive effect to a drug or dietary supplement, according to a study published on Wednesday. After the new requirement, the success...
Scientists are hoarding data and it’s ruining medical research
Paul Attewell, are you paying attention? This “deworm everybody” approach has been driven by a single, hugely influential trial published in 2004 by two economists, Edward Miguel and Michael Kremer. This trial, done in Kenya, found that deworming whole schools improved children’s health, school performance, and school attendance. What’s more, these benefits apparently extended to children in...