Is there a reproducibility crisis in science? - Matt Anticole
Vložit
- čas přidán 4. 12. 2016
- View full lesson: ed.ted.com/lessons/is-there-a-...
Published scientific studies can motivate research, inspire products, and inform policy. However, recent studies that examined dozens of published pharmaceutical papers managed to replicate the results of less than 25% of them - and similar results have been found in other scientific disciplines. How do we combat this crisis of scientific irreproducibility? Matt Anticole investigates.
Lesson by Matt Anticole, animation by Brett Underhill.
Psychology is the first thing that came to mind. Alot of experiments can't be reproduced, but students are still learning about them in schools like they're valid.
Fucking Freud.
theguyshadows I know right.
EveryDayIsAGoodDay I'm a psychology student myself man! Every damn experiment we learn is made by some old guy, the experiment is never reproduced and bam we take it as it is. To be honest psychology is not critical at all. Not sure about the other science fields though, maybe you're right, but I'm speaking from experience.
Freud didn't do experiments. He just made theories and because his theories were so important to the development of the field of psychology as a whole, we still learn about him. Nobody pretends that his theories have any scientific validity, just that they have historical importance I guess. I think it makes sense to learn about Freud himself and how he had an influence on the history of psychology, but it's such a waste of time to dedicate so much of psychology classes to learning about the details of his theories. Same with Erik Erikson and Carl Jung, they all just sat there and came up with theories.
***** I would love to see the scientific evidence that supports Freud's theories and why this evidence compels you.
2:11
If a study cannot be reproduced because it is unclear what the original study was or what the original study group did (as suggested in the video) then the original study should not be considered scientific.
More like, the study can not be considered entirely valid, until reproduced/verified. If it's of any interest and hasn't been verified yet, then whoever is interested in making use of it should be responsible for reproducing results. There isn't and never was any other way.
Klin-Klin No the point is reproducing results is impossible because it is unclear how the original experiment was undertaken.
When they say it's unclear what the original study did, that usually happens with the very specific details. As an example, a published paper could describe that researchers brought group 1 into the lab every morning and fed them 40g of raw broccoli and repeated this every day for 14 days. But we don't know what time of day this was, whether the participants were told not to eat before or after, whether they drank water with it, and other details that seem irrelevant but may effect the reproducibility of that study.
+monkiram Thats why its very annoying to me when I talking about studies to other people. There is many people that takes the study who not gives any those mini important details and people holds them as a solid fact.
agreed
universities should make courses for first and second years students to replicate studies.
this way students can have hand on experience in doing research and those research would be verified.
Sami Mas oh yeah this is already happening - I've spent sooo many hours in the lab and for many other degrees it's a daily thing! Although, a lot of times at university level it tends be heavily based on the previous work of current professors and I'm not entirely convinced they actually take our data that seriously
potatoes
kenichi 2233 mashed or roasted?
Roasted
kenichi 2233 i can have some too?
"If you thought science was certain, well, that was just an error on your part." ~Richard Feynman, scientist, Nobel Prize winner
The media and the educational system always makes it look like it's certain. I never knew richard feynman even said that. It isn't that popular of an opinion
But this shouldn't be an excuse to defend soft sciences and diss hard sciences. Thanks to hard sciences, we are able to put man on the moon.
@@cagribaba4464 And thanks to hard science, we are now in an irreversible course towards climate change. Your point?
Conclusion: Take EVERYTHING we read with a grain of salt.
And truth will always find a way, sooner or later.
That's too broad. I'd say check the sources credibility and research the issue before you come to a conclusion, otherwise you're inferring that all information is equally valid which isn't true.
I think it's "a pinch of salt"
Micah Miranda I am gonna take that with a grain of salt if you don't mind 😋
even the most trusted sources can go a bit overboard sometimes as the vid suggest
Can't emphasize enough how well done andto the point this video is.
And it's true - it really comes down to the fact that there's no motivation in the scientific community to cross-verify anything. As said, it's actually a detriment to one's career to do that, as it's a delay to career progress. Conferencies and journals are simply not interested to publish validation studies.
0:39 Idk why but the facepalm cracked me up😂
Potato Shadow It was perfectly timed. Props to the animator. 😃
It is very relatable, isn't it?
me to. it was hilarious!!! i watched that part like 5 times...:p
anyway, good thoughts in this video...some things to think about...
This is outstanding and very important. Thank you for doing it. Unfortunately, publishing negative results, if you can even do it, is not likely to lead to grant funding. Without funding, you have no science. Thus, negative studies or confirmatory studies to confirm reproducibility will continue to be underemphasized (and typically just lacking) in our current model of science and research
I honestly think we should find ways for grant funding to be decoupled from the research at hand. Maybe publishing methods and having those methods approved by peer institutions would be a good way to get funding.
Perhaps each time new research gets funding several other institutions get funding for their reproducibility trials automatically.
@@danielsykes7558 You are very sweet but apparently naive. Who is it that provides the funding? The corporations and pharmaceutical companies. Why would they incentivise / fund study that will cause their newest product to be NOT be produced and sold?
@@EtreTocsin he is not naive at all - it’s a case of regulation and law.
@@EtreTocsinit would literally have to be the government. It exists to brute force people into doing things for the public good when there is no personal incentive.
@@EtreTocsinfree market can’t solve the worlds problems
Easy solution: Get Honours students, Masters students or students in their late studies to prove or disprove recently published papers as one of their assignments... and then make Mexico pay for it.
We were actually assigned to do this by writing a Critique on a peer-reviewed published article as part of our Academic Writing course.
The thing is, it's hard to "prove" or "disprove" anything in research papers because scholar papers themselves aren't trying to "prove" anything. They only suggest that their hypothesis or "theory" is *probably* correct by showing a high probability (using statistical methods)
No paper will include a full list of their collected data/calculations; instead they conclude with their calculated results. You can only do so much with these results, and as a reader will have to assume that the researchers aren't lying/didn't make a mistake.
2:33 anyone else think of a trump reference
+Z zz lol, Ur joking right, right?
It was actually part of my first year's assignment in Psychology to reproduce the result a famous study that was shown to accept the alternative hypothesis :)
Making it an "assignment" as if they were some high school students would just have them go through the motions. Why waste such time and effort on something you're not passionate about?
0:56 that guy went on to create the powerpuff girls
I thought the same thing
Thank you for not focusing solely on psychology. This is a problem for all emperical research.
Thankfully math related subjects are not affected
Comp Sci master race
@@tannergordon8302 I keep seeing adverts that misrepresent sociological ideas as computational ones. There's a whole industry definable solely as often claiming - without explaining why - "building computers that function as advertised has ceased to be feasible". Its "work" advances outward from that point. It's a whole industry. It's called "cyber-security" like 1984. Its the activity definable as "applying the belief that it doesn't to exist". CompSci bods are mostly going to have a really surreal thousand years navigating the treacheries and retardations of that "industry".
Psychology doesn't even qualify to be part of the problem.
@@wliaputs Actually, they are. Some errors in proofs are ridiculously hard to spot. There's definitely more than one false paper out there that nevertheless managed to pass peer review.
Team work is important
It helps to put the blame on someone else :
agree?
I had seniors in lab who wouldn't publish exact measurements of their data so no one could replicate it, in case they needed to sell it for patent.
The fame, funding, and career maintenance motives certainly bias the way work gets done and produce problems. I'm starting to wonder if we'll look back at today like we do the middle ages. The church used to fund most research. Nowadays it seems to be corporations that fund most of it.
Honestly, science should be curious people coming together, and no one should have their livelihood based on certain results. This requires that people everywhere have more freetime and that we have strong peer review institutions.
We need perhaps more funding to duplicate research rather than to do original research. And that funding should be independent of corporate interests.
Good topic, great points brought up... but needs TONS MORE DETAIL!
For instance, on the subtopic of improving reproducability alone: How can we incentivize more raw data availability? Why isn't it done already? Why don't papers detail their techniques sufficiently already? Why why why why why!?!? DETAILS PLEASE!!!!!
Because it's a corrupt system that scientists don't want to meddle with. Everyone knows what's going on, scientist have the cheese and the knive to solve the issue. But they won't and you won't hear many details anywhere.
Because they don't want their idea stolen and someone else claiming credit for it. And because if general public can replicate and use it on their own then the one with the initial discovery won't make bank. Greed.
Higgins2001 They do publish papers with very descriptive techniques. Not that many read their papers, besides others in their field. Thus the manner of speaking is more precise yet somewhat confusing to the untrained.
+Higgins2001 The full lesson can be found in the description.
If they give all the incite of their ideas, how do they make money from it, and whats the incentive for others to chase other discoveries. Many scientist love their work but money makes shit happen.
“I think that we shall have to get accustomed to the idea that we must not look upon science as a ‘body of knowledge,’ but rather as a system of hypotheses, or as a system of guesses or anticipations that in principle cannot be justified, but with which we work as long as they stand up to tests, and of which we are never justified in saying that we know they are ‘true’ or ‘more or less certain’ or even ‘probable.’ -Karl Popper
Looks like the Powerpuff Girls got a new plant.
Powerpuff Girls vs Chomper...
If all "government funded" research was forced into the public domain and researchers couldn't patent said research, a lot of these problems would go away.
Great animation. Would watch again just for that.
Hi Ted ed, I hope you see this. I wanted to suggest that you cite the resources you use in the description box as many of your viewers are students probably researching for a paper or something. It would be helpful to get the resources so that we can cite them too. Thank you!
loved the concept on which is the video of Based, very nice video
Love the animation!
I remember verlisium did a video on this. worth a watch for more detail.
Sophie Matthews yup
Veritasium, yeah.
Juan Pablo Mina
yeah couldnt remember the name lol
link??
+Sophie Matthews Veristablium
Honestly, the best thing to learn is how to remove or even just heavily suppress your personal biases(Theological, Political, Racial, ect,..) when conducting Scientific research of any kind.
that would be very ideal, but in reality, I don't think that would be possible. I'm no physiology major, but hat's just my opinion
When conducting research? Thats what review boards, blinding, and meta analyses are for.
It's not really a significant problem is science so i don't see why you would say it's the "best thing to learn"
Personal bias is bad enough, but there's also money being dangled out there by organizations who _only_ want their biases validated.
@@crimsonmask3819 yes. Also, this comment reminds me of the quote from the Three-Body Problem: "Should philosophy guide experiments, or should experiments guide philosophy?"
I find some Irony in a video talking about potential dangers of non-replicated scientific studies...that uses scientific studies to back up its argument like at 3:25. I don't know what they would use instead but I still find it a little funny.
we all need to show more love and compassion for our human counterparts
If you're curious, the Italian lab is under the Gran Sasso mountain in Abruzzo, Italy. I live a hundred kilometers from that lab.
1:08 is that samurai jack
Mystery Man No, it's Professor Utonium!!
TJ Simmons they are the same person . You know the devil in the power puff girls ? He is the reincarnation of haku in samurai jack
Very important.
beautiful video
Psychology gets a bad wrap because it's more of a heuristic. The brain is very complex and hard to understand. When a psychologist seeks to understand human behavior it's like trying to figure out the inner workings of a complex math equation knowing only a few inputs and an output. They do good work though so take it easy on them, they have a very difficult job.
Unfortunately, I find that many that go into that field is underqualified to begin with. Those that really succeed (the 1%ers) will most likely succeed in other areas of study, but can't say the same with the others (99%ers).
This is a biased opinion though so lol
Anyone else notice the Little Shop of Horrors reference with the carnivorous plants on the White House?
This is so fun to watch.
This is outstanding and very important. Thank you for
doing it. Unfortunately, publishing negative results, if
you can even do it, is not likely to lead to grant
funding. Without funding, you have no science. Thus,
negative studies or confirmatory studies to confirm
reproducibility will continue to be underemphasized
(and typically just lacking) in our current model of
science and research
I always wondered if there should be a Federal
Beaural of Scientific Integrity which would go through
and look at all published papers, in the USA at least,
and help flag any that are outstandingly false by
replication if possible or b review.
Team work is important
It helps to put the blame on someone else:
agree?
Whenever there is a need for certain results, the science will usually go out the window. The answer in our predicament is to always look for several replication studies of any previous conclusion. Most people don't care enough and most youtube videos don't include this either
And i love that folks never doubt "science" yet theres plenty wrong.
Have a better method?
@@ExatedWarrioryes, the scientific method, which is better than blindly trusting everything that gets "science" stamped on it
Question everything
Very interesting. And the animations are funny too.
hey TED-Ed i hope you read your comments xd
can you make a video on the invention of calendar dates/months
While reviewing a paper, I was literally told by a senior scientist: "Don't correct that much and don't be so strict. Otherwise our own papers will get a lot closer look and more criticism". Combine that with the fact that our institute (part of one of the largest scientific societies in Europe and in the world) has a papers per person quota and you get the point...
Funny thing is, chasing KPIs in the real world (companies) has been proven to be really bad. I wonder when the academic society will realize that. Then again, many people there (especially the old farts) never worked anything practical and remotely useful in their entire life. Their whole existence will be put in jeopardy. We can't have that!
"Real science is more than just static textbooks"
Tell that to every science teacher in elementary and middle school
Exactly! Even my high school chemistry teacher was so dogmatic about methods and style. I remember when she taught us how to calculate Formal Charge and Lewis dot structure problems, she just messed up and then ended the class saying do 4-5 problems from the text you'll understand. 😆
That little shop of horrors reference though.
I always wondered if there should be a Federal Beaural of Scientific Integrity which would go through and look at all published papers, in the USA at least, and help flag any that are outstandingly false by replication if possible or by review.
Collin Bruce Unfortunately that is subjected to bias by who is checking. Ex. Human induced climate change, people tend to look for biased research inline with their views/agenda as some sort of proof.
you really don't wanna do that, You can corrupt people easily, especially when you give them power to decide which scientist is getting funding that year.
See this is criticism I need to hear.
2:50
I understood that Popeye reference!
My chem teacher told me neutrinos can travel faster than light smh
that little shop reference though...
well on the bright side I've seen videos on this subject before :-)
Nice Little Shop of Horrors reference :D
That bee made me drop my phone and swat around my head XD
It's a bit stunning how this video glosses over how severe this issue is. Whatever the motivations, this calls to question the entire body of modern science, all that is based on prior papers. In law, there's the concept of the fruit of the poisonous tree. This is far worse because nobody has been minding the store for a great many years, while we've been assured it's all super sound.
3:56 legit thought my iphone alarm went off
Like the Little Shop reference...
Flourishing always follows after crises
Lerning iz kool
1:59 Genius.
Any sort of medical study is just in it's own separate paradigm of accuracy from the rest of science. There are just far too many unknown factors, as well as the giant fatal flaw of relying heavily on people's testimonials about their condition and past history.
remind me of certain video from veritasium
Ok, but was the study on how often mistakes are caught reproduced?
except Flossing. we never check that
TED ed rocks!
yeahh.. John oliver from Last Week Tonight had already talked about it..
dat face palm, though
0.35 i know that feel
Bless Audrey2 👌🌿
Little Shop of Horrors reference..nice.
Really interesting: Everyone can make errors... But we shouldn't discourage ourselves because of that! Thomas Edison once said: "I didn't fail, I just found 10000 wrong solutions"
and then he nicked all of Tesla's ideas and made them his own ...
@@lakestreet3951 true
Math: I don't have such weaknesses
Maths has the weakness of human error and understandability, just like anything else.
Where did ted go
didn't know that about spinach!! Go to supermarket and buy some!! lol
Does anyone know the reference/source of the fact about less than 25% of pharmaceutical studies are not reproducible?
at 1:02 I spot an Audrey II!
With the study done in 1998, you have to consider that only like 3 years later, Jan Hendricks Schön’s fabrication of most of his papers was discovered which forced the peer review processed to be reviewed itself. It made the process significantly more rigorous but also more accurate.
We, as a society, need to get out of the arrogant mindset that we have it all figured out. We continue to make incredible advancements in science and technology, but when considering things like this, we still have a small window of confident knowledge about the world. Long story short, everyone from our scientists, to our politicians, to our common readers and youtube-watchers needs to stay teachable and be ready to admit when we are wrong. Let's write our beliefs "in pencil", not in stone.
did anyone else thought about the EM drive when watching this ?
1:59 is just cruel to headphone users, haha. I paused the video to kill that fly before I realized it wasn't real...
this is why we need to start teaching newer and latest textbooks than the 10 -20 year old science textbooks in high school
This would seem to be a reason not to rushing the results of experiments and put them into text books. Really, there is very little from the past 50 or even 80 years that need to be in a high school text book. I am reasonable sure that "the lack up to date text books" is the reason why US high school science is crap.
Should start teaching ancient knowledge since they weren't dumb enough to believe people stuck to the bottom of a sphere are somehow not standing upside down.
maham meher all your doing is feeding the textbook industry. They do revisions but they're either reshuffling, adding more pictures, lastly adding new content worth a few pages. Don't believe me? Try buying a college textbook a 3rd edition isn't that different from a 5th edition..
The wife and I make our own experimentals back in home.
We found exciting results :)
Thats cool 0u0
I actually thought there was a fly near my ears I closed my ears in 1:58
idea from ve?
Very common in Machine Learning publications today lol
even ted can't dodge teh trump memes dis gonna be huuuuuuuggeee
I thought the issue with neutrinos was that a squirrel ate one of the cables and got electrocuted in the process?
Does anyone know where I can find newly published science papers??
Do you have access to any journals, like through college? If so, Web of Knowledge should work, if not, .... well, Sciencehub is love, Sciencehub is life
Ohhhh thx
arxiv
+Technition
If you can't find them, you'll certainly not be able to understand them.
Frank Schneider that is straight up BS
AKA a rehashing of John Ioannidis 2005 essay.
So isn't the reproducibility issue, in part, cancelled out by the fact that if new studies are build upon incorrect older studies, eventually the results don't add up and you can track it back to what may have been false? Sure it's a waste of time, money and other resources but it does prevent false information from being acted upon in the long run (?)
was this video made cuz' of the EmDrive ? (sorry for my bad Englisch ) *if it's Bad
so, the takeaway is that we should encourage reproducibility, not that we should disregard science as a useful tool for learning new things.
"Exciting new discoveries"
Just got chills in ma body 🤠😉💜✨
Now why ya gotta play bee sounds in stereo?
0:55 is that Professor Utonium?
Proffessor was pulled right out of powerpuffGirls.
1:58
Literally I'm so stupid. I smacked my headphones off because I thought there was a mosquito near me lmao
***** Especially when it's night and there's no light to see it.
2:50 Popeye the sailor man...
...And this is why I like maths. Prove your theorem in maths, and it generally stays proved. Although statisticians do like to wobble things :)
In theory, yes. But from time to time there are errors in proofs that are ridiculously hard to spot. There's definitely more than one false paper out there that nevertheless managed to pass peer review.
How does mathematics ever "prove" a scientific theorem (other than a math theorem)? Math can be used to describe observed data or behaviors, but I don't see how it ever proves a theorem.
No replications published, huh? The greatest physical law is, I propose, the second law of (classical) thermodynamics, expressed as " it is not possible to construct a system that exchanges heat with a single reservoir and delivers net work". The classical second law depends exactly upon reproductions of experiments in which the failure to produce net work is confirmed.
Now, in number theory, Colatz' conjecture is considered to have been proven because billions of sequential calculations from the initial number (index) have all led to 1, which is what Colatz conjectured. Clausius ' mode of thought lives!
Charles A Berg
Then how can we be sure to trust the scientific literature?
Sadly, most literature these days says whatever the source of funding wants it to say and not what real science dictates.
Look at the money. Don't trust a pro-meat paper funded by McD. Also don't trust a paper funded by Beyond eat.
The tricky part is that they usually hide behind a proxy organization, so it's not that easy to spot.
for science as in (physics, math, chemistry and biology) there is no such thing as risk of losing time, as physicist and mathematicians who created concept of time and is not even subjected to the change of anyone's liking and just say they are not valid without even studying them. getting weird results is expected as scientist keep their curiosity and ambitions in understanding the world while these people who say science is not valid are just people who may just be jealous of the level of smartness and criticise science without even understanding them. this contradictory behaviour is just toxic and non-productive whereas those who say maths is not useful and inaccurate are just being ridiculous.
Unfortunately it is quite common in science for results to be non reproducible. As a biological scientist myself I have seen several cases of that. Interestingly it depends on the publishing scientist and the group behind that scientist. For example pretty much all of the results of Canadian researchers I have been working on were plausible and reproducible to some extent. On the other hand I did work on a few cell biology projects based on an Austrian researcher Manuela Baccarini from the Max Perutz Labs in Vienna. Her work was a classic example of non plausible research and likely fraud.
So it depends on the field of research and the underlying papers.
Woah! I've seen this too. Thanks to PubPeer, some karma is being dished :)
Did anyone else freak out when they heard the sound of the fly buzzing? It makes my skin crawl 😷😖
As part of the collective acquisition of new grants more likely.
Hey!