Is Most Published Research Wrong?
Vložit
- čas přidán 10. 08. 2016
- Mounting evidence suggests a lot of published research is false.
Check out Audible: bit.ly/AudibleVe
Support Veritasium on Patreon: bit.ly/VePatreon
Patreon supporters:
Bryan Baker, Donal Botkin, Tony Fadell, Jason Buster, Saeed Alghamdi
More information on this topic: wke.lt/w/s/z0wmO
The Preregistration Challenge: cos.io/prereg/
Resources used in the making of this video:
Why Most Published Research Findings Are False:
journals.plos.org/plosmedicine...
Trouble at the Lab:
www.economist.com/news/briefin...
Science isn't broken:
fivethirtyeight.com/features/s...
Visual effects by Gustavo Rosa
As the famous statistical saying goes, "If you torture data long enough, it will confess to anything"
Science is GAY!!!!!
@@jamesmonroe3043 get a life
@@alexanderabrashev1366 You must be Pro S0d0m33.
@@alexanderabrashev1366 you but hut
@@mayrokratt6195 did you have an aneurysm writing that or what?
The most shocking thing to me in this video was the fact that some journals would blindly refuse replication studies.
Maybe they're system is flawed?
The Economist publishes blindly. The entire EVidence Based Medicine does not yield dosage for medicine tested - that includes your painkillers, your blood pressure pills, your mood stabilisers and your anesthesia.
But this is not science's fault, nor is it a conspiracy of the medical system.
The scientific method is limited in its ability to give us data.
Yet, practically we need to understand the world. The only way out is innovative methods that people can actually understand their pros and cons and get published without trying to immitate the scientific methods to be listened to.
Oh, and developing the right hemisphere science. Which is what I experiment with.
We should boycott that unscrupulous journal. Does anyone know the name?
@@spiritofmatter1881 No, the scientific method can be skewed to provide misleading data; and if you don't believe there's a conspiracy in the medical/pharmaceutical system, there's a bridge in NY I can sell you for a good price. The pharmaceutical industry is mostly shady -- and medical doctors are their extended hand of national snake oil salesmen.
Journals are just newspaper for scientists.
Publication house want to publish things that people want to read. And people want to read interesting and new things, not replication studies.
Even though in modern times there are journals that accept and publish replication studies. These journals usually incur a much larger publication fee.
Nothing is perfect. But at least, everyone is trying
As a very wise man once stated, "It's not the figures lyin'. It's the liars figurin'". Very true.
Samuel Clemens
not alway it has to be intentional lie
Also true
I''m loving the quotes in these top comments!
😂😂😂
this happens because of "publish or perish" mentality. I hate writing scientific papers because it is too much of a hassle. I love the clinic work and reading those papers, not writing them. in this day and age it is almost an obligation that EVERYBODY HAS TO PUBLISH. if you force everyone to write manuscripts, flood of trash is inevitable. only certain people who are motivated should do these kind of work, it should not be forced upon everyone.
Indeed. You are totally right. I also feel that this over-publishing phenomenon has led these researchers into manipulating the data because it feels like a competition. In my opinion, researching is hard and sometimes frustrating, but you should always stay loyal to the fact. Otherwise, you are cheating for profit. Totally unethical. There are people who devoted their lives for this but still, they don't get as much recognition as these 'cheaters' do.
And not only is it "publish or perish," but you also have to pay the journals to publish your work once it is approved for publication.
@@migenpeposhi6881 Certain people don't realise that "Research" doesn't mean "Prove something is true" but instead "See _if_ something is true"
@karlrovey I choose perish
I'll tell you exactly why. The academia is controlled by rabid looney slavemasons and rabid looney jesus. Since they are both mentally and intellectually retarded rendering them incapable of any intellectual manifestation, they force grad students to do it in the publish or perish way. They actually steal and collect it for their own records because of the following reasons: 1. They think that these are valuable intellectual property which should only belong to them. Hoarding basically 2. They will pass it off as their own intellectual property subsequently in a post apocalyptic era or a different realm 3. Being intellectually retarded, they prefer quantity over quality. More the better 4. They are basically mining your intellectual property by exploiting you for their use 5. Commercial reasons just in case your papers somehow fetch money
This is actually good. You should publish more crap for the slavemasons to prevent then from getting hold of some actual research
For people freaking out in the comments: we don't need to change the scientific method, we need to change the publication strategies that incentive scientific behavior.
You *believe* that we don't need to change the method
Adamast I did not for once stated my believe. I stated the point of the video, which many seem to have missed. Ironically, you now missed the point of my comment.
+Adamast no we dont need to change the method, the problem (which was explained VERY clearly in the video) is that people arent USING THE METHOD PROPERLY. like when someone crashes a car cause they were drunk, the car isnt broken its just being used incorrectly
You do state your believes. You think you know the point of the video, where you see other people just freak out. Personally I read: _Mounting evidence suggests a lot of published research is false._ Nothing more. There is I admit a short faith message at the end of the video.
The scientific method- you go out and observe things, and develop a hypothesis, and test the hypothesis. If you run a bunch of tests and come out with the wrong deductions that is called flawed research methodology. Flawed research doesn't imply the concept of going out and observing coupling with experimentation is flawed, it just means you suck at being a scientist.
I really think undergrads should be replicating constantly. They dont need to publish or perish, step-by-step replication is great for learning, and any disproving by an undergrad can be rewarded (honors, graduate school admissions, etc) more easily than publication incentives can change
Agreed. The peer review portion of the scientific method is it's weakest link IMO.
I agree. Undergrads replicating classic experiments can also help with their education.
Undergrads do preform classic experiments ^
When I was an undergrad in the physics department, we were (and still are) required to reproduce quite a few experiments that led to Nobel prizes. It's a fairly common practice, and you do this while working in a research group that is producing new research.
@@sYnSilentStorm How common are undergrads in research groups / replicating newer works? Well trodden ground breaking-now-foundational Nobel prize works make sense for learning, but we would benefit from a more institutionalized path of replication of newer claims. Especially if we can before graduate school, which is who populates research groups in my mind.
When I was in grad school for applied psychology , my supervising professor wrote the discussion section of a paper before the data was all gathered. He told me to do whatever I needed to do in order to get those results. The paper was delivered at the Midwestern Psychology Conference. I left grad school, stressed to the max by overwork and conscience.
What was the thesis? Asking to learn if topic was especially controversial or important to particular interests.
Why are you the minority? How can people who have the power to create Utopia choose self interest. It's like I'm in the Twilight Zone. Everybody knows and nothing is done. I wish I was never born and I hope I never am again.
Why not mention the subject of the paper then, brother? You lying…?
chances are high that the paper wasn't going to reveal anything new
@@ButtersCCookie ??? nihilist looking goober
Sabine Hossenfelder: "Most science websites just repeat press releases. The press releases are written by people who get paid to make their institution look good, and who for the most part don't understand the content of the paper. They're usually informed by the authors of the paper, but the authors have an interest in making their institution happy. The result is that almost all science headlines vastly exaggerate the novelty and relevance of the research they report on."
"There is no cost to getting things wrong, the cost is not getting them published"
It's a shame this also applies to news media as well.
books are media...
@@NaatClark maybe they meant news
@@fernando4959 i did, i fixed it
@@GiRR007 At this point I think it's indisputable that mainstream news is literally state propaganda.
@@NaatClark I don't know why it wouldn't apply to books
As a former grad student, the real issue is the pressure universities put on their professors to publish. When my dad got his PhD, he said being published 5 times in his graduate career was considered top notch. He was practically guaranteed to get a tenure track position. Now I have my Masters and will be published twice. No one would consider giving you a post doc position without being published 5-10 times, and you are unlikely to get a tenure track position without being published 30 or so times. And speaking as a grad student who worked on a couple major projects, it is impossible to be published thirty times in your life and have meaningful data. The modern scientific process takes years. It takes months of proposal writing, followed by months of modeling, followed by months or years of experimentation, followed by months of pouring over massive data sets. To be published thirty times before you get your first tenure track position means your name is on somewhere between 25-28 meaningless papers. You'll be lucky to have one significant one.
Damn. I really want to be a researcher in the natural sciences one day with hopefully a Master's or PhD, but I must say seeing this is a little unnerving. Would you happen to have any advice for aspiring researchers?
I studied in Japan, now Japanese has changed their view in research, not to get to top of the world, but how the reseach could be applied and contributed to society. If you look at the ranking today, most japanese univs are not at the top ranking as they used to be. Now, univs from Korea, HK, China, singapore are climbing for top ranking. But every year these univ have suicide cases.
@@mohdhazwan9578 friend I was totally with you until you mentioned suicide. How is that even relevant in how research can be applied to society?
Not really accurate though. None of my professors have been published 30 times and they've taught at Yale, Texas A&M, Brown, that's not really true at all.
@@ShapedByMusic they might not have, but it is beginning to become a standard for new hires. 30 might be a bit of an exaggeration, but there is no way you are getting hired with under 15-20 for a tenure track position.
The problem is people are suppose to be able to replicate the results by doing the experiment over again. If I can’t find multiple experiments of a study, it’s hard for me to not be skeptical
The big problem with that is noted in this video, replicating work already done generally has no rewards.
The time, money, and need to publish to advance their careers mean even the best intentioned researchers are likely to avoid redoing someone else's study.
Some experiments are very expensive in terms of time and money but We shouldn't worry too much about our money getting into publishing false positives. At the end of the day only the true positives will be the basis for further advancements, experimental science is built on previous results and if those results are spurious, nature will stop you from discovering further real relationships. That's what this video is failing to point out, the incremental nature of scientific knowledge in the natural sciences is a natural peer review system and the best that we can ever have hoped for. So keep funding science, at the end only the true relationships will stand the test of time
As someone who studies theoretical statistics and data science, this really resonates with me. I see students in other science disciplines such as psychology or biology taking a single, compulsory (and quite basic) statistics paper, who are then expected to undertake statistical analysis for all their research, without really knowing what they're doing. Statistics is so important, but can also be extremely deceiving, so to the untrained eye a good p-value = correct hypothesis, when in reality it's important to scrutinise all results. Despite it being so pertinent, statistics education in higher education and research is obviously lacking, but making it a more fundamental part of the scientific method would make research much more reliable and accurate.
I barely passed my statistics class, and I'm in biology. Even now I feared that I might not gonna be able to interpret my data.
Observational studies in the social sciences and health sciences are mostly garbage. People who don't do experiments or RCTs need to study a hell lot of statistics to get things right. And only recently we got natural experiments to help us with good research design for those areas.
Until my masters, I was relatively well trained in traditional statistics (I'm an economist) but unaware of natural experiments. I was completely disheartened about how awful my research was, given that different specifications in my observational studies were giving me different results. I only gained renewed enthusiasm in a much better quality PhD who taught me much better research designs.
@@romanbucharist4708 dang. So how's it going now?
Also happy soon new year guys! 2023 Greetings from Florida
@@romanbucharist4708 Also happy soon new year guys! 2023 Greetings from Florida
My favorite BAD EXPERIMENT is when mainstream news began claiming that OATMEAL gives you CANCER. The study was so poorly constructed that they didn't account for the confounding variable that old people eat oatmeal more often and also tend to have higher incidences of cancer (nodding and slapping my head as I type this).
Maybe don't stand so close to the microwave oven when you cook it.
Perhaps Oatmeal isn't the problem. The problem could be with a change in the way it is grown and produced. Contamination, soil depletion, pesticides, etc. There are always variables that are un-accounted for in scientific research that invalidates the conclusions.
@@kdanagger6894 the true answer was a lot simpler than that.
Don't forget about "vaccines-autism" one
My favorite is probably the story of how it was proven that stomach ulcers are caused by bacteria and not by stress and spicy food. Big arguments for decades between the established science and its supporters vs the scientists discovering the truth. It illustrates how poorly established science treats scientists with new ideas no matter how valid.
Gotta love when a published research article states that most published research findings are false
Research has found that 73.2% of all statistics are made up.
@@cinegraphics Rumor, not research, found that.
It's a complete misunderstanding of how science research works. "Eureka" moments are rare. Instead, the truth is eked out little by little: many rounds of test, falsify, retest, improve until the truth is arrived at.
@@genepozniak Can't tell if you're taking the piss or not........
@@4bidn1 If I gather your meaning correctly, no, I'm deadly serious. Think about it. If published research was the crap he said it is, where is all this successful bio-tech coming from?
This has influenced my thinking more than any other video I have ever seen, literally it's #1. I always wondered how the news could have one "surprising study" result after another, often contradicting one another, and why experts and professionals didn't change their practices in response to recent studies. Now I understand.
Weird how popular this video is recently (most top comments are from ~1yr ago)...
None of these problems apply to the fields of virology or immunology... right?
@@thewolfin I have no idea how much or little different areas of study are affected. I assume the very worst ones are when that ask people what they eat, and ask how healthy they are. Beyond that, no clue.
Yeah, I remember how eggs were bad for your health, then they were good, then they were bad again. Not even sure where the consensus on that is at this point.
Meta analyses are difficult to conduct, but help weed out bad data and contradictory findings. Not enough of these are done.
@@bridaw8557 Some meta analyses weed out bad data, while others average it in. The trick is carefully reviewing how the original studies were done.
“p
Research shows lots of research is actually wrong
_spoopy_
Science can actually falsify science... makes more sense than you might think
Science is a battlefield of ideas, the darwinism of theories, if you will. Only the best ideas will survive. That's why the scientific method is so powerful.
Unfortunately the "best" ideas today are those that will result in profitable gadgets and not exactly those that would best propel human knowledge forward.
+Nal but that isn't science's fault, nor capitalism's. It is the fault of the consumers that value gadgets so highly.
Research shows that most statistics and published research are false, statistically speaking.
It’s almost impossible to publish negative results. This majorly screws with the top tier level of evidence, the meta analysis. Meta analyses can only include information contained in studies that have actually been published. This bias to preferentially publish only the new and positive skews scientific understanding enormously. I’ve been an author on several replication studies that came up negative. Reviewers sometimes went to quite silly lengths to avoid recommending publication. Just last week a paper was rejected because it both 1. Didn’t add anything new to the field, and 2. disagreed with previous research in the area. These two things cannot simultaneously be true.
This is very frustrating to hear.
damn
"These two things cannot simultaneously be true."
Yeah, I just wanted to say it.
If 1. is true, 2. can't be and vica versa.
Don't get me started about meta analyses. I've never heard of one being undertaken except where the backers have some kind of agenda. And I've never heard of one the results of which didn't support that agenda. The entire concept is deeply flawed.
BestAnimeFreak that’s what he said...
This is huge in my field, Structural Engineering, as people get way too lax about sample size. Thanks to testing things like full-sized bridge girders being incredibly expensive, samples sizes of 1-3 have become all too common, and no one does replication studies... Then that mentality bleeds over to things like anchor bolts that can be had for $5 a piece at any big box hardware store. It's getting dangerous out there!
I took a course on rigging moving loads. There I learned Working Load Limit is 10% of Minimum Breaking Strength.
That makes me feel safe.
@@SuperSlayer76 can you explain what you mean by rigging moving loads? Does this refer to loading up delivery vehicles and such or something else? Or is this 10% rule used for many different applications?
This is Crane operator stuff.
There is a certain "strain" that is allowed for fatigue reasons. (Strain is material stretch vs applied stress). Exceeding that strain, while still below the breaking strength, will result in weakening of the material with repeated use causing failure below it's published minimum strength.
E.g. steel might have a tensile strength of 110,000psi but a fatigue strength of only 63,000psi (63ksi/110ksi = only 57.27%).
So, for conservative use, most industries require robust safety factors to account for fatigue, use, damage, etc.
Commercial airliners are rated for +3.0g x 1.5 safety factor at maximum weight.
Bridges vary, depending on seismic requirements, etc. But it's not a good idea to cross an old country road bridge rated for 6 tons, with a 12 ton vehicle. You might survive, but the bridge will be damaged.
@@jj4791 Strain is stretch per original length, like if you stretch 2" in what was originally a 100" rod, you've got 2% strain, or 0.02
Science isn't the initial idea, it's the dozens of people who come along and test the idea afterwards
Agree. We shouldn't worry too much about our money getting into publishing false positives. At the end of the day only the true positives will be the basis for further advancements, experimental science is built on previous results and if those results are spurious, nature will stop you from discovering further real relationships. That's what this video is failing to point out, the incremental nature of scientific knowledge in the natural sciences is a natural peer review system and the best that we can ever have hoped for. So keep funding science, at the end only the true relationships will stand the test of time
An engineer with a masters in nuclear engineering, a mathematician with PhDs in both theoretical and applied mathematics, and a recent graduate with a bachelors in statistics are all applying for a job at a highly classified ballistics laboratory. Having even been given the opportunity to interview for the job meant that each candidate was amply qualified, so the interviewers ask each the simple question, "what's one third plus two thirds?"
The engineer quickly, and quite smugly calls out, "ONE! How did you people get assigned to interview me!?"
The mathematician's eyes get wide, and he takes a page of paper to prove to the interviewers that the answer is both .999... and one without saying a word.
The statistician carefully looks around the room, locks the door, closes the blinds, cups his hands around his mouth, and whispers as quietly as he can, "what do you want it to be?"
That is a good one!
i thought the punchline was going to be 5/9
there is no bachelor in statistics -.-
@@ImperatorMo there is right??
@@ImperatorMo
It's a joke. I can't tell if you're making a joke as well or trying to insert asinine information into a his humorous comment.
When something is published it's proof that someone with the ability to publish things decided to publish that thing.
_...or is it?_
*Vsauce music starts playing*
Also oh gosh why are there so many Vsauce videos in the recommendation bar to might right?
Sounds Marxian.
The human aspect ruins statistics
Or are the founder's £ of said research looking for certain results
It would be great to have a mention in this video of statistical methods that are being used to prevent issues with multiple hypothesis testing. The two most common methods control for the familywise error rate, like the Bonferroni correction, or control for the false discovery rate (aka the expected proportion of false positives), such as the Benjamini-Hochberg method. Reviewers for journals nowadays, at least in my field, are asking scientists who submit manuscripts to account for multiple hypothesis testing using one of these statistical methods.
I wanted to thank you for speaking up on this issue. The state of science today is a travesty and I’m glad to finally hear someone acknowledge this as I have been along in the dark with these troubles for far too long. I know we are creating the foundation of something great but acknowledging that the current state of science is not something we can rely on is just simply not said or acknowledged. Im so happy and so grateful that you have spoken about his issue and brought it to the public’s attention. Thank you for you work and congratulations.
The model we have is great, the problem is anything can be hacked if that is your goal. If you build a better mousetrap, Nature will build a better mouse. The problem is the incentive. Not enough money to go around for your own research and tenure is disappearing, so to do the work you want to do you need either to be a. Well known / respected in your field OR take funds to do work you don't want to do so you can do the work you do want to.
We shouldn't worry too much about our money getting into publishing false positives. At the end of the day only the true positives will be the basis for further advancements, experimental science is built on previous results and if those results are spurious, nature will stop you from discovering further real relationships. That's what this video is failing to point out, the incremental nature of scientific knowledge in the natural sciences is a natural peer review system and the best that we can ever have hoped for. So keep funding science, at the end only the true relationships will stand the test of time
This is why statistics should be a mandatory course for anyone studying science at university.
Knowing how to properly interpret data can be just as important as the data itself.
It's generally not done at undergraduate, but a massive part of a PhD is understanding the statistical analysis that is used in research. It is extremely complicated and would be way too advanced for an undergraduate stats course for, say, a biology student.
David that‘s usually not taught in undergrad in the us? Wow, that surprises me - a biology student from Germany, where we have to take a class in statistics in our bachelors.
It might be easier to understand it during your PhD if you heard about it before
As a graduate MS student in A.I., I found my research statistics course to be probably my most relevant in terms of learning to think properly as an individual with an advanced degree. I was very much taken by surprise by statistics, pleastantly so.
Mirabell97 I took an Engineering statistics class in undergrad in America. I've also taken graduate level research statistics as a Comp Sci student, which was taught at a much much higher and more relevant level.
There are also high school statistics class, which are even more watered down. So as you say, many have indeed heard about if before.
forgotaboutbre glad to hear that :)
we should open up a journal for replication studies only
With full time staff who never take any compromised funding.
Whose gonna pay for a journal no one reads?
Call it "Well yes, but also No"
darth tator definitely
warwolf6 Anyone who believes in the importance of replication studies? It’s not like everyone who reads it will have read every study. Far from it. It could be multi-disciplined also, which could be interesting to learn about other sciences
As a person that loves science but is not in the field, I’ve become quite disgusted by the lack of integrity shown by the university system. They have been corrupted to the core and need to be cleaned out. It’s become big business now and is not to be trusted if profit is the driving motivation, that’s not what universities are for.
I have no issue with for profit companies doing research and development as long as everyone knows where it’s coming from and is driven solely by profit and is treated as such.
I definitely understand that feeling. As a scientist who has spent a disenchantingly long time in academia, I still have faith in individual scientists and the prevailing winds of science overall. Look how far the world has come in such a short span of time (for good and bad). That progress is built laregly on a basis if good science; the bad stuff ends up getting filtered out. Universities absolutely operate for profit, but not everything that makes a profit is without merit in my eyes.
The problem is that academia is incentified to publish rubbish, every year 15 million people are graduating and 3 million people finishing post graduate studies. Every year they need to generate millions of papers of "new and novel" research. And if you are working in academia then your livelihood depends on publishing.
"The greatest enemy of knowledge is not ignorance, but the illusion of truth"
- After reading replies I have no idea who the author is
"The problem with quotes on the Internet is that it is hard to verify their authenticity." Abraham Lincoln (source: the Internet)
(quote is not from Hawking; probably from Daniel J Boorstin)
I've seen a variant of this attributed to Voltaire.
as a counterpoint, this quote has the illusion of truth about it. I like it!
I remember seeing one like this attributed to Benjamin Franklin. Long story short, quoting individuals on the internet doesn't really seem too productive if you ask me. The idea should stand on its own and not hinge on someone seen as intelligent having said it. Then again, what do I know? I'm just Elon Musk.
@@danielsonski the authenticity doesn't matter, just the meaning behind it
As a PhD student, I can fully agree with this. I have come to hate the word "novel". No matter how correct and in-depth an analysis is, anything that doesn't turn the world upside down is always gladly dismissed with "not novel (enough)" as a killer argument. By now I've decided for myself that I don't want to have anything more to do with the academic world after the PhD. I love research, but I HATE academic publishing.
Honestly, it's such a shame that the system is losing out on terrific men like you. I hope (if you don't go into the workforce) you can continue as an independent researcher. Hopefully one day this system will change.
Consider starting your own publication journal?
Is there anyway to pursue a research or research-like career without the problematic issues of academia?
Academic publishing is a nepotistic and simultaneously cannibalistic _industry._
Nah Christian, just go work in industry or big company. Better money HA
Thanks for making these videos they are such an eye opener for me. I never thought this would be an issue at all, now I understand.
Excellent explanation! Similarly, showing a frequency spectrum from an FFT and interpreting a peak is meaningless unless the caption includes an admission of whether the region displayed was selected before or after the peak was noticed!
Publish or perish ... and quality goes to the drains
So much research exists purely so someone can get their PhD, or bring funds into their University to keep themselves employed. When the pressure is on, no-one really cares whether the research is useful or even reliable - just got to fill the coffers and get your research published and referenced to drive up your University's rankings.
@@Thermalions Well, certainly, there are millions of theses out there, all required for the PhD. No way around that. Most of them are garbage.
@@UncleKennysPlace I would say around 95% of them are garbage
It has been estimated that it takes $5 million in funding to make a Ph.D in a STEM field. The research community has been corrupted from the base.
money is the root of all evil
I have an hypothesis. I think getting in car accidents decreases your chances of dying from cancer
...but increases your chances of dying in a car accident.
"I shall test this! >8/ " -Hopefully some scientist out there.
Good analogy.
False. Somebody just published a paper about that. You have 100% chance to die from cancer if you where in a car accident. It was a small sample size, about 1 man. He was a truck Driver in chernobyl and he has been in small accident once. He died from cancer.
"You have 100% chance to die from cancer if you where in a car accident."
so if you get in an accident, you will for sure die from cancer!
***** xD
This 12 minutes should be mandatory viewing for every course that touches the slightest bit on any kind of science, engineering, statistics, political science, or journalism. Starting in junior high school.
Lying is at the center of so many American systems, it’s quite depressing
This seems more like a problem with the publishing system over the scientific method.
Yes you are right but the bottom line is that any new scientific theory is completely unreliable. Since there is no other way to do science today other than the peer review method.
I think the problem is actually quite deeply embedded in academic research. Right from the selection of which projects get grant funding and resources onwards there is bias to show the result that the department head wants to be true. Their career, prestige and income relies on this. The careers, prestige and income of every person in every academic research department relies on only ever finding 'convenient' results.
It's human nature. That's what publishing is all about, exposing the study to other scientists, and seeing if it survives.
Publishing is about making money, so they have the exact same problem as scientists do. Go for money or go for truth. Since the publications wouldn't exist without money, they are making the only choice they can.
@@blitzofchaosgaming6737 Publishers earn (among other ways) money by selling subscriptions. They could publish anything
Interesting. I am adding this video to my research courses. My students don't always understand why we need to be critical of research.
Modern science is science. Just because you're emotionally upset over something in the news doesn't invalidate the scientific method.
J Thorsson you can't talk about the "lack of self critical thinking" after trying to say that you're smarter than someone else because your daddy is a manager at a science institute... you can't contract knowledge lol
+ nosferatu5
The scientific method is indeed unsurpassed. But what this video is about (though it doesn't say so) is a relative newcomer in academia, the NHST, and its use is based on anti-science. Does fitting evidence to theory sound like science to you? I hope not, it most certainly does not to me. But that and anecdotes is what a young academic told me was the norm in many of these fields. I'd order a retest or reinterpretation of every NHST study from 1940 onwards using the actual scientific method, complete with logic and falsifications, given these results. A failure rate of 64% is abysmal, yet predicted by Ioannides.
Don't forget something Derek left out though. Re-sampling fixes a lot of these issues. Run a sampling test including replications to get your standard deviations and get a p-value less than 0.05 (or even better, less than 0.01 or 0.001). Then rerun the sampling tests multiple times to see if you can repeat the p-value. THEN (most importantly) report ALL your experimental runs with p-values. If even one out of (at least) three to five separate independent runs has a non-significant p-value, consider the entire study with a huge pinch of salt. Most reputable journals though nowadays insist on this - the peer reviewers worth anything will at the very least.
youre students have the same "publication incentives" as those publishing these "findings".
Given recent high profile scandals in the field, an updated version of the vid would be greatly appreciated
Thanks for the analytical look at this topic. It seems timely with the recent resignation at Stanford University. It reminds me of a former colleague who shared the quip "publish or perish." In today's political world, the phrase "follow the science" is frequently and ignorantly applied, I'm glad to see science influencers such as yourself shedding light on this topic.
Just started my PhD. This video has inspired me to call in consultants outside of my supervisory team to check my methods. I don't want to be wasting my time or anyone else's with nonsense research, and I'm honestly feeling a little nervous about it now.
What is your research area?
Have you had a bad time with trying to be honest in science yet?
@@angrydragonslayer What does this even mean?
@@whyplaypiano2844 what part do you not get?
@@angrydragonslayer The whole comment? It isn't phrased very well. Are you being sarcastic, or serious? If you're being sarcastic, it's either because you were trying to be funny, or because you're--for a lack of better words--salty that people aren't honest in science. If you're being serious, it's either a sincere question, or you genuinely think scientists are dishonest on purpose. Explain the mentally you had when you made the comment, I guess?
P values of 0.05 are a joke.
Look, I'm going to sound biased, and that's because I am.
This is a much bigger problem in fields like Psychology than in fields like Physics. The emphasis on constant publication and on positive results is still a massive problem. Researcher bias is still a massive problem (although still, not as much as in Psych/Sociology). The existence of tenure helps a little since researchers become able to research whatever they want rather than what the system wants.
But we aren't claiming world-changing discoveries with P=.05. Derek brushed right past this like he was afraid of sounding biased but I'll repeat: 5 sigma is a 1 in 3 million chance of getting a false positive purely by chance. Every physicist "knew" the Higgs had been discovered years before we finally announced it and started celebrating. But we still waited for 5 sigma.
I did some research with one of my Psych professors in my freshman year. She was actually quite careful outside of the fact that her sample sizes were pathetic. We went to a convention where we saw several dozen researchers presenting the results of their studies, and it was the most masturbatory display I could have imagined. There were some decent scientists there, no doubt, but the *majority* of them were making claims too grandiose for their P-values and sample sizes, confusing correlation with causation, and most of all *failing to isolate variables.* If a freshman is noticing glaring problems in your research method, your research method sucks.
The next year I had a Physics prof. who had a friend of mine and his grad students run an experiment 40,000 times. There is no comparison. We need a lot more rigor in the soft sciences than we have right now. Mostly because science. (But also because they're making us all look bad...)
And there's also the problem that experiments might be difficult to perform in fields outside physics. It can expensive and it requires a lot of planning and logistics. Not to mention that ethical dilemmas might stand in the way, which happens a lot in medicine. In a way, the physics field is blessed by not depending on studying people, and overall physics experiments are cheap; expensive particle accelerators non-withstanding.
One thing I think Derek missed is to emphasize that one shouldn't be looking at single studies anyway. You look multiple studies for trends and toss out the flawed ones and the outliers. Or even better, look for meta studies.
I'm also unsure if changing your model / what you measure *after* you have looked at the data is p-hacking. Such a mistake seems way more serious to me, as you're basically making your model fit a specific data set. Give me any data set and I can make a polynomial fit all the points. Basically, reusing the data after changing the model should be a crime :)
On this note, simply because you mention my discipline (Psychology), I will point out that Psychology lacks any kind of unifying theory that organises the predictions it makes. It's a lot easier to be a physicist trying to confirm the predictions of Einstein and Newton than a psychologist guessing at what the underlying mechanics of the mind are.
We are all biased, but even the ones who admit it find it hard to fight it. I guess we can never win.
Another issue is that Sociology, Psychology and Economics are all black boxes that we don't know nearly enough about. In Physics, we can lower the temperature to close to absolute zero, and do the experiment in a vacuum. It is currently impossible to have that level of rigour in Sociology, Psychology and Economics. We still have a while to go.
I don't see why you need to single out psychology, even this video gives examples of neuroscience and physiology research holding even lower rates of reproducibility. When you look at the success of psychotherapy for individuals, you will find most people find it an indispensable resource in their lives, unlike the health tips or the vague claims about tiny brain regions coming out of neurology and physiology.
I love your background music. It's so early-2000s-techy in the best possible way
Frutiger aero type beat
Science might be our best tool to study and find correct answers in nature but it definitely needs to be improved a lot! Sadly up to date there are still people refusing to publish negative results, magazines refusing replication of experiments and the the whole publishing system is very subjective... and sometimes it works on friendly relationship or established connections... now that is not the right scientific way! Thank you so much for acknowledging this problem and make people aware of it!
The problem is not with Science. The problem is with peer review journals and publications. They have unintentionally ruined science. Edit: there is a reason scientists who are responsible for the progress of science are all fringe theorists like Albert Einstein and they do not come from the mainstream scientific community. There is no diversity of thought and expression allowed through peer reviewed journals. Scientific progress is unintentionally halted by peer review journals.
The lack of incentives for replication studies is obviously the biggest problem. The fact that some of those "landmark" studies were only attempted again recently...
Hopefully, as people become more aware of this (it's happening), all those journals will change their mind about replications. They should release a separate issue for them, even.
Agree. At some stage we will almost need to press "reset" and start again.
Yea, especially since almost every article in their conclusion implies that "further research in the area is needed" :p
***** Significant or not... It's always significant in some way
+JavierBacon I get what you're saying and agree. Even though your testing/conclusions don't have statistical significance, the findings are still significant. In most cases, it would still help increase our understanding of a subject if null results were published.
The best way to start is to get rid of journals telling us what is worth publishing and what isn't. Then kill the h-index/impact-factor that are genuine SHITS. Then put everything in open access, the universities have all the infrastructure necessary and could even save millions $ in subscription fees that are frankly incredibly stupid to begin with...
I had so much trouble to publish when I corrected the p-values to counteract "p-hacking" or alpha inflation. Since I tested for multiple variables, I adjusted the models to minimise false positives and low and behold, almost all hypotheses that would have shown p
Ugh, apparently negative results are so damn untouchables - the publication system really needs to chage
Stick all those negative findings in the supplementary figures of a somewhat related paper.
@@aravindpallippara1577 Just imagine, it's not compulsory to adjust the p-values. It's not mandatory to counteract alpha inflation. How much of published research must be (intentionally or not) not significant, but published as such.
But you kept your integrity.
Imagine all the human time saved from being able to get information from someone elses research without having to do it yourself.
Now imagine all the time lost from all the people who have to research what has already been done, but they cant learn it because negative results don't show up in books.
There is a pressure to publish significant results. As a research assistant, I know for a fact my professors engage in this. I was preparing the data I collected on a crop, and somehow the paper was published a week after I finished the data... didn't make sense
Definitely doesn't make sense, as the peer review process alone takes months. Could it be that you were reproducing some past experiments, or gathering the same data to be used in a future publication?
I'm so glad someone finally spoke out about something I have been worried about for so long. I thought maybe it was just my misunderstanding because everyone else seemed to be deeply engaging in this toxic/false process 🤔, but now I know it is indeed problematic. Thank you so much!
@@anchordrop1476 That's very interesting. Could you tell me the the other ways your professor would want you to draw conclusions? Also do you know which types of models statisticians are favouring?
I feel like everyone in the world needs to watch this video. There's so much crap out there an no one ever thinks past what they want to hear. This should help.
This should be a Ted Ed
+
Do you love science and all it's complixity but wish it could be a little less complex, and a lot less scientific?
Introducing TODD Talks...
You're right, people often only hear what they want, so this video would likely make that even worse. It gives people ammunition to discredit others with an informed view. People are going to see that if this is the result from honest science, then what happens to paid and biased science.
To a wider audience I think this video would likely do a lot more harm than good.
For me, If there was a video I'd like everyone to watch it'd be one purely on the benefits of science. The last thing we need to throw out to the general public is something that might look at first glace to highlight its flaws.
Oh I "Heard" that if you eat butter, you'll be healthier than those who don't eat butter. Therefore it is correct. /sarcasm
Well, as a wise man once said "Some poeple use statistics like a drunk would use a streetlamp -- not for illumination but for support".
That being said, the most frustrating bit is that the journals and financing agencies actively encourage p-hacking and discourage replicating dubious studies.
I'm stealing this quote, it's amazing.
There is nothing wrong with using statistics for support, as long as they are accurate and honest, AND you don't cherry pick them. That last part is often the biggest problem. I don't think pharma changes numbers any more, but they most definitely fund several studies and pick and choose what they want from each. That is not research, that is advertising. It's also changed a bit for them now that they have to have conclusions that at least have SOMETHING to do with the data collected. There was no requirement for that before, as I understand it.
The point about discouraging replicating dubious-or any-studies is important. There just aren’t incentives to duplicate or refute someone else’s findings, but rather come up with something ”original”.
On a similar note, as an engineer who frequently volunteers at elementary - high school science fair judging, I’m constantly dismayed at the emphasis that other judges-both somewhat “lay” and professional STEM judges-place on “originality”… at the elementary & middle school level, even, not to mention the high school level! (Ok: maybe a district or regional winner at HS needs to be decently original, but…) Many people place originality and presentation skills (not to be entirely discounted, of course, but still not #1) above scientific inquiry, larger data trials, strict controls, and even just a good, solid use of the basic fundamentals of an experience by as taught in elementary science class.
@@whatisahandle221 I believe that in experimental physics it is customary to publish independent replications of breakthrough studies in comparatively high-impact journals (as well as to cite replication studies along with the original ones in future papers). Sadly this is more of an exception that proves the rule.
In life sciences on the other hand there are so many subfields and so much competition, that far too many "original" yet shoddy papers (methodologically speaking) get published. My subjective impression is that this problem is slightly smaller in niche and/or "old-fashioned" subfields, where the odds of getting a reviewer who knows all the ins and outs of the topic are relatively high.
@@MrJdsenior They still do-and still do so much more. It is what it is.
There's a newer peer reviewed study that shows that the study "Why Most Published Research Findings Are False" is actually false. The author of the new study, however, says that disproving the original study in fact proves that the original study was correct - most published research is false. So he's apologized for his error.
Years afterwards the problem is still in no way solved, not much if anything was improved, but it's no longer talked about.
I guess silencing critics worked well.
“Science is the interpretation of data. The data is usually crap.”
Liam Scheff, science journalist and author
Ever heard of data wranglers?
Science journalist and author and he doesn't know that "data" is a plural noun? FYI, "datum" is the singular.
@@lixloon Why exactly did you assume he is talking about one point of datum? It's the less logical explanation. I'll just assume you're a moron who wanted to let the world know something that makes you feel smart.
Sugasphere and the Lancet concur
@@lixloon data is gramatically correct. It's not possible to interpret a single datum.
Years ago, I questioned some chemistry methodologies. It was very frustrating, because nobody was listening. Then a publication came out discrediting the methods used and discrediting many journal articles. Somebody had listened, or came to the same conclusions I did. Corrections were made.
TRUE., I HAVE THE SAME EXPERIENCE.
Vindicated!
When you say you questioned, did you establish a line of communication/collaboration with any of the authors or users of the method, working to test its limits, improve it, or compare it to other methods?
What were the methodologies, and were any significant findings overturned / discredited as a result? Or did it only affect small findings, with larger findings still being correct (or considered correct) in spite of some methodological errors?
You should have used an approach developed in cybersecurity research long time ago for the same issue: notify the authors that in 3 months you are going to publish your findings about all their mistakes no matter what they do. Then the authors have 3 months to retract their papers on their own and/or correct them. This solution is called "responsible disclosure" of vulnerabilities. You see, in cybersecurity the problem of "nobody listens unless you publish" has been acknowledged a long time ago. You can do this anonymously as well: from my experience, scientists are not more ethical than average human, and when you threaten their career and self-image they quite often freak out and try to hurt you back by all imaginable means -- just as many normal humans would in such a situation.
When he used a study to study the unreliability of studies…my brain raised its hand and waited patiently to be called on.
paradox alert
Nice video and a very important commentary on the flaws of research standards.
However it is fruitful to remember the lessons of "Structure of Scientific Revolutions" by Thomas Khun (the most cited work of social sciences in human history, by the way - for good or bad XD). A result is generally understood as wrong in hindsight, after failure not only of verification but also of the methodology itself (because usually it is the carpenter who is blamed, not the tools a.k.a. methodology, correct or wrong). The book discusses how the normal phase of science is necessarily a very focused study on limited topics, defined by the current paradigm. The paradigm informs not only the "right" theory, tools and criteria to do research, but also the "right" questions, i.e. the open problems of any field that are almost guaranteed to yield a solution after more "problem solving", using the standards of research in the current paradigm.
It is only when the paradigm starts failing consistently to solve new problems and becomes all patched up by "corrections", that researchers start adventuring on new topics outside of the "right" questions. It is like getting out of the green box in the upper left in Veritasium's figure and exploring the red outside, but only because the green itself is proving to be much redder than previously thought, and researchers now have little to lose by "incorrectly" making the "wrong questions".
It's definitely bad in medicine. John Ionnadis has conducted "meta-research" into the quality of medical research and concluded that most medical research is severely flawed---in fact, "80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials." Wow. There are also problems with dealing with complex systems, and with challenging scientific orthodoxy into which some scientists have invested their entire careers.
As an engineering student specializing in med tech have the strong impression that med publications are less elaborate, lower quality and contain less explanation than eng ones
I'd say 25% wrong is really really good for something as complex as medcin
I just *knew* John Ionnadis would come up here. If you want to see "mostly wrong", take a look at his record on COVID-19 predictions-off by two orders of magnitude.
His sensationalized contrarian kick has gotten people killed. There are many better, more thoughtful critics of the state of research.
I'm not saying he's always wrong. But he does go for the sensational, and is often sensationally wrong, and doggedly so.
A lot of progress has been made, especially in medicine, with pre-registration of trials, data & code repositories, etc, and I'll give him credit for helping kick-start some of that. (Preprints seem to me to be a move simultaneously in the right and wrong directions!)
But statements like "80% of non-randomized studies turn out to be wrong" isn't even well-defined enough to be falsifiable. It's a non-scientific statement. And meta-research, like meta-analysis, is itself extremely subject to selection bias. Each need to be approached with great care and skepticism.
A lot of what he says is not controversial. I'm not here to demolish John Ionnadis, but to urge people to steer clear of his broad sensationalized generalizations, and look carefully at the arguments he makes. Apply the same critical standards that he urges to his own research.
Sometimes the kettle calls the pot black-but black is still black.
There is also the problem that you can test one medication, to some degree, but if you start talking about interactions between different medications in different people, most of the bets are definitely off. People discount "anecdotal" data completely, but if that data comes from doctors reporting on those medications, it definitely has value, as well, IMHO.
The vast majority of medical research is a shell game, run by pharma. You can tell little by study conclusions: you can actually tell something by the set of study parameters. Where study parameters will produce a unwanted conclusion, the research doesn't happen or isn't published. Example: no clinical or epidemiological evidence for the safety of aluminium adjuvants in vaccines. Draw your own conclusion
It should be one of the requirements for getting a Bachelors in a field of science to do a replication study. Even with small sample sizes.
It is a useful experience and pattern of thinking to carry into adulthood.
Furthermore a meta-analysis using dozens or hundreds of further experiments would shake out all incorrect P values
An honestly, meta-analysis should be just as important to publish as novel as there is so much data out there and it has never been easier to analyze quickly.
That's essentially what they do in a lab class
The fact that null hypotheses are not published also skews studies because the data in those is not referenced which would dilute the influence of non null hypotheses referenced.
Thank you so much, it's such a thought-provoking video.
The point is clear and striking, the cost of being wrong is always low but the real cost is not getting published (cruel scholar world).
The most interesting question raised here is "how often we deluded ourselves when there's not statistic and peer review", which again reminds that most of us just live in our self-constructed little bubble and continuing building things based on what we subjectively believe.
In Indonesia, many of supervisors in medicine would reject replication studies, expecting new studies and publication and therefore, causing us to have nearly zero epidemiological data. We prefer "good-looking research" to actually researching anything. Better not research than not looking good
Your supervisors are speaking the language of gods
"Data doesn't speak for itself, it must be interpreted"~ and there we have it people the point of this thesis.
I believe a good example of selective interpretation of data is in the small print at the bottom of TV ads for things like cosmetics and pharmaceuticals where it says “83 of 127 people agree”. It’s a section of a larger study group that has been narrowed down to the most favourable ratio possible, hence the small, odd sample number.
These thing have always been in my head. Thanks for saying it out and being articulate on this.
When the contestants found out one of the walls would contain an erotic image, they enabled their inner chakras to get it right
Short answer: yes. That was a real wake-up call when I was doing my Masters degree literature review - how often university professors push publications using "academic standard" statistical analysis to come to a demonstrably wrong conclusion. It is scary, not only how often this was the case, but how often these studies would be cited and their misinformation spread through academic circles without question.
Most academics doing the research are young and inexperienced in the real world. The people managing the research departments have a vested interest in only promoting research that finds 'convenient' results that will enhance their chance of getting bigger budgets next year.
Maybe we should take people with 30 years of industry experience and put them in charge of research in academic institutions.....
@@davidwebb2318 Unfortunately true. If, as a young scientist, you talk to the head of your lab or department about your work and what your ideals are or what your idea of good science is, you will quickly be taught. You don't know anything! No, you really don't know what is important in science. What you know even less about is what "good work" is and what is expected of you. The most important thing is neither "good science" nor a prestigious publication. At the very top of the hierarchy is an accepted proposal letter! No funding, no research. All other output must be directed towards this goal and are just means to an end. The larger the organisation (Pareto Principal), the greater the pressure to meet this requirement. Exceptions exist.
@@haraldtopfer5732 I agree. Academia has become a big industry with big careers to support. The priority of the people heading up departments is to build bigger empires, secure bigger budgets and increase their personal exposure/status. This secures their jobs and the jobs of their colleagues/friends. That trumps everything else in many cases.
It is really obvious in the climate change industry where nobody ever proposes or approves any budget for spending on anything that doesn't support the pre-existing narrative. They carefully choose and support only work that adds weight to the doom stories because this expands the 'importance' of their industry. Their future careers and their salary depends on doing it so they embrace it and steer all the research in one direction. The system is really flawed and has created a monster where half the world are intent on economic suicide to cure a problem that is relatively minor and will only have any impact over generations.
@@davidwebb2318 Well the thing is virtually every study that disapproves climate change are very usually well funded themselves - There is a vested interest among the folks with resources to forward that narrative as well, and they have resources, profits they can lose. Not to mention these studies also have to exercise pretty big mental gymnastics as the mounting evidence grows.
Money does make the world go around after all.
Wouldn't you agree?
@@aravindpallippara1577 No, I wouldn't agree. The climate change industry is mostly based on an emotional sales pitch pushed by celebrities and political activists who haven't got the first clue about the actual data concerning the climate.
This is obvious because the main activists are pushing the idea that humans will be extinct in under 10 years. Politicians who are too weak-minded to work out this is complete lunacy have simply demonstrated their lack of intellectual horsepower by going along with it.
Money does not make the world go round. It is just a convenient method of exchange used to buy and sell goods and services. Of course, the political activists that are using the climate change narrative to promote their political agenda will try to persuade you that money is evil (or that only evil people have money so they should take it and give it to people they consider more worthy).
Philosophical Theory of Underdetermination. We can never know the answer truly is but with science we can slowly eliminate some things that are not true. As a chemist working in pharmaceuticals, it's the hard truth pill to swallow.
When I was in undergrad, I was so obsessed to do postgrad for the sake of "in search of knowledge", now, i am a master student, and although I love the university library, I just feel "empty" inside, knowing that we cannot rely on "human knowledge" ... and even more interested in religious studies. Looking for the "pure" knowledge is hard today...
As someone interested in postgrad, would you recomend it to someone fully expecting the absurdity of human knowledge as an absolute truth?
@@bubblesbomb8949 yes, just do it. I just realized that, it's part of the learning journey....
How can we tell this research isn't wrong
oh the endless loop - there have been a fair number of attempted replications recently that have found pretty dismal results. When you consider they are all in agreement, that biases exist, that incentives are skewed, that .05 is not all that low, that p-hacking occurs, it is fairly unambiguous that a sizeable fraction (if not a majority) of research is actually false.
+Veritasium Wouldn't the odds of the exception being wrong be higher, than the odds of the norm being wrong? There's a reason why there's such a thing as peer review, after all. The scientific model is there to make sure you can replicate the results and methods of published papers. If something doesn't stand up to peer review, it's bad science, as it means something didn't add up.
you don't
sadly low sample sizes are a very common problem due to lack of finances or various other reasons.
That's the point: when deciding which papers to publish, the scientific method isn't being respected. There's selection bias tending toward publishing mostly positive results and not the inconclusive ones, and there's a complete lack of respect for replication since those studies are often rejected outright.
As a researcher, I find those numbers very conservative, even when I'm 4 years late to the video.
I also feel like there's a reason missing for the false-positive results category which is a deviation from the main objective. Some true positive results shouldn't be considered as such when you make an in detail analysis of their methods, statistics and final findings just for the pure reason that, mid-study, some parts of the objetive were changed to accomodate the findings. This is also an issue that pisses me off, especially in my research field where there's such a huge mix of different scientific areas that it's next to impossible to verify anything at all in detail because everyone just pulls the results their way.
As some people already mentioned here, some authors do withold critical pieces of information for citation boosts. If people can't reproduce something from a study, they can neither be proved wrong by the paper's information alone (as long as it checks out in theory) nor can they be denied autorships and citations from other papers which effectively boosts their 'worth'. The fact that researchers are evaluated using citations/autorship numbers is also one of the leading problems as to which false-positives exists in such large numbers (I don't believe false-positives are only ~30% for a damn second, but this is my biased opinion) and why some papers, even though everything checks out in theory, can never be truly reviewed on a p2p manner on the practical results sides of things.
Anyone who works in research knows there's a lot of... misbehaving on most published works, regardless of the results. Therefore I have to disagree with the fact that researchers are fixing some of the problems. It's not that we don't want to fix them, but because the system itself, as it stands, is essentially rigged.
We can sift through p-hacked results. We can't, however, sift through p-hacked results if the objective is mismatched with the reported findings (if someone told me that was involuntary, I'd believe them because I know how easy it is to deviate from it) nor from a paper which withholds critical information. And the worst part about it is that this is further fueled by higher degrees thesis such as masters or PhD's where it's mandatory to cite other people for their work to be 'accepted' as 'valid'.
You have to approach published works with a very high level of cynicism and with some time and patience on your hands if you're even dreaming of finding a published work that remotely fits your needs and actually shows a positive result on most scientific areas.
I hope someday a scientist gets very rich and decides to devote his/her money and time in creating a healthier scientific publishing environment.
When I finished my undergrad, I worked compiling a database for a retired professor. One day he asked me to find an article that had been recommended by one of his peers during review. He already had the author and subject so it was pretty easy to find and got me a nod in the paper for my invaluable research assistance. The paper was on how long bones had been drawn incorrectly in every medical text forever. Someone had drawn it incorrectly once and everyone had copied the original mistake.
@@pattygould8240 What happened with the paper? Is it available?
@@ejipuh I have a copy that he gave me when it was published but it's packed away somewhere and I frankly don't remember what journal it was published in. I worked for him summer and fall 2004 so that's when it was published.
@Luís Andrade doctors have been learning from those textbooks for over a century, the mistake in the drawing didn't have an impact or someone would have pointed it out sooner. It took a scientist studying bones to point out the error.
I think a simple fix could be that *publishers would be held accountable for any research they publish.* As a direct result, they would be *forced to publish any replicate research that results in different results* from previously published research.
If publications do not want to do lots of replicate research publishing, they should improve their own quality checking (basically more elaborate peer review) before publishing.
Thank you very much for the insight. As an academic, it is important to keep that in mind. I would like to say the situation can be much worse than you outlined because science grows on what has been discovered before. Important findings are likely to be used in subsequent experiments, even if just as biases, invalidating an exponential (logistic?) number of results. Replication should occur fastly, with a speed proportional to the importance of the study (the exponential growth estimate).
This is the kind of material CZcams needs more of
It's something extremely ironic, but CZcams encourages other types of content, just like journalism encourages certain kind of results in science
I'm curious about the comment you made at the end that "as flawed as our science may be, it is far and away more reliable than any other way of knowing that we have."
I'd love to see a video on:
1) What are the "other ways of knowing that we have?"
2) A critical evaluation on why science is better than those "other ways of knowing"
~ A loyal fan
well, there's using logical deduction to eliminate improbable causes.
have you ever heard of IB theory of knowledge? These are exactly the type of questions we discussed in class in high school, it really opens your mind
That's the scientific way of knowing, isn't it?
What is the "IB" stand for?
International Baccalaureate
While getting a psych ba I wondered why journals are pretty much unregulated. The fact that a journal can publish findings then refuse to publish studies that disprove or refute them is troubling to say the least.
This video needs to be revisited in 2023 in light of the Marc Tessier-Lavigne scandal and the Greg Semenza retractions, just to name two of MANY academic frauds that have been exposed since this video was published.
Outstanding video. It wasn't until I really started getting into research at MSc level that I began to realise so much of the research I was appraising was deeply flawed. At undergrad, I assumed that it was ME who was flawed every time I saw a glaring error. At that level, you don't have the confidence to criticise the work of experienced researchers.
We had to write a literature review on a chosen subject for our B.Sc. I read through dozens of articles on my subject and to my horror I realized that the results weren't in line at all. It seemed that some scientists had worked with rats and some with mice and they got different results. Still, many sources quoted each other regardless. It was difficult to piece through that mess and know who to trust.
industry influence is everywhere unfortunately. Climate science in an example of that. It's sad because you grow up learning to trust others. Now it seems so confused that we are starting to rely in religion, faith, miths, and so on. In Italy the misinformation campaign is tragic 😷
An undergraduate whom I knew, spent months trying to replicate a chemical synthesis that had been published in a journal. He failed repeatedly. Finally he contacted the authors. They told him that there was a typographical error in the article: the concentration of one chemical was listed as being 10 times higher than it was supposed to be. With that correction, his synthesis worked on the first attempt.
Kevin Byrne Does this mean that scientific journals don’t publish errata?
@@ephemera... -- The errata often don't appear until months after the original article. And the errata are often buried. It would also be helpful if authors checked the galleys.
Each and everytime I see some article that says "According to studies by scientists...", I always and always read with skepticism.
Good! Always read with skepticism. That only benefits science.
Yes, that's the point ;)
which means how much more skeptical we should be of everything else, "alternative news" sites, alternative medicine, health blogs, mom blogs, etc etc...
Read with skepticism and report them to the authorities!
And then to think that scientists are bound to produce more truth than anyone else, you need to question everything and everyone around you
I was reviewing statistics on articles for my wife's thesis. The data on two of those articles the team used the sexagesimal system to measure time, then took them to get statistics markers as if they were on the decimal system. It hurt my eyes to see "an average time of 15min 89sec" in the article, not because I'm an engineer, but because anyone with elementary education would notice something was wrong.
Well this aged like wine
Anyone who reads articles online about "new research" needs to watch this
or people who hear science quoted (sometimes incorrectly) by Today Show, Dr. Oz, even Time Magazine etc.
I'd like to point out, as he hints at near the end, that the underlying reason for much of these "p-hacked" studies is due to human nature and not the scientific process itself. Stopping a sample size when you find convenient, not getting published to counter-study, people only interested in unique findings; these are all human fallacies. A manipulation of the scientific method.
There's no "the scientific method." That's a complete myth. You should read "Against Method" by Feyerabend. Even if he goes overboard in his argument (which I wouldn't necessarily agree he does), it's naive to think of a defined, precise method in which science *is* done, or *ought to be* done. It's really "anything goes," as long as you convince your peers. Hopefully truth is convincing.
@@d4n4nable That's wrong. There's no set methods, but a general guideline to seek the truth. As OP said, if it wasn't for bias in the publishing, then the system would work fine. The scientific method is more of a way of thinking and general guidelines in how truth can be determined.
@@neurofiedyamato8763 You act as if epistemology were solved. There's no consensus as to how to get to "truth." There are various methodologies implemented in various fields of research.
@@d4n4nable I can just smell the narcissm from across the screen .
@@arhamshahid5015 Narcissm? Why? Because I'm pointing to a classic contribution to the philosophy of science? It's not that I wrote it. I just read it, like thousands of others. How in the world is that narcissistic?
"Is most published research wrong?"
According to most published studies on the matter.... yes!
Thank you for putting this together and sharing in a way that will reach more people!! Kudos 😊
I think people without any knowledge of statistic will not get the point of this video. And that is most people.
I disagree and that's why I made it!
i think its the other way around. i think people with a lot of knowledge of statistics actually dont learn anything new from this, but people that dont will learn so much more.
Veritasium Thank you for making this video! big fan. I just think it is a good opportunity to introduce statistics to more people. Come to think of it maybe squeeze in how P value works in this video is not a good idea.
Not really. It's not that hard of a concept to understand, and I would hope that of the 3.6 million subscribers (and the fraction of those that actually watch his videos consistently), most of the people would know what a p-value and statistical significance is.
people who finished high school should understand, and of course if they are interested in science they are more likely to understand.
When I first came across this problem, I wanted to become a scientist who simply redoes the old experiments. I am still very far away from becoming a scientist but I hope this becomes a legitimate job. Having a subset of scientists who simply redo the experiments with a little or no tweaking.
Problem is who will pay you for it.
We need this. Can someone start an organization that does this? Not me, I have another thing to start. :P
Also, there are AI to analyze data of experiments regardless fo the human conclusion. I think those are pretty helpful in sorting out truth from falsehood.
there is almost ZERO funding for this important task . more money is spent each year to study the mating behaviour of saltwater mud worms. I'm not even kidding ....
"scientist" What does that even mean?
You have to study a certain field and then you can get a job at a university where they'll pay you for your research.
@LazicStefan If you're talking about climate change, it's real and the effects are observable outside of papers
Too much money is now involved. It is serious business now. I was a researcher assistant that was involved in every level of research at a prestigious institution. I have had multiple top Physicians tell me that if you can't produce results, the pharmaceutical companies will find someone else who can. It is not fringe researchers doing it. People who won't/can't play ball, just don't get promoted to leadership positions in research.
It is more flawed than Veritasium understands-to the point that I seriously question his conclusion. :( :(
I appreciate that Veristasium looked at this issue.
Caveat:
That is technically not what the p value means. It’s NOT whether the hypothesis is true, it’s whether you can or cannot reject the null. Then, the significance is the probability of finding an equal or greater effect given the distributions of your data.
Oooooh man that's a good point. This whole discussion is predicated on the false concept that papers are somehow declaring what is true and what is not true.
Why would anyone give this a thumbs down?
Spent most of my life in research, painful yet true....
Ignore most of the thumbs down. 10-year-olds and trolls will down-vote a good video just to agitate people. It doesn't mean anything.
Life gave them a thumb down. Ignore 😂
Reason 1: Some one worked so much only to add a number to the papers published but not quality. Some other person points out a mistake in those papers.
Why not? Maybe they dont know dont recommend video function , so they thought thumb down this video will result in similiar type of video became featured in their homepage.
9 times out of 10, the answer to this question is BOTS. They have to like/dislike videos at random to try and fool the algorithm. That's all it is. I'm so tired of seeing "how could anyone dislike this GREAT video??" IT. IS. BOTS.
Yes. Results are not science until verified/replicated! This is the scientific method.
Very informative video, thank you.
Too bad the studies you see on Doctor Oz (the studies most of the sheep enjoy listening to) are never fact-checked because that would cut into profits.
Tell that to sociology majors and they'll call you a bigot.
Sadly though, the same can be true to a greater extent for legitimate science. Replication studies weren't getting funded much back then by the government or other sources precisely because it's not bombastic or groundbreaking enough to advance the field, so basically only a trace number of replication studies ever gets funded and published.
In short, landmark studies didn't get fact-checked and replicated a lot because it would cut into their grant money application and prevent them from conducting the studies in the first place.
Good thing it's changing nowadays though.
Sorry cant afford to replicate this experiment, the client didn't give us enough of their product to do further testing beyond the results they requested we deliver. We are a private laboratory and need to be profitable.
I feel like the structure of how studies happen in general gives way for them to be messed up. It's a classist environment, because somehow you usually only get to do research like this if you have the privilege of doing/having a PhD. The amount of studies that we would have that would check each other if more people with just a B.A. and M.A. got to do their thing (properly supervised & peer-reviewed which once again is hard because of how the field is structured) is wild. Psychology would probably look very different when it comes to brief therapy interventions (currently very underresearched).
Always excellent and insightful!!
If you publish a concept that the "Optometrist majority", does "NOT LIKE", well you will never get published.
Been there ... DONE THAT!
I've been a world-class AI researcher for almost three decades now. I have personally, during this time, witness much deliberate scientific fraud, including rigged demos, fake results, and outright lies. Additionally, numerous colleagues have admitted to committing scientific fraud, and I've even been ordered to do so myself. I have always refused. I will not, as a scientist, report results I know or suspect to be misleading. My family and I have been severely punished for this. So I recently returned to mathematics, where true and false still seem to reign. And lo and behold, instead of abusive rejection letters, written on non-scientific grounds, I get best-paper nominations. PS: don't believe any of the current hype around AI.
That's terrible , there are many stories like this that keep popping up. Stay strong this crap will change soon .
Could we talk I'd love to hear your thoughts on this Christer
Christer Samuelsson why would I believe this?
Dang man I think you got out of A.I. at the wrong time lol. People dont have to fudge their results anymore because the results are real and improving every day now.
Let me guess- it was natural language processing wasn't it?
I intend to live forever. So far, so good.
i like that haha
Who wants to live forever?
Ifyou live forever you'll see everyone you know die and then everything you know die because the universe willend.
+RedEyes Cat dude no way! That must be a record
"If you live forever you'll see everyone you know die and then everything you know die because the universe willend."
If the universe ends, something will come along to replace it. I'd be quite excited to see that.
Plus, it's not necessarily true that "the universe" will end, although that's a widely spread myth so I can't really blame you for assuming that.
With most of the World’s activities trying diligently to habitually and continuously ascribe to its own peculiarities? Why would the presumed outcome ever need to lie?
as a phd student this is so true, our university as an example force phd student to publish 2 articles ( to get ehir phd) or more and in journals indexec in databases such as scopus or web of science and this make lot if students p-hack and tweak results or falsefy them completly and it's the same in the market of research or trying to be a professor the competition is by the number of published articles some students have over 20 or 30 article which is making people do whatever possible to publish
The real problem here are the journals. They have established themselves as the primary way of publishing. There are other ways, but in the end, the journals get you recognition and jobs.
That results in many studies being done with the intent of publishing. Scientists cant be blamed for that. After all, they not only do the research but also have to contantly beg for money.
The actual goal of optaining information gets lost along the way.
Exactly. One high-impact publication can set up a career, and leads to 'light-touch' peer review at other good journals, soft authorships on colleague's papers and requests to be co-investigators on other people's grants. More publications leads to more funding. Even as a Co-I that doesn't actually get money from a grant, you have demonstrated 'grant funding' success. The incentives to join that group are high.
It seems even more absurd to still have these gatekeepers publishing a limited number of papers when we live in the era of long tail economics
Researchers also have to pay the journals to publish their work, who in turn often charge you to read them.
Like all other systems and institutions, scholarly research and the academy is a game, with numerous irrational inputs and agents in pursuit of self serving interests.
Indeed. I lost interest to pursue to phd bcoz of this reason.
3:32
but
what if that paper is false
bacon
best get started at trying to verify their results then
if that particular paper were false, that would mean most published research is true, which would mean that paper is most likely true.
Indigo Osmanthus Paradoxial...
Hamza Aman
and round and round we go...
best Veritasium in a long time! More data analysis!!!
thanks, now i don't know what to do with my life. i'm a senior in highschool wanting to study physics, but i have watched a ton of videos that explain the reaserch paper publication strategies, and the way academia works in general, and now i realise that the perfect knowledge making science world i wanted to be a part of is nothing like i though it was....