AMD says these are the same... We DISAGREE. - Testing 12 of the same CPUs for Variance
Vložit
- čas přidán 2. 06. 2024
- Remove your personal information from the web at joindeleteme.com/LinusTechTips and use code LTT for 20% off.
NexiGo is hosting product giveaways every month! Enter for your chance to win here: viralnation.link/LTTxNexiGoGi... (Contest is open to anyone 13+ in the United States)
Do you really know if your CPU is performing the same as the ones we review? We don’t know. But we know that if we want to increase our testing capacity, we need to PARALLELIZE. But that means we need nearly identical test benches. And trying to make that happen sent us down a far deeper rabbit hole than we could have anticipated.
Testing Process Doc and Forum Post: linustechtips.com/topic/15546...
Buy an AMD Ryzen 7 7800X3D CPU: lmg.gg/s8Y1K (Canada: lmg.gg/yZnVo ) (Micro Center: lmg.gg/EkzYG )
Buy a NVIDIA GeForce RTX 4090 FE Graphics Card: geni.us/Won6A
Buy a GIGABYTE X670E AORUS XTREME Motherboard: geni.us/FADHuEG
Buy G.Skill Trident Z5 NEO 6000MT/s CL30 2x16GB DDR5 RAM: geni.us/TYRaJf
Buy a Samsung 980 Pro 2TB NVMe M.2 SSD: lmg.gg/aL0BL
Buy a Noctua NH-D15 CPU Cooler: geni.us/lK6GI9O
Buy a MSI MEG Ai1300P PSU: geni.us/NRCy
Purchases made through some store links may provide some compensation to Linus Media Group.
Purchases made through some store links may provide some compensation to Linus Media Group.
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloatplane
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
► EQUIPMENT WE USE TO FILM LTT: lmg.gg/LTTEquipment
► OUR WAN PODCAST GEAR: lmg.gg/wanset
FOLLOW US
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.com/us/album/sup...
Artist Link: / laszlomusic
Outro: Approaching Nirvana - Sugar High
Video Link: • Sugar High - Approachi...
Listen on Spotify: spoti.fi/UxWkUw
Artist Link: / approachingnirvana
Intro animation by MBarek Abdelwassaa / mbarek_abdel
Monitor And Keyboard by vadimmihalkevich / CC BY 4.0 geni.us/PgGWp
Mechanical RGB Keyboard by BigBrotherECE / CC BY 4.0 geni.us/mj6pHk4
Mouse Gamer free Model By Oscar Creativo / CC BY 4.0 geni.us/Ps3XfE
CHAPTERS
---------------------------------------------------
0:00 Intro
2:25 Why Same Model ISN'T Same Performance
5:32 Sources of Variance
8:21 Gaming Results
10:43 CS:GO is Wonky
11:41 Gaming Results Cont.
12:39 Productivity Results
13:59 Our Selections
15:15 Testing Mobos and RAM
17:16 Final Discussion
21:26 Outro - Věda a technologie
This is LTT using their higher budget compared to most tech channels for something genuinely useful. This is really good to see! Well done LTT, this deserve genuine praise.
Gamers nexus did the same thing a while ago
@@zypher5876 Gamer's Nexus not listed as other media. Funny
@@zypher5876 yeah, true. But Linus has a far greater scale than Gamers Nexus, as of now, reviewing everything from CPUs, GPUs, laptops, etc, so it's cool to see that level of standardization on a large scale to make sure the testing is consistent.
Well , those overclocker do lots of binning and probably run through more samples than the average LTT benchmark review
@@joaomendes3907 I'm pretty sure that's not out of malice, since Linus' not the editor for it 🤷♂️
glad to see factorio used as a gauge for cpus
As it should be lol it's mental intense on ticks. Causes some real aggro on multiplayer systems with different spec.
haha, guage, i get it
Factoriohno.
The issue is all of these reviewers are doing the factorio benchmark wrong.
The game has a 60 tps cap as that's what the engine runs at. What needs to be tested are heavier maps that push the cpu BELOW 60. These above 60 tps benchmarks are as good as a synthetic benchmark in that they're not realistic.
And yes, results absolutely do change and so does the cpu hierarchy.
@@whatistruth_1not to mention Factorio scales more with memory performance than core perf or cache.
As a data scientist, I absolutely love this video and greatly appreciate this perspective. The silicon lottery is absolutely real, and sample size of one is far from sufficient. Glad to see someone doing more in depth analysis. I don't expect the methodology of ltt labs to dive into frequentist vs bayesian stats, but it would be interesting to use public benchmarks (for an incredibly nerdy perspective/audience) as a prior distributions and see how the results differ. Regardless of the depth of it, just seeing more stats in hardware/software benchmarks is a fantastic breath of air. Keep up the good work!
La muestra es real, en el consumidor final la variación será más grande pero le darán la culpa a los otros componentes como cantidad de ram, ssd u otros factores, AMD solo garantiza ka velocidad del reloj en ciertss condiciones, en el hogar la temperatura ambiente también ocasiona ruido en los datos. Lo cierto es que no hay un tercero que obligue a cumplir un estándar legal porque es complejo medir que los productos sean iguales
Agreed! Would also be interested in seeing a bayesian analysis with public benchmarks as a prior (though I'm also one of those nerds who would love more of this kind of stuff). I also appreciate how they detailed their similarity metric of using euclidean distance, along with explaining what that looks like in a higher-dimensional space for those not accustomed to seeing it used that way since it's such a common approach for any sort of clustering problem.
When public is put in test, there's always possible cheating or possible than manufacturer goes in game too.. if you understand what I mean..
LTT should hire a data team to take care of all this data. It would be great to have it properly stored and analyzed, maybe you guys could ge into testing neural network performance of GPUs too. I find it hard to find data on how well GPUs run AI tasks.
@@diegoberan7883I second the request for GPU benchmarks for AI tasks. Looking at some of what Puget Systems has done for their AI/ML benchmarks would probably give a good starting point for the LTT team.
I never realized how much work goes into ensuring consistency in benchmark testing. It's worrying to think about the lack of oversight in computer hardware compared to other industries and the issues that could ensue.
Now imagine real scientific testing and trying to get perfectly consistent results and not within 0.25%
As nice as it'd be, it kind of admittedly makes sense to an extent - Cars aren't tested and regulated because they're expensive, they're tested and regulated because they can kill people
Actually it's not an oversight. Just like Nvidia GPUs, they boost automatically to the max possible on the silicon. Though AMD should've never advertised the max (up to) clock speeds, because 90% of people won't get it. Nvidia did it right by advertising the base/lowest possible clocks, Intel tells you both the lowest turbo boost and the max with TVB.
Genuine praise to linus for taking time to address this and how confusing and weird these companies are becoming launching their cpus and gpus in today’s market
The silicon lottery's always been a thing. I didn't think it would still be this bad though, and it's nice to see it tested
Yeah, this video was like the old LTT content we used to get.
@@Blast_HardCheese I think it's always this bad but the manufacturers just locked the maximum potential performance so then it won't vary once it reaches the end consumer (so your typical consumer won't feel aggrieved because of it,) and you can always unlock it through overclocking.
But nowadays they use a different sophisticated method which almost makes overclocking obsolete and it is already unlocked for the end consumers, so we can see the various results between each chip.
@@Blast_HardCheese It's not that bad, all they had to do was find the maximum frequency that all 3 CPUs could run at and fix them at that speed. Then the testing should result in minimal difference. For customers, you should be glad if you get a golden sample and don't care if you didn't because you got what you paid for which are the guaranteed speeds. Everything over the guaranteed speeds is a bonus that you may or may not get.
@@alexandruilea915Agreed, I just assumed they would have it fully reined in by now
Looks like the transparency we wanted. You clearly stated the challenges, how you worked around them and the variables you can't really control. It was very interesting, at least to me. Good job guys!
Statistically,from experience in semiconductor industry,
It will take at least a sample size of 30 cpus to establish proper sigma. Ideally you would want to do this experiment with 1 motherboard first and repeat with other boards to gauge variation.
The next step would be to test about 100 cpus from different time stamps (hopefully getting different lots),to gauge the variability of their process.
Also at JEDEC to remove differences in memory controller overclock quality.
Of course those hundred CPUs are gonna cost about $35,000... for every model of CPU that you test.
Seems like Gamers Nexus did not make the list of media outlets in 17:24
LTT Saltmine be like
LTT staying "classy"
Why would they? Gamers Nexus is a backpack warranty review channel, not technical.
Wow what a shocker 🙄
Lineypoo seems like the kind of guy that holds a grudge even when he's the one who's wrong lol
It is cute though that they still do all this testing when anyone with half a brain knows not to trust their data, it's like he has to justify spending all that $$$ on a team & equipment somehow.
So funny lol
Did I miss something between Gamers Nexus & LMG?
Incredible leveraging youtube videos to finance an independent lab to preform tests on all kinds of tech hardware and then making vids showing and explaining all of the data collected which in turn can finance the next round of tests. Bangin job LTT I love it.
Yeah, they have the equipment but no clue how to use it properly, as shown in this video. They pulled 11 tray CPUs, probably from the same tray. Does not reflect the consumer experience at all. Of course enterprise hardware will be more consistent, even if it is going in a consumer product. Imagine if a company like HP had people doing side by side tests of the same model and getting a 20% variance in benchmark scores...
In other words, the major manufacturers of complete systems get the best and what goes in the consumer boxes is like a random grab bag of everything else that wasn't good enough for HP or Dell.
This is why I benchmark test new CPUs and GPUs and have and will send them back solely for unreasonably low benchmark scores. If everyone did that, it would be a completely different market and the variance would become statistically insignificant for everyone, not just HP, Dell and LTT.
Oh, and don't think you can get around it buying a tray CPU from AliExpress. Those will be bottom of the barrel, just consistently so. You're a dreamer if you think that was some leftover from HP's order. Those were ordered by some no name company with the name produced by randomly mashing a few keys on the keyboard and were binned and priced accordingly. I'm sure you can buy 5600X CPUs there all day long that are really a 5500. And it is AMD that is the one running the scam there.
@@Lurch-Bot Did you miss the part where 11 of the CPU's came from 11 different stores across 3 countries (Canada, USA, and looked like UK)?
These 12 chips (including LTT's pre-launch test sample) are going to be from different batches across the 6-month window they were collected, which will emphasis the differences between wafers let alone between chips on the same wafer.
@@Lurch-Bot 22 seconds from the start of the video, that's how long it took for them to tell you how the CPUs were acquired, and you somehow still managed to make this comment. Wow.
@@Lurch-BotYour comment shows an astonishing lack of knowledge about silicon manufacturing, distribution and basic economic dependencies.
You also show a lack of text and information comprehension that might explain most of the rest.
If I were you, I'd scale down the confidence level of what I'm posting lenghtily and take a step back to evaluate what I actually know and what not.
Ummm, that's literally how businesses work. Spend money to make something, sell it so you can make more, and sell more.
i love the fact that you come out and say "yes, our tests are inconsistent, and here's why" in such a fascinating, entertaining fashion. there's a reason i always watch every LTT video that pops up in my feed.
you do know gamer nexus is better then this
@@AK-tf3fcnot comparable lmao, this video specifically shows us a big difference in gamers nexus testing vs ltt
@@lilcheatyIn that case lets throw all reviewers performance results out the window because of a few percent variance, I do agree with you though there is a huge difference in GN and LTT testing and that difference is GN can at least be trusted with their numbers but LTT cannot. "Look guys there is a variance so its not our fault we put out shonky reviews and fudged numbers". Silicon lottery has always been a thing but of course Loonus spins into this to obfuscate the fact their testing methodology is bogus at best. Wanna see a good correction video? HUB's latest video where Steve apologised for his mistakes and moved on, LTT "Silicon lottery cause number to go boom".
@@GsrItaliabecause gamers Nexus has to the best of my knowledge never came out and explicitly confirmed their testing method and if they have, it's not even comparable to this, no one else in the industry is doing what LTT is doing because they don't have the money to just buy 11 top of the line. CPUs and at least 10 4090s just for data accuracy reasons.
This isn't necessarily a knock against anyone else on this platform, they just don't have the money to do this
Yeah, how many times are they gonna have to say it before they realize they don't understand scientific method?
@17:24 discretely showcasing bad blood between Gamers Nexus and LTT 😅😂
I thought the same thing
@@minijag972Never trust pirate w/o ship. Such a big company, such a small sample.
@@minijag972 Yeah I always recommend you check out my competitor's shop after he smashes my windows 😂
Gamers Nexus incident made this video possible IMO. LTT forced to up their game
@@minijag972you really think that it's genuinely him holding a grudge between that? Linus is not an editor for it 💀
I love seeing this kind of testing across multiple channels.
And another shining example of why you should always check across multiple review sources.
(Loved the Pokemon referencing of the processors)
(and I've only known a couple years now that Raikou is long o sound (ish). And i have the gbc Crystal bundle. Best Christmas ever, lol)
As PhD researcher, this is my favorite video that you guys have published in a while. Love the data and scientific process used here guys. Keep it up, go labs go!!
Waiting for the first LTT Labs paper to hit Science
It's great to see. All this work will make sure their internal comparisons are reliable. You can look at all LTT Labs testing when you want to compare DIFFERENT products for performance and know it's just the products you're looking at that vary, then look across different reviewers testing for the SAME products to get a glimpse the actual market quality variances for that one product. LTT Labs is actually benefiting the whole techtuber community, not just LMG.
lol "PhD researcher"
@@chairface1859yes, one who researches PhD candidates 🧐
@@chairface1859 now you look a bit daft. See working in a University I know this is a proper job title. Just because you have little understanding of job roles within a University doesn't make a job title any less real. I have a very unassuming job title I work within the Estates/Operations department of the University that I work at my job title is Building Support. But dig a little deeper and Im actually part of a team that run three of my uni's flagship research buildings one doing research into neuroscience ( like finding a cure for dementia Huntington's) and that has the Compound Semiconductor research Institute ( yes we have the ability to fab our own silicon wafers) also in the same building is the Chemical Catalysts Institute. This build brings together the school of chemistry school of physics and school of engineering. The final building houses the Social sciences with departments within it that research help form public policies which work with The British Government also the business side of the university. Out of the 5000plus staff at my university about 1000 are academics
you can maybe work around the inconsistencies in red dead 2 by using cheat engine with the unrandomizer, it forces the random number generator to always return the same values, so you could end up with a deterministic benchmark run :)
that's a great idea, do it for every game if possible or matters
I don't think they can automate it
I came down here to check to see if someone had suggested similar.
As a former game dev (not for RDR2): This will have a minor impact on performance of games that utilize system random number generators especially (because of the syscall overhead) but importantly the result should be changed the same (incredibly tiny) amount. In exchange, you will get drastically more consistent runs from the behaviors in game. We actually had a debug build we did that replaced all the calls to random things like this with static values, so that we could apply internal test harnesses, especially during building our tutorials. I am not intimately familiar with how Cheat Engine goes about derandominzing, but all the sane ways I can think of it would do it, give you more in repeatability than you lose in cost to run it, or in deviance from player system behavior. (You're simply selecting one player system behavior and repeating it)
@@Deveyus this may be a stupid question, but would the (albeit minor) performance difference be mitigated if you leave in the random number generation but just don’t use it and use the static value instead (therefore still creating the “load” of generating a random number while also keeping your consistency)
I am not sure if a result gathered with the default randomized behavior turned off is representative of the average gamer experience. Since the goal of the labs is to provide statistically significant testing for the masses, introducing irregularity is maybe not the right solution.
I'm glad to see linus and his team working towards doing better, especially after that whole thing where they were admitting lack of accuracy and that they failed their community. Seeing them be so passionate about this really brings a smile to my face.
You say that like they're the only ones that don't have perfect accuracy, You say that like putting in the tens of thousands of dollars in equipment and man hours is something that they can just do overnight, You say that like gamers Nexus was at the time he made that disgusting disingenuous video was in the habit of retesting products to use that data in comparison charts for new product launches. But no, none of those things are true and I still don't think gamers Nexus retests. Although if you can provide me with real evidence that he does retest stuff I will happily change this opinion
GN doesn't have the money to do real hardware testing at any kind of scale anymore. Steve himself admits that they put a huge deal of money into the "investigative journalism" they do. When most of his sources we just have to take his word for it because "they cant be revealed" which is fine. Its great that he focus hard on shit companies doing shit things. The the recent EKWB crap, Yes they are garbage, yes the management is terrible and they deserve everything coming. But be honest the only reason his Tech news is successful is because it gives people someone/something to hate. Also, the one thing in common with his investigative videos? He always contacts the company for comments prior to the video even being released. Did he do that with LTT...No. He took what he could from LTTs videos and forums and put them on blast for people to hate, gaining him views. The world today thrives on hatred and in some ways Steve is using that for his own gains. So yeah everyone on the internet, LTT or GN play on peoples emotions and thoughts to get your views. None of them are innocent. but also none are guilty. If LTT failed so did GN. One for testing data the other for using hate to garner views.
This is absolutely what Gamers Nexus loved to see LTT's testing become, even though LMG seems forgot to mention them at 17:24.
(It can be just oversight, but not including Gamers Nexus, the most through on testing tech reviewer before LMG have the Lab, seems a bit intentional.)
The issue is that GN isn't trustworthy.
@@Hathos9how, cause they called out your best boy linus? go toutch grass please
@@martine5923 That is the issue. GN has a legion of toxic sheep that love drama and attack whoever GN says to. I like technology, not drama.
@@Hathos9 that was very funny, please continue. People who love tech-not-drama don't watch ltt much, the whole point of this show is ~ 70% entertainment 30% info. It's basically a discovery channel tv show talking to the lizard brain off its TA. Nothing bad in it per se, some videos are pretty entertaining, but that's what it is primarily - a tech-related entertainment business.
@@BoraHorzaGobuchul I'm fine with entertainment/tech. It is better than the non-stop drama and negativity of GN. LTT seeks views by making fun videos. GN seeks views by attacking others.
Well done. Possibly the most thorough and well-explained video on the effects* of the Silicon Lottery I've ever seen.
*Not covering much on the causes, or it would've been a two hour video!
In fact, can we have an updated video on that please?
It's a 1 minute explanation:
Each transistor within the silicon will be electrically different. Differences in needed gate voltage (what vCore is) for current to pass, differences in leakage (the amount of current loss through electron migration, aka heat), differences in output capacitance (the inevitable result of having current paths in parallel), etc.
With these differences, we have a different required voltage to drive the transistor to operate properly given a clock speed, different heat output given a voltage and different power usage given both.
Each modern clockspeed boosting algo for Intel and AMD take these into account to be both stable and maximize performance (aka, clockspeed) given the current load on the CPU.
So with all of that in mind, you can see why CPU A and CPU B will perform slightly differently given the same workload. Their clockspeeds will differ because of power, voltage and temperature differences down to a transistor level.
@@winebartender6653You're 1 minute explanation assumes a fairly deep level of knowledge about how a CPU works and seeing as LTT's target audience is not scientists and engineers where these kinds of assumptions could be made, your 1 minute explanation with likely turn into a five or 10 minute exclamation if they were being insanely information dense, at the average information density of an LTT video, I would say the hour-long prediction is not unreasonable given all the other stuff they would have to then cover to make it a complete video
@@the_undead ? Absolutely not. A simple preface of "A CPU is made up of billions, or even trillions, of transistors, which are essentially tiny electrical switches". You don't need to know anything about voltage, current, capacitance or anything else that you think is a low level concept.
The point of the 1 minute explanation is to touch on the fact a transistor will always be different when made in a CPU, that's it.
You could absolutely expand it out, but it is unnecessary to get the point across
As a professional test engineer, I am truly impressed with the level of detail, thought and effort that has gone into the LTT Labs.
👏👏👏👏
Now if you could just coming and explain this level of dedication to some of my colleagues in other disciplines of engineering that would be great. They do seem to think that everything will just work, and don’t appreciate that us test professionals have to think around corners sometimes…
@@terrylyn what's your issue?
@@terrylyn yeah its pretty terrible
Majority of the population do engineering for money, not because they are overly passionate. It would be useless to explain
Your comment is not constructive and is worthless. Please tell your own brilliant idea to improve the testing methods. Else don't type anything.@@terrylyn
What's pretty Terrible about it? @@FlagerMiszcz
Superb work! The only thing missing was repeating the testing on another motherboard to check if the CPU performance depends on individual CPU or the combination of motherboard power delivery and CPU. Would the same CPUs perform the best in any motherboard or did those CPUs simply happen to work the best with that specific motherboard?
I never got lucky in silicon lottery but once. That was an Northwood P4 that can be pushed from 1.8 to 2.8 Ghz. For RAM I'm usually happy if I get the values printed on the sticks to run stable.
Always excited for labs stuff! That was exceptionally well presented and explained. Really great how you adressed inconsistencies and the multitude of possible causes. Probably one of the best ways I've seen to explain testing methodology, the reasoning behind it and the resulting discrepancies when compared to real world applications. Loved it!
The thing I'm not sure about, is why they couldn't just completely fix the clock frequency to a set value, instead of letting the CPU run AFAP. Sure, you won't get the best performance out of it, but if you want to use them to test a GPU, then it shouldn't be part of the bottleneck either...
HUGE respect to all members of the team. It must have been really exhausting finishing all these tests.
It is mostly automated at this point.
How many months did they work on it?
@@sierra2632 it's like you are a naive who watch cringe linus instead of gamer nexus
@@AK-tf3fcYou say that like gamers Nexus is dramatically better, does gamer is Nexus upload videos showing their processes that they used to get the performance specs. Has gamers Nexus spent over a million dollars on scientific equipment and learning how to use said equipment to get better more accurate numbers, I don't f****** think so, feel free to say he makes more entertaining videos all you want, because that is an entirely subjective point. But the purpose of this video was never entertainment, It was to show their processes and the struggles they are having getting data accuracy to a point where they are happy. Where as far as I can tell gamers Nexus doesn't even care about accuracy
@@the_undead gamer nexus video is educative while linus video is entertainment only. Not to mention only one of them exploit people and sell their merchandise
6:50 Smiled when I saw the only thing you add is Notepad++ because this is truly the only thing I miss since going from Windows to Mac on my work computer...
Notepad++ is great for it's autosave and multiple tabs, though now the regular notepad (WIndows 11) has the same feature🤔
Same lol.. vscode isn’t nearly as good
@@Seedheh really?
@@Seed sure wish np++ had support for practically any language.
Vscode is way better the only thing notepad++ does better is beeing a bit faster@@Seed
Thanks for the breakdowns; appreciate the work, team! Can the audio tech please level the volume on his expostulations? I'm listening on headphones because others need silence.
When presenting Euclidean distance data you can use a dendrogram to present the pairs of CPUs that are closest to one another. Might help for intuitive visual interpretation of the data
Or using something like PCA plots or tsne bonus points for showing the audiance how to read scientific plots made for this kind of data ^^
@@frederikex4545 This, or the new kid on the block - UMAP. Makes graphing any kind of multidimensional data a breeze
@@moonik665 or their older more linear cousin, tSNE
@@frederikex4545PCA also would potentially highlight which benchmarks/games have significant impacts on differences between performance. Add in a clustering algorithm like DBSCAN that can catch outliers and you're set
This guy diagrams
The silicone lottery strikes again, It's crazy how much variation there can be between CPUs, especially within the same model
Yep, Linus dose all these videos just to keep the best stuff for himself.....You think the stats is for us but it's not.
Disagree, it s more like: it s crazy how good the production process for such delicate tech has become that there s so little deviation and the results being so insanely close!
@@AdventuresOfDetroit l o l
Theres not Much variance its +-2%.
especially for more bottom bin chips, since the premium ones are all reserved for the higher end chips
1:00 yes, silicon lottery, btw it happened the same with 5K series with AM4, they simply overclock themselves until 90c
Thanks, enjoyed the detailed description of the testing methodology.
Inb4 copyright strike by Nintendo for using Pokemon names
Nahhh, Linus's 4D chess move of that Corsola pronunciation will evade the Nintendo ninjas.
They're currently overburdened with Palworld, so nows the time to do this.
Nah wouldn't be a copyright strike. A trademark violation cease-and-desist maybe though!
@@XxZannexX😂
nah nintendont is busy with palworld this month
Loved this video as a statistician, and really like the approach you’re taking. It may be a bit much, but if you did an equivalence check pre/post for any gpu lineup you’re reviewing, that would be pretty solid evidence that any differences you found in the tests across the gpus is due to differences in the gpus.
The other option would be to model the specific test rig in the regression, but then you’d need to put each gpu in each test rig, which would defeat the purpose of parallelizing the test in the first place.
This is the type of analysis I'm looking for! Very good experimental design and good presentation of these results. Are you going to publish this in a peer review journal?
been looking at the labs webpage the last few weeks wondered why so little was there expected there must be limitations on what youve been working on looks like im right glad yall are taking the time to do it right!
(Tech horror story) There's an Azure server out there with a haunted CPU. We had a lot in the cloud, on one cluster we always had more errors with exactly the same deployments. We got around it for months by updating to a new cluster. Just a few months later the errors were back. I think that the old compute server got moved to the new cluster when the old cluster got decommissioned. And to this day.... I moved to ARM. Just because i know that x86 box is going to get me.
Change to a different availability zone?
@@Jimmy_Jones company policy had reasons I can't remember for keeping availability zones the same. (I think there was a pre purchasing compute agreement ) Fortunately I moved companies and it hasn't found me yet.
Try getting some techpriests to help
@@duckbilldaniel"it hasn't found me yet" lol
Would it be more expense than its worth to run a profiler on the containers to check performance? Can your company seek compensation for being placed on faulty hardware?
11:53 Colours on 1% low and Avg are inverted
Gamers Nexus about to make a third rant video
Triggered my OCD - Thanks LTT.
From collab to no mention at all... Where is GamersNexus?
It look like you missed something 😅
the saltmine runs deep at LTT
A histogram could be a useful way to show the CPUs side-by-side. Perhaps 11 was too few to properly flesh out a histogram, but I know that I struggled to look at the column charts and really get a good feel for who, if anyone, was an outlier.
Oh boy, people are going to be loosing their minds again for the average 1% differences.
@@jake20479I have a 7800x3d. Am I the minority here for buying a chip, and if it runs my programs and games at an acceptable speed that I wouldn't notice a difference unless I lined it up with 11 other chips, that I'm just happy with my purchase and enjoy my games?
(Run on sentence, I ran out of breath trying to read it. I'm also not fixing it though.)
@@jake20479 you know that that is basically impossible, in almost every industry. no one can build 100% the same product every time
@@jake20479 Except you'd be comparing that 'any percentage less' to reviewers who most certainly will not share the same test bench, software variables, temperatures, humidity and several other variables.
So, this is good for LTT to keep data similar, but for your average consumer, the rule of 'within margin of error' very much applies.
@@jake20479 I mean, I'd much rather a cheap cpu than a consistent one. Do you have any idea how expensive it would be for chip makers to get their consistency down under 1%? I guarantee whenever you buy anything at a gorocery store theres at least a 1% variance to someone else who bought that same thing, it's just pointless to make stuff that consistent.
@@jake20479if you want the best you buy the best. We all know they bin the chips, we all know they don't all turn out identical. This has been the norm for decades.
Throw in a benchmark running under a minimal Linux install for good measure, will give a good indication of raw performance without many background tasks.
How many of us game on Linux ? They run tests under windows because that's what the humongous majority of us use to play
@@thegeekno72 But they remove a lot of the bloat anyway, they try to get the raw performance of the CPU's. Not the performance of the CPU on Windows.
I'm not sure why they don't use Linux, maybe they've tried and something didn't work out?
@@rigf1997 Linux raw performance and Windows raw performance are not comparable things, they're software that have wildly different operation, it could be analog to comparing X86 and ARM, it's just not the same in the way it behaves
@@thegeekno72we are talking about cpu performance variance doesn't matter which platform. linux might have given better consistent results because of less bloatware. Regarding arm and x86 they are different cpu architecture you comment doesn't make sense.
@@rigf1997linus said that he wants to be relevant for the general user and that pretty much explains it all. Yes maybe using linux will give interesting results, not gonna lie but things change once you change use cases and os and the difference might become minimal for general users tasks.
crazy to think that the 1-2 % sillicon lottery difference could be the power of thousands of old 6502 cpus
lots of interesting info about how 2 of the same model cpus can perform differently, i do have 2 main thoughts though
firstly, if the end goal isnt to test the cpus themselves, but just to have cpus that perform the same to test gpus, would there be any value in setting them to a fixed clock speed of say 5ghz rather than letting them turbo? similar to the point about older cpus/ram, would that not remove the factors of how willing a particular unit is to turbo (and maybe also remove factors like the thermal compound application)
secondly about the point of comparing to automotive industry, i do agree that there could be some benefit in an official standards agency, but im not sure it would help too much with the issues described here, those official tests that are performed on cars are generally only checking that the car as at least as good as some regulation/manufacturer claim, cars dont fail the tests because they performed better than the manufacturer said it would, and generally i dont think consumers are going to be upset if the product they purchased is better than the manufacturers claims (whether thats a car or a computer). tbh i think the reason its not actively regulated is just because the stakes really just dont matter in the grand scheme of things, the consequences of how a car performs on different road conditions or in a crash is literally life and death, how many fps a gaming pc can deliver probably isnt even life and death for the virtual character you are controlling nvm any real people.
This is industry defining data. Genuinely. Anyone notice at ~9:50 when the charts fly by, that Corsola goes from being in the top 3 CPUs for cyberpunk at 1080p, to suddenly being the worst CPU in the test by a wide margin at 1440p. I expected different games to stress the CPUs in different ways, but I did NOT expect a single game to completely change the order of a single CPU. I guess this goes back to the performance controversy with Halo Infinite that Hardware Unboxed covered, where they found that the specific part of the game you benchmarked could completely change the order of the top performing GPUs. I suppose the same logic applies within the same game, for each resolution. That's crazy.
Also, good catch with CS GO. Showing the differences between CPUs is great, but at ~700 frame per second, a microscopic difference in latency has an outsized effect on the FPS value. That's because the time between frames is actually the inverse of the FPS count. Which is fine when you're roughly around AAA game refresh rates between 30 and 120, but that inverse relationship starts skewing your data more and more the higher your FPS number is. High FPS games always have a larger impact on average performance, just because of the way math works. I'd like to see someone normalize using "time between frames" to measure those differences instead of the actual FPS number. That would probably give a number that better represents how different it feels to use two CPUs. It would also give more weight to games with lower FPS values, which is actually where performance matters the most. Almost no one can notice the difference between 400 or 500 FPS, but people can definitely notice the difference between 40 and 50 FPS, even though the "percent difference" is exactly the same. Absolute values matter, and the percentage difference calculation everyone uses can obscure very important data that actually matters. When I'm looking at buying parts, I don't care if a CPU gets 5% more FPS in a game that's already maxing out my monitor's refresh rate. Worst case games generally matter more, since the esports titles with high FPS counts are always going to be easy to run.
People dog on CZcams comments... But then sometimes you get good ones like this.
Uhh... Except, because of the way math works, the inverse relationship between FPS and frame time doesn't affect the percent difference at all, and regardless of which way you show it our perception is a logarithmic scale.
ie. The difference 40 to 50 FPS is 25%, and the difference 400 to 500 FPS is also 25%. Invert that and the difference 20 to 25 ms is still 25%, and the difference 2.0 to 2.5 ms is even still (surprise) 25%.
@@Renegade605you're getting it but you're not getting it. Say an anomaly increased latency in testing by 2.5 ms. The 20ms case would increase to 22.5ms, i.e. 12.5% and the 2ms case would increase to 4.5ms, i.e. 125%
@@ridwaan2443 what you just said is the exact opposite of the point OP was making.
You're correct that a 2.5ms hitch will feel much worse at 2ms average frame times (400 fps) than at 20ms, except that OP said small differences in frame rate matter more at low average frame rates.
Both are true, but for very different reasons in very different cases.
You just can't simplify the data that much and at some point you have to trust that your audience is capable of understanding when and why the different numbers matter.
Getting "um acktuallyed" in youtube comments by someone who repeated exactly what I said. Perfection.
@@Renegade605 For reference, I said "the percent difference is exactly the same". That's a verbatim quote from my unedited post. Check yourself before you wreck yourself. We agree, so I'm not sure why you're telling me I'm wrong about the percent difference.
On my other points, I'll clarify. When I mentioned "High FPS games always have a larger impact on average performance", I was specifically talking about average performance charts like the one at 8:37. Charts that take the FPS values from each game, and average the result. That's a graph where the inverse relationship skews the data. Specifically, it heavily weights the graph towards the result of the game with the highest FPS values, and reduces the weight of games with low FPS values.
You can compare the graph at 8:34 that included CS GO and the graph at 11:41 which excluded CS GO, to see how much a single outlier game affected the entire average result. That inverse relationship definitely skews the results when you average the FPS values between games, as basically every reviewer does. That's a big problem, because an extra 3fps in CS GO mean basically nothing when it's already maxing out your monitor, but 3fps more in cyberpunk at 4k is a pretty big change. However, both those changes are averaged out on the same graph. That's why the inverse relationship is a problem. Rewatch the 10:43 "CS:GO is Wonky" segment to have Linus explain why games with framerates that high aren't valuable for benchmarking. There's some CS GO specific stuff there, but there's also a lot of statements that can be generalized to all high FPS games.
If anyone is still confused, I can clarify further.
One of the best videos you've made in a long time. I really like the labs content and the transparancy that you're trying to give. There is a bunch to learn about testing to learn in this video. Really looking forward to more of this content.
Could be fun to see how things perform under a veryyyy paired down linux install and running the games/tests in wine/proton so that you can have near 0 background tasks and just the game running see how close things get then
Hey Euclidean Distance! I remember that gem from my grad school days. We used it for satelight to classify pixels as land types/change detection
This video feels like it would be great for teaching high school science students about isolating variables and the impact it can have on results! Such an interesting video!
As someone who used to do statistical process control for safety critical electrical components in the automotive industry, I approve of the comprehensiveness of this. Now that you have the process laid out for doing this, it will only get faster and more routine.
you can isolate a lot of the problems with pbo by setting the clock rates and voltages, then you can measure a lot of the inconsistent impacts of software (OS and benchmarks) and then you can give a reasonably confident statistical deviance due to those factors and find the actual impact of binning
really enjoyed the testing here! Silly question, but why not just run the CPUs at a fixed clock speed for GPU benchmarks? I’d assume that’d resolve nearly all of the hardware related variability across benches. It seems like an easy enough thing to implement. I’m pretty sure GN uses a locked Intel CPU for their GPU benchmarks.
Respect for the Vegemite shout out.
Love, Australia
expect thats not vegemite ,real stuff is black not brown
I just imagined Steve in the frame looking over his shoulder critically all the time.
... and Gamers Nexus wasn't among the "multiple outlets". 🤔
Gamer’s Nexus used to be referred to regularly when Linus mentioned other outlets (check old wan shows).
Steve repeatedly set fire to the bridge that they had. The exposé was when he took a wrecking ball to it.
My personal largest gripe:
Steve took issue with LTT saying in a video “Our numbers seemed completely wrong, so we checked in with AMD to make sure there wasn’t something screwy”
Then, in the next AMD Card review, Steve says “We had weird numbers in this benchmark, so we checked in with AMD and received this response”
I mean I like the GN content generally but I can see why LMG wouldn't exactly advertise that channel after some of the less than comradely stuff they pulled.
@@tappy8741 well if you literally got the other guys phone number I would have at least shown them the courtesy to make a call to make my point before pouring gas into a public shitstorm but maybe that's just me.
kinda funny that GN literally did this video a couple months ago, with derbauer, and actually hardcore overclocking, and got pretty close to the same variance... and they were all just like "yup, seems about right" rather than "government pls help, my test bench isn't as easy as I want it to be"
@@pitmaster226 yeah, that's a fair point. I get both perspectives. I hope they can work it out in the future.
In comparison of the results in some sports - they get also scores from the jury + the results itself for example in ski jumping. They always cut the best and worst Jury scores out to prevent bias/prejudice.
I have personally experienced this with the machines I have built as a system integrator. On top of this, I have also seen varying degrees of stability and ability to PBO.
8:31 honestly really disappointed to see that diabetes didn't win out over the other CPUs. I had my bets on this one :(
Well, it makes progress in the real world tests.
I loved how much meme-age they worked into their BSOD in the intro. They even got the QR code go to the LTT store lol
Absolutely no surprise re marketing included.
I think it would be great if you would take those worst performing samples and try to bring them back to stack by changing Pbo settings. If they are falling behind due to thermals and thick IHS, it might be a good tuttorial for people that ended up with preety non - gollden samples.
I mean that even if particular chip might not perform well in stock, might be better for OC or maybe will be a king of efficiency and allow for a high voltages drops? Even if somebody is totally unlucky, maybe at least you will come up with a few setup tricks to increase it's efficiency.
You could also use other similarity measures like cosine similarity or Jaccard distance. You may also consider trying different norms. ;)
Awesome video! Lab is gaining a lot of credibility after delivering such great and in depth analysis. Impressive!
I must admitt however that is a bit sad to see at 17:24 no mention of gamer nexus.. I guess the community would love to see a reconciliation... I'll keep dreaming
The ball’s in Steve’s trunk.
“If an article contains personal or serious allegations or claims against an individual, it may be appropriate and necessary to give that individual an opportunity to respond to these claims, or to deny them if they wish”
- Independent Press Standards Organization
But they’re UK based, so why should Steve care (:
yeah, a bit of a shame. especially since GN, HUB and Der8auer did a video on this exact thing
czcams.com/video/PUeZQ3pky-w/video.html - GN
czcams.com/video/dGbW7orZS-A/video.html - Der8auer
I noticed the GN omission too. Really sad to see.
Amazing work @LinusTechTips!
Your discussion on sources of variability really got my brain going.
As someone who works with psychological data, this is something we often discuss (inter-trial vs inter-participant).
Have you looked at using linear mixed models (aka hierarchical models, aka multilevel models)? They could allow you to control for multiple sources of variability without having to aggregate data so much.
About the car-industry comparison. I'm not sure if the car industry is much different. I do believe that there is some batch-to-batch variance in engines etc. The quoted horsepower can differ, and some manufacturers provide an underestimated specification to the customers, such that any car is guaranteed to meet it (same as with the CPUs). I think that there is no problem as long as the quoted specs are met. If we get into supported memory speeds vs. the overclocked memory speeds that are sometimes used in their marketing, then we run into a zone that I personally like much less.
I’d say the variance is actually higher in autos. Everything is tolerances, and a piston on the lower end of it’s accepted tolerance and the cylinder being on the high end would mean a slightly larger than normal gap around the piston, which can introduce some oil burn if the car is meant to take thinner oil. This isn’t a huge deal at low performance, but high output engines can end up with problems from something as innocuous as this. Hell, a piston being at its upper tolerance and the piston head being as it’s lower tolerance could increase the risk of seizing.
One way to think about it: 99% of V6 Toyota 4Runners in the late 90’s were perfectly fine and even famously reliable, but some minor flaws in the design in that specific era of V6 meant that, if you were unlucky with a marginal tolerance engine, you could end up with cooling issues that could blow it up.
Kudos to the use of euclidiean distance in determining similarity. Other measure of similarity can be used is cosine similarity, jaccard, etc. Its nice to see LTT actually doing Data Science work. Maybe could use Principal Component Analysis (PCA) to see which of the features is the main contributor to the difference between the highest and lowest rated CPU's
Yeah after you explained it I am not nearly as mad about this as I thought I'd be. If they design it and manufacture it to achieve a standard, but include features that may be able to get even MORE than advertise, then I don't really care about the variability of how much more it gets.
It's kind of like rock climbing gear, where a carabiner will be rated to 22kN of force. 99.7%+ of the time it'll be stronger than this, and it could easily vary from 23kN to over 30kN. That's be like a CPU having a performance spread of nearly 40%. The thing is, the lowest number of 22kN is still 5x stronger than any realistic safety situation, and so maybe we should think about CPUs in terms of the demand we expect to place on the. Eg. "This CPU will have a 99.7% chance of running *game* at more than 60fps/144fps/250fps"
For your testing, maybe Linux with Wine/Proton would be better in terms of minimisation of background tasks?
The lack of oversight vs automotives and, indeed, rock climbing gear kind of checks out because if your CPU is 5% slower, you aren't going to find yourself in an accident where your brakes fail and you, your partner, your two kids and another random family all fucking die. It is definitely worth doing in a decentralized, nongovernmental fashion though. Maybe a consumer union could be made where manufacturers have to provide representative samples in order to get a trademarked seal of independent testing.
Yeah I think you are right. There are margins in probably every industry so I think the consumer has to be fine when the product reaches its rated performance. When we talk about cars for example, there you can often measure differences between the power of engines. But then the average consumer doesn’t care about fuel quality, tires and regular oil services that much, so probably those have more impact on the actual performance on the road than the engine itself. The same is probably true for pc part’s performance in different countries with different average temperatures. So in the end I guess you should buy a product by it’s confidential rated speed and be happy if you get more than you paid for but not be necessarily sad if you don’t gat that much of a benefit. Your friend might have ended up with a better chip itself but maybe he has a low quality power supply, which makes the cpu perform worse than yours. Or maybe there are also margins in psus?
@themisterx8660 the margins are in EVERY industry. If it's not federally regulated, it's financially regulated or consumer regulated... excerpt vitamins and supplements...
the problem with using Linux is that it's just not a realistic OS to use for benchmarks. Linux has many problems working with NVIDIA GPUs, but the main thing is that most people just don't run Linux so it's better to just use Windows, even if it's bloated and makes getting reliable test data harder
@@memethief4113 True, but it can be used for non gaming tests. As it stands out, Linux will have to be used in specific tests. Like you said, having Linux be usable for ALL tests, we might get there one day, but not today. Not this year either. Maybe next year, in the most optimistic timescale, but realistically I think it's more 5 years, IF it continues to grow like in the last 2 years.
@@memethief4113 I guess the argument would be that it would be fore controlled testing. If one chip ran a game with proton on linux at 100fps and another at 105, then you could meaningfully say the second is 5% faster, even if on people's actual systems the real world framerate would be 45 or 145 or whatever.
The problem with testing in realistic scenarios is that realistic scenarios have a lot of uncontrolled variables. If you tested on windows you could get 145fps or you could get 95 depending on what windows fuckery is happening in the background and you couldn't know without doing 100s of tests and averaging it. Also Linux gaming, even with NVIDIA, is my personal main use case :P.
Anyone notice how they didn't show Gamer's Nexus when mentioning "other outlets"?
I noticed a bunch of channels that I sub to that they didn't put in there. Very angry right now.
Not surprises though after the nutpunch GN delivered to LTT. Eventhough it made LTT force to do beter.
I literally scrolled to find this comment as soon as I saw it :P
Lmao. Yes. Very clearly on purpose. I knew he wouldn’t show it as soon as I what they were saying. Hilarious
TBF, GN also isn't doing the same exhaustive kind of testing in the same space. Their niche, and by definition the type of testing they do, is completely different.
Where do you get those cool little cpu holders from?
Linus, how about checking for games on which cores they run and on which cores other tasks(background things)run?
It can be that in one single CPU there can be better or worse cores inside the chip.
Also if game latches to a core that is occupied with GPU interrupts already performance can be degraded a bit.
Usually the system manages threads and cores well enough but there can be flops in that automatic optimisation.
Love the focus on consistency and variance, and would love to see error bars on future comparisons to quantify this!
17:30 haha GN conspicuously missing
Noticed that. Also, GN just did this exact same video a few months ago.
Sad to see they clearly haven't buried the hatchet, but I can't exactly blame them. Steve wasn't wrong with what he said, but he clearly took a huge swing at Linus' reputation without seeking comment. That comes across pretty malevolent to me. Linus' lab guy who trash talked GN appears to be the instigator, but Steve's response was not proportional, nor particularly professional IMO now that the dust has settled.
@@TheVillainOfTheYear I agree about everything other than Steve not being wrong. He was absolutely wrong, and was stirring drama for both views, and to undermine his main competitor. He didn't want solutions, he wanted to gain market share. That's integrity in the same way as Apple criticizing Google.
Nice test and as far as i can tell statistically sound... But I think it would also benefit from a raw benchmark without operating system influence to establish some kind of base line. After all, you're comparing CPU's in the first place and CPU's running applications in the second place - at least in this specific test.
Quick question about the testing methodolgy:
You guys pointed out that you've disabled GameBar for Lab's Testing.
How do you then account for CPUs like the 7950X3D that need the GameBar to function correctly. F.e. the 7950X3D assigning tasks to specific cores through the GameBar?
We aren't using the 7950 X3D. This process is for these benches. - LS
17:24 Kinda weird not seeing Gamers Nexus... wonder if that relationship will ever recover.
I was sadden by that too. It would have buried the hatched and allow everybody to move forward. But it's so hard to have nice things today :(
It makes sense, they have no obligation to mention GN and given what happened it's understandable
@@Winnetou17 GN pretty clearly used the situation and drama to drive traffic and profit and weren't entirely honest or genuine in their handling of it. LTT denying them further attention is only appropriate.
@@waldolemmer It keeps filtering my comment, they likely have keywords in place to prevent discussion of it anymore, but there are videos out there with analysis of it, one rhymes with mechmechfotato.
@waldolemmer (We'll see if this goes through) As an example, they were dishonest by not actually giving an unbiased or full picture on the events they were reporting on, they did not contact LTT to get the full picture, and presented his one sided view as fact, then he refused to respond to further discourse on the subject, claiming to be holier and denying any poor handling of it on his part.
as someone who works in big data and analysis of it the way you guys described all your statistic analysis in a way easy to digest for regular people was amazing 10/10 vid
How about adding Error Bars to your results?
I wonder what the average tempurature was for these cpu's, as that can have an effect on clockrate and voltages being applied to the cpu's.
Love the update on testing methodology! Very detailed and I do appreciate that! I did notice that Gamer's Nexus didn't get a spot in the B-roll alongside the other creators, I honestly hope you two make up and I strongly recommend looking to them every now again for sanity checks because they are really great and highly respected.
I noticed that as well. The Steve's are my go to reviewers for almost all purchases on parts
I lost all respect for that pile of garbage when he made that hour-long video, as someone who knows a great deal about human psychology, it is obvious to me that that video was not made for any reason other than revenge. And more specifically revenge about an LTT labs employee calling him out specifically on his data testing procedures, more specifically calling into question how accurate gamers Nexus testing is. And seeing as Steve's response was that hour-long video instead of an hour-long video detailing his processes or the updates he's made to his processes, I'm going to suspect those same concerns are still valid today, and I feel this is partially confirmed by the lack of gamers Nexus in that b-roll shot, Yvonne and Luke would not allow Linus to hold a grudge for this long, also, linus isn't even the CEO anymore, so I suspect there was a reason other than the controversy for the lack of a gamers Nexus in that shot
@@the_undead "as someone knows a great deal about being a fanboi"
If my firm would be slandered like it happened I'd do exactly the same. And oh btw to clear your delusional POV even more... Surely LLT just issued apologies and shut down testing because there wasn't any truth about the claims... surely! If you fire a gun you better be goddamn ready to get backfire. Back to School or where ever you gained that dubious knowledge about human psychology.
This is some incredible research. Not many, if any review channels that do testing would go to this level of effort to obtain the results you have here. I think the amount of effort that it takes to obtain these different samples is probably a reason why. I have always heard of the silicon lottery but did not think about that the lottery may be based on the piece of silicone being used were the environments of the factory. As an afterthought it seems obvious but of course you would need to obtain units from different manufacturing runs to get samples that would perform differently. I appreciate the effort that it took to do this and this demonstrates the true value of The LTT lab.
You scaled distances? According to Mahalanobis distance? Or how did you determine the correct scaling factors?
It's soooo cool to see you guys using math that I learned getting my bachellor in system information. Keep up the excellent work!
I love this kind of videos when you show the behind the scene of the lab.
This is the most literal "computer science" that could be done. ^^ (Usually, "computer science" is not about computers as Edsger Dijkstra once said.)
As a totally unsolicited advice: I might just suggest considering the Chebyshev distance instead of the Euclidean distance since you are interested in avoiding having a sample be an outlier in one of the tests. (Other high-order Minkowski distance would trade the exact rejection of outliers with some "average closeness".)
Gamers Nexus doesn't seem to be one of the featured other review outlets, odd.
Maybe also measure the power consumption for each CPU. Der 8auer did that and it was very interesting.
Possibly the most thorough and well-explained video on the effects of the Silicon Lottery I've ever seen. The only thing I found missing in the video was the thermal performance of these chips and how they maintain their clock speeds. I have been there when I compared my Ryzen 5 3600 with another R5 3600 of my friend with the same mobo, RAM, and Storage. The difference in thermal performance and the clock speeds was noticeable... that was the day I realized silicon lottery is a thing too.
Great job guys! It would be nice to see another video the same detail on the latest 13th gen and 14th gen Intel CPUS
If you ever name something similar to like you did with the Pokemon, I would consider doing it alphabetically. So the first one starts with A, second starts with B, third starts with C and so on and so forth. It's just one of those nice-to-haves that doesn't change much but makes the presentation a bit easier to digest
10 years watching LTT, it keeps getting better! Great job team
Something worth thinking about is adding measurement uncertainty into the graphs
That is what makes synthetic benchmarks more consistent and deterministic. But are less relevant to real life performance which is not deterministic.
Freaking amazing video, well done everyone. It would be cool to see if the results are still as varied if the clock speed was locked across all of the cpus. I assume this wasn’t looked at because that’s not as equivalent to the real world use of the chips, but still could have been an interesting add.
I know this was extremely time-consuming and detailed, but it gives me confidence in watching your channel and the results you obtained. Just want to stop and appreciate the hard work everyone is doing at LTT.
What if you ran multiple benchmarks synchronously on the same CPU? if allocation occurs appropriately, it should be even, and not comparable to other tests that don't use the same format, but you could use a formula to guesstimate the actual performance relative to other cpu's.
honestly, HUGE respect for doing this, first of all, the video is hella interesting and second of all knowing that you guys do this is extremely reassuring. great job dudes.
@@boscotheman82 thank you! but I did not ask
I like how GN is not included in the other media outlets part
Subtle shade
Neither was jerryrigeverything, nor dawid, nor a bajillion other (including many major) tech YT channels/media outlets.
You might be over-reading between the lines
@tiagobelo4965 or its a valid observation of the absence of perhaps the most prominent hardware reviewers on the internet. He doesn't carry the moniker of 'tech jesus' for no reason.
Let's get the pitchfork, guys! /s
He's mostly called tech Jesus because of his hair. Just saying...
What about the differences between the gpu's? They also have golden samples
What kind of coolers are used for this testing? 50RPM +/- on a fan or a fan that sits 1mm higher could maybe cause this small difference too.
This is a good investigation! I've been waiting for someone on this investigation to come out because, as a system integrator for CAD pc's I thought I was losing my mind! Ever since the 13th gen intel and the X3D line that AMD brought out, I've been seeing these inconsistencies. And I was really worried that something in our testing wasn't right! Glad to see my sanity is in check... But I REALLY think you should also take a look at intel! The 13th gen was already a real 'silicon lottery' thing, but the 14th gen intels are... WAY out of spec now and then! PLEASE PLEASE check intel too!!!
", but the 14th gen intels are... WAY out of spec now and then!"
But in general that is not the CPU but the motherboards. Sadly intel allows the boardpartners to run insanely bad automatic overclocks as the default settings -.-
@@ABaumstumpf OK! Nice to know! But the thing is. If I swap the same cpu with the same cooler between different motherboards (of the same type that is, we use the b760m aorus elite a lot and the b760 tomahawk) I don't see as many differences as when I change the cpu's out... When I only swap mobo, I see a difference of about 2% max in cinebench. But between different cpu's on the same motherboard I can get up to 7% difference!
So, what am I doing wrong in testing?
@@LightningTechNL "So, what am I doing wrong in testing?"
Pretty much nothing. Getting an identical CPU in the same motherboard with Intel normally stays within 1-2% of each other, at least according to the 3 companies i know the hardware off.
If you see bigger differences then first thing to check would be temperatures and powerconsumption as many boards increase Voltage and thus powerdraw even without affecting clockspeeds which can lead to shorter boost-durations.
This is done excessively with Z-chipsets but i had seen that with B boards too.
Had built a 12600K for a relative (for broswing and looking at pictures :P ) and disabling all the "enchancements" and running it according to specs has dropped the powerdraw to sub 100W at full boost.
there are always some differences in how efficient each CPU is and the boards increasing powerdraw for no reason will exacerbate those differences greatly.
Very useful video. Interesting that you didn't list Gamers Nexus despite them making an even bigger 68-CPU test back in August.
Also noted that, too bad if they are not friends anymore
Linus' ego is probably still bruised.
@@ShapershiftIt's Steve's ego, he was insulted big time when an LTT employee on a tour criticism he's way of testing. That was started it all... 😂
@@skak3000 Dude, if someone criticises your work, would you do nothing? Well, he did, and he also proved with data and examples that actually his testing was better, and more reliable than the LTT ones, which were ridden with errors. Everyone knows Linus is a little narcisistic, he said so himself, and he apologised.
That said, I watch them both and other outlets to make my decisions, I just don't understand why people side with anyone in cases like this, research everywhere and make your own decisions. Watching only one outlet and trying to protect them like they are your friend is just the stupidest parasocial shit. Fuck GN, fuck LTT, they are means to an end. Stop thinking emotionally and start thinking rationally.
@@soul_slayer7760 People who say they are thinking rationally are cringingly hilarious - and peak irrational. Anyone who thinks they or other people "think rationally" are so ignorant and arrogant it isn't worth discussing.
About Cinebench and increasing priority - I have noticed that quite a long time ago that there is a lot of variance in the results and increasing process priority significantly reduces it, and then was always annoyed that no one was doing that, especially given just how much modern versions of windows tend to do in the background. IMO, with anything past windows 7 (and even on win7 I think it would be a good idea), setting highest process priority should be the standard for benchmarks just to ensure predictability and I was always surprised that no one ever talked about it (or, highest that doesn't interfere with the benchmark itself - I ran into cases where setting too high priority for example makes audio stutter because the process that handles some of that doesn't get enough cpu time).
Excellent video, very informative. Can see the work that went in this one.
Computer testing not being required similar to cars, is because for cars it is all safety related, your cpu being 12 percent slower than expected, isn't a safety issue.
Next time, for similarity measures, there are a lot more similarity measures that you should explore. For example cosine similarity or Pearson similarity. But to be honest, such a limited dataset could just be brute forced in a few seconds trying every combination
Exelente video, se valora muchísimo el esfuerzo en hacer pruebas tantas veces y transmitir la información de manera tan clara
you maybe able to remedy things by tweaking voltages in the bios for corsola but it's time intensive and kind of a pain in the ass just to get the same performance as other chips which are already more performant.