Its exactly this. People that make money on selling stuff dont like the results of abx testing because they loose money. We in the sound science and recording industry love it because we like to debunk the myths and snake oil crap.
You are correct. The whole area is replete with those who prefer to use the power of suggestion. There are, of course, subject aspects to music; it's an art form, but the same is not true when it comes to the technical accuracy of reproduction. I manufacturer X claims their extremely expensive loudspeaker cable is better than generic cable (of an appropriate gauge), then there's no reason why that can't be double-blind tested. It doesn't even need to sound better, just different would be a start. The same then goes for a whole host of things, like audio quality power cables, or digital signal cables, or network switches. That is not to say that there might not be strong preferences for those things that can actually make a difference, but in those cases, a double-blind test would show that people hear there is actually one. But for many components this is not the case, and particularly not for those that do simple, functional roles rather than, say, act as transducers.
Hi Paul. Don't get me wrong, but I think you have a wrong idea about ABX testing. In ABX you try to guess what the X is and you can listen to A and B any time. So you know for example that A is the old equipment and B is new. Then you can ask to listen to A and B any time you want. You can also listen to X which is either A or B and then your aim is to know which one X is. There are no tricks or fooling around. AB and ABX tests have different purposes. AB is used when you want to know which one of two has the desired quality. For example, which one of the two pieces of equipment sounds better. In this test, you don't know which one is which. ABX test has a different purpose that is can you spot the difference. Here you know what A and B are and you try to guess which one the X is. There is never a C involved or the person doing the test doesn't know what he/she is doing. Both of these tests are super useful in audio but they have a completely different purpose. I also completely understand people who don't want to test. I don't want or need to test some things. But for some things I want to see if I really can spot the difference between two pieces and this is where ABX testing is very useful.
i'm a total cheapskate so rigorous testing has value to me. The idea of spending a few hundreds of dollars extra on something I can't even hear in a controlled environment is pretty concerning.
If you don't feel comfortable with an ABX test, then that's because you're afraid of being exposed. And being confronted with the reality that you wasted tons of money on snake oil. If you can't tell the difference in an ABX test, then you can neither under normal conditions.
Double blind testing doesn't work? Just about the entire scientific community disagrees. And it is easily transferrable to audio. The reason why PS audio and other audio companies (Audioquest etc) don’t like it is because it is an existential threat. If ABX testing scientifically demonstrated that your kit "sounded better" then it would plastered all over your literature. If your expensive kit sounded better as claimed then ABX test it and show us the results.
You are just writing a lot of nonsenses. Paul very eloquently describes why it’s useless and why A/B blind test works fine. As he says it works for pharmaceuticals but that’s something else.
@@dieselbrodeur Two years and this is your response? (that the vested interest of an MD selling overpriced kit knows better than the whole scientific community). It’s almost as if these companies rely heavily on snake oil because they have no peer reviewed scientific ABX evidence and cant scientifically prove their claims - if they did publish such results it would be damaging to their business model. Buy hey you believe Paul has perfect pitch, can hear a pin drop from 50metres away and the golden ears of a dog. A 60-80 year old biological marvel defying the laws of human aging, if you will… deary me.
@@dieselbrodeur The real reason why they do not ABX test is because they cannot objectively demonstrate their subjective claims. Its almost as if there is a business model that relies heavily on snake oil to justify the price tags. Overpriced cables, mains conditioners and plethora of other items conjured up are a complete and utter con. In the beginning I was also conned into spending many thousands on kit that made no objective difference (rather, it made me feel better at the time). I say this as a public service announcement for you: Wake up you fool.
Audiophiles have an incredible superpower of being able to notice tiny and subtle differences and changes between cables and accessories. Until you blindfold them.
Exactly ! Once upon a time a sells person even came to my house , cause I asked him if we can test his super expensive speaker cables . He agreed to blind test , and guess what I was playing my old very inexpensive cables and his , switching in between . That blinded fool guy said , that my inexpensive speakers cables sounds super good . He was so stupidly sure it was his cables , haha .
I remember back in the 1980s when a top stereo magazine reviewer tested two amplifiers. One was airy and transparent and one had better imaging and tighter bass. He was challenged to an ABX test and scored as random. I don't even want to mention the infamous 1980s Stereo Review test of amps from $169.00 to $6000.00. I think appearance is part of the equation too.
@@harackmw, not only that, but such things make it hard for equipment makers to get you spending $$$ on stuff that either doesn't do anything (power conditioners, LOL), or does it no better than other stuff that is 5 times cheaper.
i've done it myself and with some friends, years ago when i was still into audiophile stuff. basically, dac, 0 - 0.5% difference. amps, most listeners could actually hear a difference, but most of the time can't tell which one is better and nearly 50%/50% would end up 'preferring' the cheap entry level ones or the premium ones. cable? well i didn't even bother embarrassing myself for trying
Don’t symphonies now routinely, in order to keep out bias, when auditioning new players put them behind a screen so that what the player looks like is separate from what the player sounds like? Wouldn’t that be a kin to a blind test between two amplifiers or two sets of r speakers, what am I missing?
They actually play behind a screen so that the selection panel is unaware if one of the musicians is personally known to any individual on the panel. Appearance does come into play later when they want to make sure the musician blends with the other musicians and does not distract from the ensemble and performance.
@@raulgarcia1718 Apparently it was also a problem of knowing what the person was playing, as in what type of instrument and make, all things that people judge others on.
The ABX test or blind test is not the problem. Being defensive is. Being afraid is. Overcome that and you will be able to make an objective decision. Furthermore, learn to accept that you sometimes cannot hear a difference. 'I don't know' is a valid answer. Lastly, having money invested complicates things. That's why having a team of independent testers is preferable.
Placebos do work, and that is precisely why the control group in medicine / farmacy *has to be* the placebo group if we want to find out whether a medicine actually does work. If it has the same effect as a placebo - which is more than nothing - then it doesn't work. I take offense with the method of switching the HDMI cables as Paul describes. This process takes too long, you cannot compare sound that is separated by even 5 seconds of silence. Not only does it make it hard to tell if there is a difference, it also makes it easy to imagine a difference when there isn't any.
"it also makes it easy to imagine a difference when there isn't any." That isn't really relevant. In a properly conducted double-blind test with sufficient samples, then it is easy to work out if any differences claimed are statistically significant or essentially random. That's assuming that it is properly set up of course (and that's not always as easy as it sounds). As for the 5 second, or 10 second or whatever wait time is required, then the audiophile industry is always trying to tell us that there are dramatic differences, often described using some ill-defined terms, like brilliance or clarity. If the difference is so clear cut, those few seconds won't make any difference. The same is true of clinical trials. Those in the test and control groups are often asked very subjective questions, such as how much pain they are in, or whether they slept well, or what side-effects they experienced. It's only after crunching the numbers you work out if there's a real pattern or not.
@@TheEulerID You are very right on all points. I know that for me a 5 second silence makes it pretty much impossible to tell a difference between two pieces of well-sounding equipment (not broken or just horrible). Unless it's speakers, the difference between different speakers is always large enough. And yes, a blind test does remove the problem of imagining things (not even necessarily double blind).
I’ve done these, with a foot switch I made that I hook up with two cables that are exactly the same, coming from the back of my cabinet so I have no clue which I’m selecting.
@@harackmw that would be scary. when i swapped out my Musical Fidelity dac for a DirectStream dac..... i wouldve been able to tell the difference even if i was wearing earplugs. it was the single biggest jump ive ever experienced. _s t a g g e r i n g_
Just imagine. Someone spends a ton of money on a component and has an emotional investment in it, and as a result hears something "better." Who woulda thunk it? Ya know, what you're pushing here is acceptance of bias. No good!
Great approach. Really makes sense to me. It has to be blind because I can easily imagine something is better. I would react the same way to ABX. It would be useful as a final test to determine that the decision is worth making.
To test accurately you have to remove the "emotional" aspect. That is exactly what causes the placebo effect and makes you think your hearing something that isn't there. You REALLY ARE hearing a difference but it's in your brain and not the gear, that's the placebo effect! Manufacturers are experts at exploiting this.
If you play your guitar just by your ears and brain and don’t play with emotional feeling, most people will be way less interested in what you are playing..
@@nicktube3904 We are not talking about putting emotion into a musical performance, but about taking emotions out of a listening experience so as not to be unduly biased when listening to and evaluating different equipment. I wonder what would happen if you took two speakers, A and B, one costs $500, the other $1,500. You display the stickers on the speakers and have people come in to listen/audition the speakers. Well, I would guess most people are going to think the $1,500 speakers sound better. Now... I wonder what would happen if you switched the price tags/stickers and bring in new people to listen, you think most people are going to think the $500(really $1,500 speaker) is better? My guess would be many would not(the speakers tagged with the $1,500 sticker but really only $500 would sound better to them) -- cognitive bias at its best. If I were listening to a speaker/amp/preamp, CD player, DAC, cables, whatever, I would want to make an objective decision, the best one I possibly could, and so I would want to remove whatever subjective biases and preconceived notions I may have about whatever piece of equipment I'm hearing. More importantly, I would think the one that sounds better to me would also be the one that moves me more emotionally, and price/brand/looks should not bias that perception.
@@mydogskips2 my English isn’t that good, but yes it’s good to be objective. But for me personally I choose my gear 49% on ‘technical’ sounding good to my taste and 51% how emotional I got evolved by gear. I agree with you that expensive gear is currently not also the best sounding. The funny thing is that in practice (And I have listens to allot of gear, brands cheap and ridiculously expensive) until to a certain budget it is always a compromise. Emotional engagement vs technical good sounding. For example tonal rich, wide soundstage, organic human voices which are emotional engaging already for me.. VS really high resolution, lots of detail and razor sharp imaging.. In the end I don’t understand what people like about this analytical sound, I have never heard a Cello in a razor sharp image. In real life the frequencys and tones are room filling, that’s why I prefer gear like Audio Note!
@@edoreumerman4537 I'm talking about evaluating pieces of equipment, not listening to music. You have to differentiate between those two. And Hi end audio is all about sound. If your really listening to music, the gear doesn't matter. A music lover uses gear to listen to music, an audiophile uses music to listen to gear.
Paul says, in conclusion, "ABX testing doesn't work because it negates the entire emotional aspects of a very emotional endeavor which is high end audio listening." I'm wondering what is "high end audio"---Is it mainly an "emotional endeavor"? Is it like buying and owning a Louis Vuitton handbag, and even though by AB testing it holds the same amount of stuff securely as a $20 handbag, it is only "the emotional aspects of a very emotional endeavor" of owning a high end LV bag that truly matters to the owner? So high end audio is not about hearing higher fidelity sound with lower noise and distortion that can be objectively tested by ABX testing? If that's the case, why not simply put a low end equipment in a gold case ornamented with diamonds? I'm sure there is a highly emotional aspect of owning such a piece of high end audio jewelry.
If you want to use an analogy, I always thought cars work well. For the average consumer, a car gets you from A to B and reliability is key, but for the enthusiast the drive is part of the experience. Those are the people who will go out and purchase a luxury car. Some prefer big muscle cars, some prefer very sleek and well beautiful cars, and some prefer cars that has great handling, etc. Then there are the few high end enthusiasts who will go purchase a Lamborghini, Ferrari, or a Bugatti. So when Paul says ABX testing doesn't work, it's because while you can measure how fast a car goes you can't measure how much excitement or joy it brings because it's subjective. However, to the individual its easy to determine whether you prefer one car over another when you get to test-drive both. Same with audio.
@M Pi Well like with wines there is also an element of training involved. And double blind testing can fool most except a few talented winos. JBL has another take on blind testing of speakers I understand.
I'm sure a lot of high end equipment out there is the same design that well-made middle-end products have, but in a prettier box with better marketing.
Sorry but I disagree on this one. Double blind testing is a universal concept that applies to every field of research, whether it is medicine, wine tasting, or hifi gear. It is the only way to distinguish between the real value of an amplifier and the psychological impact that aesthetics, brand, general opinion may have on the entire listening opinion. I agree that listening to music comprises all of the previous factors, but as a consumer I want to know how much money I am pouring into real technical value and how much into the rest. Some may call the rest snake oil, some listening experience, but it doesn't change the fact that it is important to split the two.
The likelihood of being able to achieve true double-blind testing when it comes to audio equipment is virtually zero. It works for medicine (with the additional element of the placebo to make it gold standard) but it is impossible to replicate for audio equipment. People bandy the term "double-blind" about without understanding that double-blind means that nobody - not one person - in the chain knows which is which (in medical tests that means the patients AND the doctor are all unaware what they are given/taking) - and that makes it practically impossible for implementation outside of medicine. Wine tasting is never double-blind - it is always blind tasting, meaning that the participants are unaware what they are tasting but the host does (before you ask, 15 years as a wine merchant before a change in career - thousands of wines tasted and many, many long evenings of blind tastings). If any audio company made the claim that it was conducting double-blind testing I would call them out on it.
@@richardt3371 Good point, lets make it a single-blind test! Doesn't change the fact that some "blindness" in the process would help exclude snake oil form real value. That's what an open minded reviewer thinks about blind reviews: czcams.com/video/qJwy5ww2bKA/video.html (minute 4:35).
If you need long listening to hear a difference, it has no use to make a change. I did numerous modification. If I did not hear immediate (ca 20 seconds listening) difference , I decide the mod makes no difference and I roll back the mod.
Great observation. Similarly for hi res audio file formats. When we listen to music, we only listen to one copy, not constantly doing A/B with two file formats of the same recording. If the difference takes strenuous A/B testing by trained testers to discern, it's unlikely to be noticed during single copy listening of music. It doesn't matter for enjoying music at all.
Exactly! But if we remove that very modest improvement from High End Mftrs, they would all go broke. That is what they sell is that minuscule advancement (which sometimes isn’t any advancement in sound, but only in appearance).
Agreed. But subtle here, subtle there, subtle everywhere and it adds up. Worth it or not lies in the capability of the buyer, we are no one tell them that it is not worth it or it is too expensive. Me personally? I cant afford expensive shit like that.
It doesn't have to be a test, just a listening experience for you to figure out which setup, speaker, piece of gear, i.e. A, B, X, etc. you like more, just a way to listen for the differences. The "test" isn't about being right or wrong, it's about learning something, about determining what is better, in the most objective way possible. If the placebo effect is real, and I believe it is, then so too are price tags and preconceived notions/expectations of a given piece of equipment, probably even more so. I mean, the price of something and our knowledge of a product inevitably bias our perceptions of it; the only objective test is a blind test. And if you cannot accept being wrong, or won't even risk being wrong, I might suggest you take things a bit too seriously and have a bit too much of an ego. Personally, while I don't like being wrong, I think realizing when you are wrong can be quite useful, if you learn something from it, it's better than persisting in one's ignorance, unless that's your bliss. I'd rather know, even if it is initially a blow to my ego, I know I'll eventually get over it, and hopefully be the better for it.
I was involved in a blind test of amplifiers many years ago and I felt pressurised right from the start. The problem was that, quite simply, I didn't want to be proved wrong and end up feeling foolish.
Yeup. Here is a fun experiment. Try making a video of yourself abx testing two similar pieces of gear. If you can't tell the difference between the gear, review the tape. You would be surprised how often people can hear the differences in a very poor and low quality video, but never hear the differences in an ideal ABX setup.
Hi Paul, Thanks for your videos. I agree that being comfortable in the environment is important and that simple A-B testing works fine provided the listener absolutely does not know what A and B are.
Bose did this with there cube spks and a hidden sub.put a screen in front with a medium sized cabinet then another screen with the little bose cube playing.ov course low freq energy gives the impression of size in a driver
Dr Floyd Toole might not disagree entirely with Paul’s point. Paul is pointing out that the visual does change his opinion which is why Dr Toole removed that from his many experiments to drill down to just the perception of the sound & none of the input of the physical design. So Paul may feel he gets a better understanding by seeing it, but that is only because he is using other receptors than his ears. For we all know, ears don’t care if we can see or not. They don’t work any better for short periods if we remove our visual input. I think what Paul is saying is that in order to receive the full sensory enjoyment, we need to use all of our senses, but that doesn’t negate the value of double blind testing like Dr Toole has done for the last 5 decades.
Placebos are absolutely relevant, for the reason that Paul actually touches upon. Sometimes placebos 'work' because people fool themselves into thinking the placebo is doing some good when it's actually doing nothing at all. Now, read that sentence again but replace 'placebo' with 'snake oil'.
What is a placebo audio component? Because if I have someone tie my CD player to my amp with an interlink and a piece of rope made to look like an interlink, I will definitely be able to tell the difference through which one I am listening.
@@D1N02 A rope is not a placebo for an improved cable; it's just not a cable. It's the same as giving people rat poison in a clinical trial and call it a 'placebo'. I guess an example of a placebo audio component would be a cheap cable painted gold with a high selling price. The gold paint and the high price are the placebo. The "placebo effect" that Paul touched upon wrt clinical tests is that our subjectivity (thinking we actually took a working medicine) can actually have a positive effect on our healing process. This is indeed a real thing when testing medicine, but also completely irrelevant in audio testing. Our subjectivity has no effect on audio gear as it's not part of our body. And yet, salesmen of audio snake oil products always depend on the placebo effect. Subjectively they will absolutely work if you believe in them. It will all sound so much more "revealing", "3D", so much more detailed and realistic! But objectively they do nothing. That's their business model. And I believe the truth is that many of those audio snake oil salesmen actually fell into their own trap; they actually believe what they're selling actually works when in reality they don't do anything.
We can compare all electronics , speakers etc...in a blind test of lots of speakers , open panel models were preferred by majority ....no baffle coloration ....
What i hear easily is clipping and distortion, but at low volumes its not easy....but the difference between a mc inrosch ma 5100 amp and a class a electrocompaniet was so clear by lack of subjectif distortion of the class a , when it was hot....when cold it distorted
The end user/buyer perspective is totally different than the manufacturer's side... Buy what your ears say is the best sounding. Paul's food/drink analogy is spot on but not the way he meant it. Eat the dishes you like and who cares what someone else says you SHOULD like.
Exactly. This will make some people puke but I love the sound of tape and also the sound of properly crafted FM radio over many so-called benchmark formats. Call me weird.
@@InsideOfMyOwnMind Remember when FM radio stations had their own live studios? No wow and flutter there and I would record it on a hi-fi VHS tape which is still analog with almost no wow and flutter. Only a few good FM stations left that have decent sound, limiters turned up too high, no dynamics.
Apparently CZcams knew just what video to recommend when introducing this channel to me. Though it was a video from a week ago, I'm leaving this comment here because it's your most recent upload. It was the video asking, _is there any reason for high-resolution audio files?_ I put it in my watch later queue. But then I began browsing your past uploads. . _.oops._ Now I have this backlog of *at least* 15 videos to go through.¹ 😁👍 🎧 *SUBSCRIBED!* 🎧 Thank you CZcams! Looks like you did something right for a change. ··•✺•·· ¹ ─ And that was just when _I decided_ to stop, because clearly there were more I wanted to see! 😄
With AB testing we are not only selecting a preference we are learning to distinguish A from B, and in a development environment that is important. You might get the preference result with ABX but it would be slower and the learning element would be greatly reduced. Which of makes it obvious why audio designers tend not to use it.
True. If you feel like you are being tested, your brain could blank out. Happens to most of us. Remember taking your final exams back in highschool? You know that shit, you have learnt it and you memorized it. Come the big day, you sit there like a dumbfcuk looking at that question like you have never seen it before or struggle to recall what the answer is. After the test, EVERYTHING becomes clear again and you start telling yourself how stupid you were during the exams. Human mind can shut down when stressed or feel pressured.
Listening is done largely subconsciously. If you are putting your conscious brain in the way of your subconsciousness it's like putting on glasses that do not fit your eye sight. You do not listen with your brain but with your entire being. The brain plays a large roll but it is a nexus.
ABX testing is one of the corner stones that are responsible for your headphones and speakers sounding as good as they do. It was not at all obvious what constituted good speakers early on. And yes, I am very emotional... about my music! Not my speakers. They are simply electromechanical transducers, and the reason I embrace the science is I _HATE_ listening to speakers. I want to listen my music, not compression, severe response variations, harmonic and intermodulation distortion, resonance, and all the other very unmusical things they do. Setting things up properly allows me to tell the speakers to fuck off so I can carry on being emotional...
This is a person who A) cannot explain the principles of abx testing and B) is essentially arguing against the principle of hypothesis falsification which is the foundation scientific method and the philosophy of science. ABX does not require a "placebo". It is called a "control". In the case of audio it is usually: A&B: No Change With change X: A or B randomised If you need to be "relaxed" in order to "hear" the differences without "defensiveness" then you are engaging in a psychological process called confabulation. This is the audio equivalent of the flat earth hypothesis. "Getting through" to some people means they believe they have understood something that they cannot justify rationally: i.e. supersition.
There's no denying that a system has to make you feel something and respond to the music in some way, on some visceral or gut level, not just intellectually, and that's fine and good, just don't say you're being objective when you say something is better than another. If you know the price of the equipment or the details of a system, you are inevitably biased by such knowledge. Personally, I'd be very happy to know that a $500 speaker sounds just as good as a $5,000 one, some salesmen may not prefer that though. Now that I think about it, I think we should do speaker auditions blindfolded, it takes out the bias that our eyes provide us with. Don't tell me the price, and I don't want to know what the speakers/setup looks like, just let me listen blind, ABX, and I'll make my best decision, that's the decision based solely on what sounds best to me without any outside factors to influence me. After I know what I'm listening to, I would do a more in-depth long-term evaluation of the system with my eyes wide open so to speak, and see if it still moves me, if I feel the same way.
We are not speaking of speakers here nor are their beliefs that the same applies to speakers; speakers definatly sound differently no one is claiming otherwise.
What Paul said makes sense to me, and for his use case I think he's right. However for you to prove something as factually correct it needs to be with double blind listening.
Congratulations, you proved to yourself that your ears, your system or your room are not capable of resolving the difference. Therefore, you can save money, by not purchasing more expensive cables. What you have not even come close to proving is that more expensive cables don't make a difference. Surprise Surprise
@@markwagner1997 how should my ears be able to tell a difference if expensive measurement gear can’t? Good cables for 50 bucks are as transparent as super expensive cables.
Took and entire day to do a very controlled ABx test on a single $10,000 speaker cable vs 14ga lamp cord It was really freaking hard. The group of 20 "Golden Ear" Audiophiles was completely spent by the end of the day. Conclusion. For the most part we could not decisively tell which one was which. Length of wire was a couple meters.
@@juliangst measurements only show what can be measured by a specific test, not how a cable can effect sound. For my ears and the level of resolution my system achieves, 100 - 150 dollar Zu speaker cables and Morrow or Canare interconnects made a noticeable improvement. I borrowed some high end cables ($800-$1000) on a couple occasions and once again, my ears and my system did not find any improvement, so I happily did not feel any need to spend more. My experience is not representative of all people's ears or systems, nor is it a judgment on all cables.
@@wilcalint People who buy those expensive cables want to hear a difference so they claim stuff like ‚this cable has more low end‘ or ‚this cable is brighter‘ but in reality all of those cables don’t alter the signal by the slightest amount.
I've heard greater differences between two similar CDs than different transports ... i When performing AB/x comparisons I find it far better to eliminate the secrecy and reveal just what you ARE listening to ... the 'blind ' bit just confuses you !
I agree. I did an ABX test comparing 16/44 and 24/96 tracks and got 16/16 right by taking my time and really comparing and listening and having full control. As I started to challenge myself to see if I could hear the difference on a single pass, it struggled more. I also saw that the longer I was testing, the more fatigue I got and I wasn’t enjoying the music. ABX testing helps identify BIG differences. But subtle differences, which admittedly do not represent the “best bang for the buck” do get influenced by the stress of the test. Of course high end audio is never about value since chasing the last 10 percent of performance skyrockets the price.
ABx testing is VERY HARD. The closer the A & B are the more difficult it is to tell if x is either A or B. It can take hours and hours and by that time your completely spent.
It means the test should be carried out on a person, who has no emotional relationship with tested equipment. Not by the producer or audiophile, but for example by a musician.
It doesn't matter whether the person is biased towards the equipment if they can't actually see or otherwise know what they're listening to, that's the whole idea of blind testing.
You're right that a lot of subjective 'artistic' things cannot be tested in an objective manner. Is wine A better than wine B? Who's to say? We can say that there are differences in how they were made, what types of grapes are in them, when they were harvested etc, but none of this tells us if A is better than B. But here's the thing - ABX does not attempt to answer the question on whether A is better than B - it only attempts to answer the question of whether A is *different* from B. The former question is subjective, and the latter question is very much objective. In the same way, we can run a chemical analysis of the different compounds in wine A and wine B; if we find that every single molecule in wine A is completely replicated in wine B in the same composition and concentration, we are then forced to conclude that wine B is the same as wine A, and any question about whether wine A or wine B is better is moot. There is no way to disntiguish between them, and hence any perceived difference is completely illusory.
Both sides in the argument are right. Subjective sound is only one small part of the "HIFI experience" Visuals are important (valves that light up, big thick interconnects and mains cables, solid marble stands with gold trim), touching physical media (vinyl records, CDs, DVDs etc). I think the point at which you decide an interconnect sounds better is as soon as you make the purchase! If it enhances the whole listening experience then you have achieved an upgrade to your system or more accurately "an upgrade to your experience". Hifi is way more than just sound.
Real gem today. The comfort factor is EVERYTHING! I just got put through the ringer doing an upgrade project on my motorcycle because I became too unrelaxed. The frustration only created more petty stumbling blocks. Made more work for myself and pushed my completion date further. You lose all clarity when you let the stress get to ya.
Think of it like going to an exam - many people underperform or fumble at exams, because they get nervous as fight/flight kicks in. Guessing it some degree of that he wishes to avoid. (not gonna argue with it, just explaining it).
Paul, on the subject of A/B testing: Yesterday, you posting a video about the superior sound of DSD over PCM: czcams.com/video/KMYBd7Ukcuc/video.html Yet, you did not describe anything about how you conducted that test, other than saying that you record both formats at the same time. Do you have duplicates of your microphones, analog-to-digital converters, interconnects, etc, all of the same revisions? Did you use the exact same transport, etc, when you played back the DSD and the PCM files, for your artists to evaluate? Did you switch boxes, cables, etc. between your two set-ups, to ensure that those identical boxes really do sound the same? I want to believe you. But life experiences have taught me to be skeptical, even when dealing with well meaning, professionals (they make honest mistakes, too). Just as The Absolute Sound does, and just as Stereophile does, would you please list all associated gear that you used for both your DSD recording and playback, as well as all associated gear for your PCM recording and playback. Thank you.
The reality and the sad part is the DSD component may very well be better suited to those other set of components than the PCM component is. Too many variables. One needs to test the whole setup and see if it works for him/her/it/they and the room it is played in.
I think that people prefer the testing method that best confirms their biases. People who live by double blind testing tend to believe that human ears cannot be trusted to make objective evaluations. But ultimately human ears will do the evaluation. Just because a piece of equipment measures well or is created to A/B well against comparable products, it doesn’t mean that it will be enjoyable to listen to long term.
The problem is this believers cannot prove their beliefs all they can do is talk about them, try to convince others etc. If the differences between two amps are so miniscule how important are they? You cant even detect them when you are 'blind' !
I think best way is to listen to the new component on its own to familiarise yourself with its sound for a while before doing any back to back comparisons. Then get yr peak db to 88-90 dbA at the listening position and listen to 30-45 second segments back to back. All the mumbo jumbo you read by some reviewers that this new amp/dac/speaker whatever it is "completely blows away" the one they heard 6 months ago is total nonsense. Aural memory is very short indeed, so trust yr own ears in yr system in yr room. Then do an extended listen to the new gear and ask yourself if the new equipment is more enjoyable, engging, moving and satisfying to listen to than the older one. If not save your money and buy more records or CDs instead.
Makes sense Paul. As you said, placebo (expensive cable alert!) does work, and that's the only risk behind not doing blind ABX testing. It's also important that the subject be relaxed, and as you said if the walls go up, and the listener feels like he or she is the thing being tested, that will absolutely taint results.
People don't want to believe a lot of stuff that hurts their revenues or ego. Doesn't mean ABX testing is somehow flawed for audio. Selling things is all about manipulation, which is the aim of marketing. The real world however presents far more simpler challenges. The most common problem I see is people listening to music through expensive speakers in horrible spaces, negating the need for expensive speakers; or the second problem is listening to garbage, stuff that won't sound good on the best of audio equipment. The real world works on mass deception through the power of suggestion and shame. Works in politics, art, sound, auto sector, etc.
It is very simple, connect the cable A and B and say the result. There is no need for any emotional aspect in the environment of your home ?! You recognize black color in your house or in the laboratory. What this man claims is that you only see black in ideal conditions. :))) The ideal conditions are not to think with your head but to listen to him. It is easier to sell devices without any professional objective fair test. That’s why abx "doesn’t work" if you ask me..
@@larrywe3320 that's the point it mucks with your objectiveness when evaluating. Great when your just enjoying music but not when doing a/b testing. Unless that is your basis for evaluation.
Paul, now that you’re in your 70’s, shouldn’t the critical, final decisions on sound be made by your chosen (younger) appointees? As Joe says, Come on Man.
I'll, try to be brief and respectful. Paul is wrong on this one, and always has been. ABX testing is NOT to tell which one is better. It's to tell of you can tell a difference between A and B. I will agree with him it's VERY hard and takes the emotion completely out of the process. I have had the very controlled ABX test experience a number of times and that with 20 other I'll call them "Golden Ear" Audiophiles. IT'S REALLY FREAKING HARD. Especially using things like speaker cables. You can spend hours and hours doing it. At the other extreme doing an ABX test, tubes vs solid state most of the time is a no brainer. You can easily identify if X is either A or B, not which is better, A or B.
@@allansh828 I don't agree with you on the power cable thing as our group did an ABx test on that one. Result, we could not decipher A or B from x. But the interconnects there was some Golden Ears who could successfully pick correct 70/30. Not great but some success.
When the stake is "i might have wasted tens of thousands of dollars on something meaningless", of course you can get nervous. :) On the other hand, if you can tell something apart only when you aren't being scientifically observed, how good can that untestable opinion be? ;)
I agree if we’re talking about trying to objectify someone else’s a/b test and make a blanket statement out of it. BUT…if you were blind testing only for YOURSELF, that makes really good sense to me. Why not know the real truth for yourself? My only caveat would be to make the a/b tests long term tests, and not merely quick a/b ing back and forth; there are elements of sound you can’t hear when jumping back and forth.
Using any kind of switch box will color the sound so you can't hear the diferences. I once heard the difference between a switch box and no switch box. It colored the sound a lot.
Comparing the sound from quality electronics such as CD transports or amplifiers is probably a waste of time and may be misleading. Unless there is a defective product then differences in audio may be very subtle. A may sound different to B, but what does this mean? Does this difference mean “better” or “worse”. This is highly subjective. Room acoustics and loudspeaker choice may have far more effect on audio quality and just moving speaker position can affect perceptions of the electronics.
Dear Paul, I don't always agree with your explanations. But this time you nailed, great video! Maby this video will help me with some annoying cable non-believers xD
I think Dr. Floyd Toole would disagree on this fact. Nobody had more experience with the field of psychoacousyics and how seeing the product influences what people think. Almost 20 years at the NRC doing some groundbreaking research on how we hear and what people like . Sorry Paul you're wrong about this one.
ABX tests YOU and not the equipment. If you cant spot a difference, that means YOU cant spot the difference, it doesnt mean there are no differences produced by the equipments. Tests like these are made to fail you, even if 6 out of 10 or 7 out of 10 people get it right, people will still say that is very near to chance. They expect results to be 9 out of 10 or 10 out of 10 to shut them up. Everybody's sense of hearing is different, those 5, 6, 7 people that pass the test are put under and categorized as 'chance' which is pure bs. Its a game that you will never win.
You'd be right if your sample size is only 10 people, but what if it's 10,000 or 100,000? Not that that's realistic, but if it were theoretically performed and 5,000 out of 10,000 could not hear a difference over not just one track but 20 tracks, then it could be the difference is irrelevant to the human ear and not worth the money. Statistics is not a perfect science, but it can give you a degree of confidence when your sample is big enough
@@Chessmapling if 5000 out of 10000 people could not hear a difference, that means 5000 more COULD. Putting these actual humans under 'chance' is just ridiculous. Statistics are useful but dumb when used this way.
Correct Paul,most people don’t know about the emotions involved in listening 👂 witch I think 🤔 is the most important criteria in audio listening playback pleasure,and having long listening sessions
I promise I'm not being disrespectful but I'm genuinely trying to find a reason to not find this take anserine.. and can't. Even ignoring the fact that he doesn't understand ABX testing there's a problem when his argument against his misunderstood idea of ABX testing is "I would get defensive" which doesn't mean 'abx testing doesn't work' but at best "I'm unable/unwilling to do abx testing".. Wild imo...
Lol at all the deafies in the comment section pleading at the altar to ABX testing, as if it is some perfect infallible method that can't possibly introduce biases of its own into the equation. It's alway such a coincidence the people who do this are also the people who do not own actual hifi systems.
Well, as long as you are doing blind A/B, your fine! And, being a Neuroscientist, I can tell you it's the same in Biology and Medicine. I think the introduction was not very clearly worded , and it confused some people into thinking you are not doing blind A/B testing. And of course, you are right, having the "X" added makes things more difficult, with no gains at all for your work.
Not at all. If you have a properly designed switching system so that A/B/X are clickless and quickly switched (time aligned, level aligned, etc) then you allow the use of short-term loudness memory, which is shown to be the most sensitive analysis of any kind of perception involving partial loudnesses.
@@AALavdas I refer you to the term "partial loudnesses" not "loudness", in other words, the detected loudness across the basilar membrane as a function of frequency and time. With a proper ABX, one can most easily detect the DIFFERENT pair, by comparing A with X and B with X. This is still a pairwise comparison in a sense, the key being (as always) to listen for difference, rather than similarity. A/B only offers you "same" or "not same", and does not give you the permanent A and B references (A and B should not change identity throughout a single test series, although they can move in another test series) and by doing so, removes the sensory anchor that makes hearing the difference an easier task.
@@jamesjohnston3643 Well, saying "across the basilar membrane" does not add anything to this sentence. And there is no way to know that, unless you have a probe on the basilar membrane. I agree that there are different merits in different approaches, but blind A/B is fine.
@@AALavdas Well, I didn't come here for an answer, but I didn't expect to encounter somebody who denies settled science. I would suggest that you check the works of Zwislocki, Greenwood, Zwicker, Allen (Jont, not Jon), Hall, and others as far as being able to estimate, at various levels, the excitation on the basilar membrane. You are completely incorrect in your dismissal. Furthermore, you might learn what "partial loudnesses" are. I can create two very different signals that have the exact same loudness (in sones, phons, whatever you want) that will have entirely different partial loudnesses. Allen's work on self-training tests shows, conclusively, that proximate, clickless comparison at the subject's will provide the smallest dl's.
All hail Topping DAC's !!!! Ops, mine just broke.... It might have the best SINAD score but does it sound better? If perfectly flat frequency response is your thing, try to listen to a perfect 40 RTA car audio. I love my tone control and spices in my food thank you.
omg - - the comments are soooo predictable - - thanks Paul, this vid was certainly insightful, reassuring actually, at least at some level, simple listening tests are taken into account - - I'd bet you could spend an hour going over all of the "other" testing that must go on at PS audio - - so all those commenters that are falling off of their chairs to criticize, take a moment and try to imagine all of the "other" testing that of course goes on well before PS Audio brings any component to market. Ya think Amir is the only one out there with a high end analyzer and oscilloscope?
I agree that blind AB testing is useless. To really gauge a piece of equipment you need to live with it for a while not 5 minutes. A more in your face sound will grab your attention in a quick comparison. After living with that sound for some time you may find that is ultimately hard and fatiguing. This is why most budget hifi sounds so awful. It is designed by the manufacturer to grab the attention so that it's the one you choose in a quick demo.
Well, but he just ended his explanation saying he uses blind AB testing, but not ABX. Scientific methods are quite useful in industrial environment. At home you use your emotions, in a factory you test the hell out of it.
So Paul you basically admit that audio perceptions are due to emotional states which are not to be standardized operationally? Nice argument in order to circumvent the need to objectify claimed acoustic differences of hifi gear. Apart fr that you confuse a randomized control trial with ABX testing. Again you are arguing with anecdotal evidence and heuristics. The confidence needs to be derived from the perceptions. If that does not work for you, you admit you do not hear the differences. If you hear them, you will hear them in the ABX. Nice that you admit that. All I needed to know.
LOL!!! You object to a test for being tested? That is sadly funny. The whole point of ABX testing is independence. You are right in that highend is emotional, and that is why so often people are fooled. Take the emotion out and you get to actual differences (or lack thereof). ABX or bust. Amir recently did a good video on ABX testing, supported with actual data. Now that is a good video.
No, he objects to a scientific test conducted unscientifically. You simply can't eliminate emotional response, or tastes, or preferences, from a blind test of audio equipment. ABX testing is inherently flawed where it involves humans expressing an opinion. You cannot take the emotion out of subjective listening, blind or otherwise. Data-driven testing is a completely different thing.
@@richardt3371 One can easily test for emotional preference. Which joke is more funny? Ask 100 people, and you will get an answer. Or the infamous Coke vs. Pepsi test in neutralized containers. You can see whether people can tell the difference, and then state a preference. And you can evaluate that statistically. No problem. The same applies to audio. Can you hear a difference between two cables/DACs, etc. and which one sounds better to you. The problem is, that most people cannot hear a difference, and that is a problem for sales.
@@danielgeiger7739 Sigh. You're missing the whole point of blind testing, which is designed to eliminate preconceptions or tastes, and which is why it is of no use whatsoever as soon as you introduce emotion. Asking someone if they prefer Coke or Pepsi, and testing them blind, gives you no useable information - you can't "evaluate that statistically", what you end up evaluating is human response, not whether one is actually better than another. It's an interesting experiment but it's also dead data.
@@richardt3371 OK: statistics 101. First you can statistically figure out whether one person can distinguish between Coke and Pepsi. Give them as much time/drink as they want to familiarize themselves with each, then do the blind test where they need to assign the sample to either coke or Pepsi.> 8/10 correct, they pass. Re preference, that is a n =1 issue, of course. So you get 10 (or more) people who can distinguish the two, then determine their preference. Then you see whether there is a population preference for Coke/Pepsi. If there is a >= 8 vs 2 distribution, then there is a population preference. Same can be done for cables, DACS etc. If it is
Yeah I can understand your argument, and I find it valid. But it's sad to see a professional that aways use technical information to explain things suddenly warp words to try convince people of his argument, instead of just presenting it as is. Placebo work sometimes, yes, but never "more often than not". Especially when talking about medicines, you need to be way more responsible than that.
Love this answer, Paul - absolutely love it. The number of times I'm asked for ABX testing of the products we install and it really upsets clients to be told no, we don't do that. Fire-testing? Sure. Soundproofing? Sure. Those are scientific, quantifiable, data-led comparisons, and have validity. Does one sound "better" than another? With the back and forth: Now try that again? Now are you sure you're listening to modular type A or type B? Nope - it's too nebulous and too open to interpretation and preference. Audio is emotion, it's emotional response. Getting bogged down in technical specifications is clinical and simply doesn't work. I hate it when posts on here scream for "ABX testing data please!" Nope.
Don't you see how conflicting this is coming from someone who sells expensive audio equipment? Of course emotion is highly important. We all know that but most products are sold with pseudo-scientific, emotionless oxygen-free quadruple-shielded false mambo-jambo.
Makes perfect sense to me. The concept of a double-blind test appeals to my scientific side, but as Paul relates his experience I can certainly identify with his ‘cognitive dissonance.’
It would mean nothing outside the clinical concept. "Double blind" means that both the patient and the investigator are blinded. There is no "double blind" outside clinical medicine. There is no patient here, just investigator. Even if there were 10 investigators, it would still be just "blind test" if they are all blinded.And that is enough.
@@AALavdas, it would be double blind testing if you called a bunch of your friends to come to you and listen to some equipment comparatively, and you yourself wouldn't know at which point which equipment is being listened to. If you - the organizer of the test - did now and only your friends - the listeners - didn't, then it would be simple blind testing (not double blind).
So if there is no way to know that you hear a difference you do hear it.. but if we can actually test it you get to stressed to hear the difference? That is very convenient. Just saying.
ABX tests in audio works perfectly fine. It's just you don't like the results
hahaha, nice.
Yeah bad for business
Its exactly this. People that make money on selling stuff dont like the results of abx testing because they loose money. We in the sound science and recording industry love it because we like to debunk the myths and snake oil crap.
You are correct. The whole area is replete with those who prefer to use the power of suggestion. There are, of course, subject aspects to music; it's an art form, but the same is not true when it comes to the technical accuracy of reproduction. I manufacturer X claims their extremely expensive loudspeaker cable is better than generic cable (of an appropriate gauge), then there's no reason why that can't be double-blind tested. It doesn't even need to sound better, just different would be a start. The same then goes for a whole host of things, like audio quality power cables, or digital signal cables, or network switches.
That is not to say that there might not be strong preferences for those things that can actually make a difference, but in those cases, a double-blind test would show that people hear there is actually one. But for many components this is not the case, and particularly not for those that do simple, functional roles rather than, say, act as transducers.
Because you can’t pass the test.
ABX test in high end audio industry is a big NO.
Hi Paul. Don't get me wrong, but I think you have a wrong idea about ABX testing. In ABX you try to guess what the X is and you can listen to A and B any time. So you know for example that A is the old equipment and B is new. Then you can ask to listen to A and B any time you want. You can also listen to X which is either A or B and then your aim is to know which one X is. There are no tricks or fooling around. AB and ABX tests have different purposes. AB is used when you want to know which one of two has the desired quality. For example, which one of the two pieces of equipment sounds better. In this test, you don't know which one is which. ABX test has a different purpose that is can you spot the difference. Here you know what A and B are and you try to guess which one the X is. There is never a C involved or the person doing the test doesn't know what he/she is doing. Both of these tests are super useful in audio but they have a completely different purpose. I also completely understand people who don't want to test. I don't want or need to test some things. But for some things I want to see if I really can spot the difference between two pieces and this is where ABX testing is very useful.
i'm a total cheapskate so rigorous testing has value to me. The idea of spending a few hundreds of dollars extra on something I can't even hear in a controlled environment is pretty concerning.
If you don't feel comfortable with an ABX test, then that's because you're afraid of being exposed. And being confronted with the reality that you wasted tons of money on snake oil. If you can't tell the difference in an ABX test, then you can neither under normal conditions.
mic drop, well said.
Double blind testing doesn't work? Just about the entire scientific community disagrees. And it is easily transferrable to audio. The reason why PS audio and other audio companies (Audioquest etc) don’t like it is because it is an existential threat. If ABX testing scientifically demonstrated that your kit "sounded better" then it would plastered all over your literature. If your expensive kit sounded better as claimed then ABX test it and show us the results.
You are just writing a lot of nonsenses. Paul very eloquently describes why it’s useless and why A/B blind test works fine. As he says it works for pharmaceuticals but that’s something else.
@@dieselbrodeur Two years and this is your response? (that the vested interest of an MD selling overpriced kit knows better than the whole scientific community). It’s almost as if these companies rely heavily on snake oil because they have no peer reviewed scientific ABX evidence and cant scientifically prove their claims - if they did publish such results it would be damaging to their business model.
Buy hey you believe Paul has perfect pitch, can hear a pin drop from 50metres away and the golden ears of a dog. A 60-80 year old biological marvel defying the laws of human aging, if you will… deary me.
@@dieselbrodeur The real reason why they do not ABX test is because they cannot objectively demonstrate their subjective claims. Its almost as if there is a business model that relies heavily on snake oil to justify the price tags. Overpriced cables, mains conditioners and plethora of other items conjured up are a complete and utter con. In the beginning I was also conned into spending many thousands on kit that made no objective difference (rather, it made me feel better at the time). I say this as a public service announcement for you: Wake up you fool.
Audiophiles have an incredible superpower of being able to notice tiny and subtle differences and changes between cables and accessories.
Until you blindfold them.
haha!
Exactly ! Once upon a time a sells person even came to my house , cause I asked him if we can test his super expensive speaker cables .
He agreed to blind test , and guess what I was playing my old very inexpensive cables and his , switching in between . That blinded fool guy said , that my inexpensive speakers cables sounds super good . He was so stupidly sure it was his cables , haha .
Then that would be a blind test? If you weren't blindfolded then you would be subject to expectation bias.
@@zulumax1 That was my point, yes
I remember back in the 1980s when a top stereo magazine reviewer tested two amplifiers. One was airy and transparent and one had better imaging and tighter bass. He was challenged to an ABX test and scored as random. I don't even want to mention the infamous 1980s Stereo Review test of amps from $169.00 to $6000.00. I think appearance is part of the equation too.
shhhhhh, such things are too hard to handle! :D
@@harackmw, not only that, but such things make it hard for equipment makers to get you spending $$$ on stuff that either doesn't do anything (power conditioners, LOL), or does it no better than other stuff that is 5 times cheaper.
@@VioletGiraffe Yup, or those insanely priced cables.
i've done it myself and with some friends, years ago when i was still into audiophile stuff.
basically, dac, 0 - 0.5% difference. amps, most listeners could actually hear a difference, but most of the time can't tell which one is better and nearly 50%/50% would end up 'preferring' the cheap entry level ones or the premium ones. cable? well i didn't even bother embarrassing myself for trying
Don’t symphonies now routinely, in order to keep out bias, when auditioning new players put them behind a screen so that what the player looks like is separate from what the player sounds like? Wouldn’t that be a kin to a blind test between two amplifiers or two sets of r
speakers, what am I missing?
Well, apparently audiophiles can't stand to have their golden ears tested, they're too sensitive.
They actually play behind a screen so that the selection panel is unaware if one of the musicians is personally known to any individual on the panel. Appearance does come into play later when they want to make sure the musician blends with the other musicians and does not distract from the ensemble and performance.
@@raulgarcia1718 Apparently it was also a problem of knowing what the person was playing, as in what type of instrument and make, all things that people judge others on.
thing is you can definitely apply the scientific method to "emotional" endeavors - sampling food or drink, etc.
The ABX test or blind test is not the problem. Being defensive is. Being afraid is. Overcome that and you will be able to make an objective decision. Furthermore, learn to accept that you sometimes cannot hear a difference. 'I don't know' is a valid answer. Lastly, having money invested complicates things. That's why having a team of independent testers is preferable.
Placebos do work, and that is precisely why the control group in medicine / farmacy *has to be* the placebo group if we want to find out whether a medicine actually does work. If it has the same effect as a placebo - which is more than nothing - then it doesn't work.
I take offense with the method of switching the HDMI cables as Paul describes. This process takes too long, you cannot compare sound that is separated by even 5 seconds of silence. Not only does it make it hard to tell if there is a difference, it also makes it easy to imagine a difference when there isn't any.
"it also makes it easy to imagine a difference when there isn't any."
That isn't really relevant. In a properly conducted double-blind test with sufficient samples, then it is easy to work out if any differences claimed are statistically significant or essentially random. That's assuming that it is properly set up of course (and that's not always as easy as it sounds).
As for the 5 second, or 10 second or whatever wait time is required, then the audiophile industry is always trying to tell us that there are dramatic differences, often described using some ill-defined terms, like brilliance or clarity. If the difference is so clear cut, those few seconds won't make any difference.
The same is true of clinical trials. Those in the test and control groups are often asked very subjective questions, such as how much pain they are in, or whether they slept well, or what side-effects they experienced. It's only after crunching the numbers you work out if there's a real pattern or not.
@@TheEulerID You are very right on all points. I know that for me a 5 second silence makes it pretty much impossible to tell a difference between two pieces of well-sounding equipment (not broken or just horrible). Unless it's speakers, the difference between different speakers is always large enough. And yes, a blind test does remove the problem of imagining things (not even necessarily double blind).
I’ve done these, with a foot switch I made that I hook up with two cables that are exactly the same, coming from the back of my cabinet so I have no clue which I’m selecting.
Kinda glad I did some research before looking at dacs especially, saved myself a lot of money.
i know youll enjoy your DirectStream DAC..... it is incredible!
@@googoo-gjoob But it is only $6000, I clearly need a $10000 dac
@@harackmw , that tongue in your cheek..... does it hurt?
i own the DS DAC. it truly is incredible. 30 day in home.... nothing to risk.
@@googoo-gjoob The risk is trying a blind test and failing to spot the dac. :D
@@harackmw that would be scary. when i swapped out my Musical Fidelity dac for a DirectStream dac..... i wouldve been able to tell the difference even if i was wearing earplugs.
it was the single biggest jump ive ever experienced. _s t a g g e r i n g_
Just imagine. Someone spends a ton of money on a component and has an emotional investment in it, and as a result hears something "better." Who woulda thunk it? Ya know, what you're pushing here is acceptance of bias. No good!
yup.
Great approach. Really makes sense to me. It has to be blind because I can easily imagine something is better. I would react the same way to ABX. It would be useful as a final test to determine that the decision is worth making.
To test accurately you have to remove the "emotional" aspect. That is exactly what causes the placebo effect and makes you think your hearing something that isn't there. You REALLY ARE hearing a difference but it's in your brain and not the gear, that's the placebo effect! Manufacturers are experts at exploiting this.
If you play your guitar just by your ears and brain and don’t play with emotional feeling, most people will be way less interested in what you are playing..
@@nicktube3904 We are not talking about putting emotion into a musical performance, but about taking emotions out of a listening experience so as not to be unduly biased when listening to and evaluating different equipment.
I wonder what would happen if you took two speakers, A and B, one costs $500, the other $1,500. You display the stickers on the speakers and have people come in to listen/audition the speakers. Well, I would guess most people are going to think the $1,500 speakers sound better. Now... I wonder what would happen if you switched the price tags/stickers and bring in new people to listen, you think most people are going to think the $500(really $1,500 speaker) is better? My guess would be many would not(the speakers tagged with the $1,500 sticker but really only $500 would sound better to them) -- cognitive bias at its best.
If I were listening to a speaker/amp/preamp, CD player, DAC, cables, whatever, I would want to make an objective decision, the best one I possibly could, and so I would want to remove whatever subjective biases and preconceived notions I may have about whatever piece of equipment I'm hearing. More importantly, I would think the one that sounds better to me would also be the one that moves me more emotionally, and price/brand/looks should not bias that perception.
@@mydogskips2 my English isn’t that good, but yes it’s good to be objective. But for me personally I choose my gear 49% on ‘technical’ sounding good to my taste and 51% how emotional I got evolved by gear.
I agree with you that expensive gear is currently not also the best sounding.
The funny thing is that in practice (And I have listens to allot of gear, brands cheap and ridiculously expensive) until to a certain budget it is always a compromise. Emotional engagement vs technical good sounding. For example tonal rich, wide soundstage, organic human voices which are emotional engaging already for me.. VS really high resolution, lots of detail and razor sharp imaging..
In the end I don’t understand what people like about this analytical sound, I have never heard a Cello in a razor sharp image. In real life the frequencys and tones are room filling, that’s why I prefer gear like Audio Note!
Music without emotion is sound and that is not what high end audio is all about
@@edoreumerman4537 I'm talking about evaluating pieces of equipment, not listening to music. You have to differentiate between those two. And Hi end audio is all about sound. If your really listening to music, the gear doesn't matter. A music lover uses gear to listen to music, an audiophile uses music to listen to gear.
Paul says, in conclusion, "ABX testing doesn't work because it negates the entire emotional aspects of a very emotional endeavor which is high end audio listening." I'm wondering what is "high end audio"---Is it mainly an "emotional endeavor"? Is it like buying and owning a Louis Vuitton handbag, and even though by AB testing it holds the same amount of stuff securely as a $20 handbag, it is only "the emotional aspects of a very emotional endeavor" of owning a high end LV bag that truly matters to the owner? So high end audio is not about hearing higher fidelity sound with lower noise and distortion that can be objectively tested by ABX testing? If that's the case, why not simply put a low end equipment in a gold case ornamented with diamonds? I'm sure there is a highly emotional aspect of owning such a piece of high end audio jewelry.
If you want to use an analogy, I always thought cars work well. For the average consumer, a car gets you from A to B and reliability is key, but for the enthusiast the drive is part of the experience. Those are the people who will go out and purchase a luxury car. Some prefer big muscle cars, some prefer very sleek and well beautiful cars, and some prefer cars that has great handling, etc. Then there are the few high end enthusiasts who will go purchase a Lamborghini, Ferrari, or a Bugatti.
So when Paul says ABX testing doesn't work, it's because while you can measure how fast a car goes you can't measure how much excitement or joy it brings because it's subjective. However, to the individual its easy to determine whether you prefer one car over another when you get to test-drive both. Same with audio.
@M Pi Well like with wines there is also an element of training involved. And double blind testing can fool most except a few talented winos. JBL has another take on blind testing of speakers I understand.
@@dan.nathan The same goes for bicycles and here’s another analogy, the setting up of a bicycle is as important as that of a hifi system.
I'm sure a lot of high end equipment out there is the same design that well-made middle-end products have, but in a prettier box with better marketing.
PCM or FLAC are they the same ? If not which is Better for streaming music. Ty Paul
FLAC is PCM audio. Do you mean DSD vs PCM?
Schrodingers audiophile: i both can tell the difference and I cant tell the difference. Its quantum physics really.
Sorry but I disagree on this one. Double blind testing is a universal concept that applies to every field of research, whether it is medicine, wine tasting, or hifi gear. It is the only way to distinguish between the real value of an amplifier and the psychological impact that aesthetics, brand, general opinion may have on the entire listening opinion. I agree that listening to music comprises all of the previous factors, but as a consumer I want to know how much money I am pouring into real technical value and how much into the rest. Some may call the rest snake oil, some listening experience, but it doesn't change the fact that it is important to split the two.
The likelihood of being able to achieve true double-blind testing when it comes to audio equipment is virtually zero. It works for medicine (with the additional element of the placebo to make it gold standard) but it is impossible to replicate for audio equipment. People bandy the term "double-blind" about without understanding that double-blind means that nobody - not one person - in the chain knows which is which (in medical tests that means the patients AND the doctor are all unaware what they are given/taking) - and that makes it practically impossible for implementation outside of medicine. Wine tasting is never double-blind - it is always blind tasting, meaning that the participants are unaware what they are tasting but the host does (before you ask, 15 years as a wine merchant before a change in career - thousands of wines tasted and many, many long evenings of blind tastings).
If any audio company made the claim that it was conducting double-blind testing I would call them out on it.
@@richardt3371 Good point, lets make it a single-blind test! Doesn't change the fact that some "blindness" in the process would help exclude snake oil form real value. That's what an open minded reviewer thinks about blind reviews: czcams.com/video/qJwy5ww2bKA/video.html (minute 4:35).
If you need long listening to hear a difference, it has no use to make a change. I did numerous modification. If I did not hear immediate (ca 20 seconds listening) difference , I decide the mod makes no difference and I roll back the mod.
Great observation. Similarly for hi res audio file formats. When we listen to music, we only listen to one copy, not constantly doing A/B with two file formats of the same recording. If the difference takes strenuous A/B testing by trained testers to discern, it's unlikely to be noticed during single copy listening of music. It doesn't matter for enjoying music at all.
Same here
Loved this video
The difference doesnt have to be real... you just have to convince others it is.
Paul, a fellow hifi reviewer has said servo control subwoofers are a waste of time....can you do another vid on this topic ?.....thx
If it's that subtle, it s something i can live without
Exactly! But if we remove that very modest improvement from High End Mftrs, they would all go broke. That is what they sell is that minuscule advancement (which sometimes isn’t any advancement in sound, but only in appearance).
Agreed. But subtle here, subtle there, subtle everywhere and it adds up. Worth it or not lies in the capability of the buyer, we are no one tell them that it is not worth it or it is too expensive. Me personally? I cant afford expensive shit like that.
It doesn't have to be a test, just a listening experience for you to figure out which setup, speaker, piece of gear, i.e. A, B, X, etc. you like more, just a way to listen for the differences. The "test" isn't about being right or wrong, it's about learning something, about determining what is better, in the most objective way possible.
If the placebo effect is real, and I believe it is, then so too are price tags and preconceived notions/expectations of a given piece of equipment, probably even more so. I mean, the price of something and our knowledge of a product inevitably bias our perceptions of it; the only objective test is a blind test.
And if you cannot accept being wrong, or won't even risk being wrong, I might suggest you take things a bit too seriously and have a bit too much of an ego.
Personally, while I don't like being wrong, I think realizing when you are wrong can be quite useful, if you learn something from it, it's better than persisting in one's ignorance, unless that's your bliss. I'd rather know, even if it is initially a blow to my ego, I know I'll eventually get over it, and hopefully be the better for it.
This is great, excellent explanation of your process. I am convinced. Thanks Paul!
You’re convinced that science doesn’t work by a non-scientist?
@@philiprobar I was convinced that Paul has a process that works.
@@Wacoal34dthat both works and sells
I was involved in a blind test of amplifiers many years ago and I felt pressurised right from the start. The problem was that, quite simply, I didn't want to be proved wrong and end up feeling foolish.
Yeup. Here is a fun experiment. Try making a video of yourself abx testing two similar pieces of gear. If you can't tell the difference between the gear, review the tape. You would be surprised how often people can hear the differences in a very poor and low quality video, but never hear the differences in an ideal ABX setup.
Hi Paul, Thanks for your videos. I agree that being comfortable in the environment is important and that simple A-B testing works fine provided the listener absolutely does not know what A and B are.
And that is going to depend a lot on the set up of the testing hardware.
Bose did this with there cube spks and a hidden sub.put a screen in front with a medium sized cabinet then another screen with the little bose cube playing.ov course low freq energy gives the impression of size in a driver
Was this an April fools post?
Dr Floyd Toole might not disagree entirely with Paul’s point. Paul is pointing out that the visual does change his opinion which is why Dr Toole removed that from his many experiments to drill down to just the perception of the sound & none of the input of the physical design. So Paul may feel he gets a better understanding by seeing it, but that is only because he is using other receptors than his ears. For we all know, ears don’t care if we can see or not. They don’t work any better for short periods if we remove our visual input. I think what Paul is saying is that in order to receive the full sensory enjoyment, we need to use all of our senses, but that doesn’t negate the value of double blind testing like Dr Toole has done for the last 5 decades.
Placebos are absolutely relevant, for the reason that Paul actually touches upon.
Sometimes placebos 'work' because people fool themselves into thinking the placebo is doing some good when it's actually doing nothing at all. Now, read that sentence again but replace 'placebo' with 'snake oil'.
What is a placebo audio component? Because if I have someone tie my CD player to my amp with an interlink and a piece of rope made to look like an interlink, I will definitely be able to tell the difference through which one I am listening.
@@D1N02 A rope is not a placebo for an improved cable; it's just not a cable. It's the same as giving people rat poison in a clinical trial and call it a 'placebo'.
I guess an example of a placebo audio component would be a cheap cable painted gold with a high selling price. The gold paint and the high price are the placebo.
The "placebo effect" that Paul touched upon wrt clinical tests is that our subjectivity (thinking we actually took a working medicine) can actually have a positive effect on our healing process. This is indeed a real thing when testing medicine, but also completely irrelevant in audio testing. Our subjectivity has no effect on audio gear as it's not part of our body.
And yet, salesmen of audio snake oil products always depend on the placebo effect. Subjectively they will absolutely work if you believe in them. It will all sound so much more "revealing", "3D", so much more detailed and realistic! But objectively they do nothing. That's their business model.
And I believe the truth is that many of those audio snake oil salesmen actually fell into their own trap; they actually believe what they're selling actually works when in reality they don't do anything.
I wouldn't call it placebo effect, but confirmation bias. You spend a lot of money on something, so it has to be good.
You can't have a placebo in an audio test. What do you suggest - don't play music and mime?
@@richardt3371 Are you deliberately missing the point?
We can compare all electronics , speakers etc...in a blind test of lots of speakers , open panel models were preferred by majority ....no baffle coloration ....
What i hear easily is clipping and distortion, but at low volumes its not easy....but the difference between a mc inrosch ma 5100 amp and a class a electrocompaniet was so clear by lack of subjectif distortion of the class a , when it was hot....when cold it distorted
The end user/buyer perspective is totally different than the manufacturer's side... Buy what your ears say is the best sounding. Paul's food/drink analogy is spot on but not the way he meant it. Eat the dishes you like and who cares what someone else says you SHOULD like.
Exactly. This will make some people puke but I love the sound of tape and also the sound of properly crafted FM radio over many so-called benchmark formats. Call me weird.
@@InsideOfMyOwnMind Remember when FM radio stations had their own live studios? No wow and flutter there and I would record it on a hi-fi VHS tape which is still analog with almost no wow and flutter. Only a few good FM stations left that have decent sound, limiters turned up too high, no dynamics.
Spoken like a true politician !
Apparently CZcams knew just what video to recommend when introducing this channel to me. Though it was a video from a week ago, I'm leaving this comment here because it's your most recent upload.
It was the video asking, _is there any reason for high-resolution audio files?_ I put it in my watch later queue. But then I began browsing your past uploads. . _.oops._ Now I have this backlog of *at least* 15 videos to go through.¹ 😁👍
🎧 *SUBSCRIBED!* 🎧
Thank you CZcams! Looks like you did something right for a change.
··•✺•··
¹ ─ And that was just when _I decided_ to stop, because clearly there were more I wanted to see! 😄
Cool story bro.
@@Frankonero12 Thanks brother. As I said, I'm brand new to the channel, and _I am loving these Q&A's!_
Contrary to his reasons for not blind testing (wanting to know if A or B is playing), he contradicts this @ 5:28.
เจ๋งครับพรี่
With AB testing we are not only selecting a preference we are learning to distinguish A from B, and in a development environment that is important.
You might get the preference result with ABX but it would be slower and the learning element would be greatly reduced. Which of makes it obvious why audio designers tend not to use it.
True. If you feel like you are being tested, your brain could blank out. Happens to most of us. Remember taking your final exams back in highschool? You know that shit, you have learnt it and you memorized it. Come the big day, you sit there like a dumbfcuk looking at that question like you have never seen it before or struggle to recall what the answer is. After the test, EVERYTHING becomes clear again and you start telling yourself how stupid you were during the exams. Human mind can shut down when stressed or feel pressured.
Listening is done largely subconsciously. If you are putting your conscious brain in the way of your subconsciousness it's like putting on glasses that do not fit your eye sight. You do not listen with your brain but with your entire being. The brain plays a large roll but it is a nexus.
ABX testing is one of the corner stones that are responsible for your headphones and speakers sounding as good as they do. It was not at all obvious what constituted good speakers early on. And yes, I am very emotional... about my music! Not my speakers. They are simply electromechanical transducers, and the reason I embrace the science is I _HATE_ listening to speakers. I want to listen my music, not compression, severe response variations, harmonic and intermodulation distortion, resonance, and all the other very unmusical things they do. Setting things up properly allows me to tell the speakers to fuck off so I can carry on being emotional...
This is a person who A) cannot explain the principles of abx testing and B) is essentially arguing against the principle of hypothesis falsification which is the foundation scientific method and the philosophy of science.
ABX does not require a "placebo". It is called a "control". In the case of audio it is usually:
A&B:
No Change
With change
X:
A or B randomised
If you need to be "relaxed" in order to "hear" the differences without "defensiveness" then you are engaging in a psychological process called confabulation.
This is the audio equivalent of the flat earth hypothesis. "Getting through" to some people means they believe they have understood something that they cannot justify rationally: i.e. supersition.
There's no denying that a system has to make you feel something and respond to the music in some way, on some visceral or gut level, not just intellectually, and that's fine and good, just don't say you're being objective when you say something is better than another. If you know the price of the equipment or the details of a system, you are inevitably biased by such knowledge.
Personally, I'd be very happy to know that a $500 speaker sounds just as good as a $5,000 one, some salesmen may not prefer that though.
Now that I think about it, I think we should do speaker auditions blindfolded, it takes out the bias that our eyes provide us with. Don't tell me the price, and I don't want to know what the speakers/setup looks like, just let me listen blind, ABX, and I'll make my best decision, that's the decision based solely on what sounds best to me without any outside factors to influence me.
After I know what I'm listening to, I would do a more in-depth long-term evaluation of the system with my eyes wide open so to speak, and see if it still moves me, if I feel the same way.
We are not speaking of speakers here nor are their beliefs that the same applies to speakers; speakers definatly sound differently no one is claiming otherwise.
Hope dearly that even the last measurement/ double-blind-test ignorant now finally got it …, !!!
What Paul said makes sense to me, and for his use case I think he's right. However for you to prove something as factually correct it needs to be with double blind listening.
Always do ABX testing. It saved me a lot of money by not getting expensive cables because *surprise surprise* they don’t make a difference
Congratulations, you proved to yourself that your ears, your system or your room are not capable of resolving the difference. Therefore, you can save money, by not purchasing more expensive cables.
What you have not even come close to proving is that more expensive cables don't make a difference.
Surprise Surprise
@@markwagner1997 how should my ears be able to tell a difference if expensive measurement gear can’t? Good cables for 50 bucks are as transparent as super expensive cables.
Took and entire day to do a very controlled ABx test on a single $10,000 speaker cable vs 14ga lamp cord It was really freaking hard. The group of 20 "Golden Ear" Audiophiles was completely spent by the end of the day. Conclusion. For the most part we could not decisively tell which one was which. Length of wire was a couple meters.
@@juliangst measurements only show what can be measured by a specific test, not how a cable can effect sound.
For my ears and the level of resolution my system achieves, 100 - 150 dollar Zu speaker cables and Morrow or Canare interconnects made a noticeable improvement. I borrowed some high end cables ($800-$1000) on a couple occasions and once again, my ears and my system did not find any improvement, so I happily did not feel any need to spend more. My experience is not representative of all people's ears or systems, nor is it a judgment on all cables.
@@wilcalint People who buy those expensive cables want to hear a difference so they claim stuff like ‚this cable has more low end‘ or ‚this cable is brighter‘ but in reality all of those cables don’t alter the signal by the slightest amount.
I've heard greater differences between two similar CDs than different transports ... i
When performing AB/x comparisons I find it far better to eliminate the secrecy and reveal just what you ARE listening to ... the 'blind ' bit just confuses you !
The blind bit it crucial, it removes bias. Any non-blind testing is biased, even if you don't realize it (usually you don't!).
I agree. I did an ABX test comparing 16/44 and 24/96 tracks and got 16/16 right by taking my time and really comparing and listening and having full control. As I started to challenge myself to see if I could hear the difference on a single pass, it struggled more. I also saw that the longer I was testing, the more fatigue I got and I wasn’t enjoying the music.
ABX testing helps identify BIG differences. But subtle differences, which admittedly do not represent the “best bang for the buck” do get influenced by the stress of the test. Of course high end audio is never about value since chasing the last 10 percent of performance skyrockets the price.
ABx testing is VERY HARD. The closer the A & B are the more difficult it is to tell if x is either A or B. It can take hours and hours and by that time your completely spent.
That 10% is imperceptible by 90% of human ears. And 99.99% of ears above 40 years old.
I guess I misunderstood how blind abx testing worked.
I would love to play poker with you.
>Reads comments
Welp, time to get the popcorn, there is some drama afoot.
It means the test should be carried out on a person, who has no emotional relationship with tested equipment. Not by the producer or audiophile, but for example by a musician.
It doesn't matter whether the person is biased towards the equipment if they can't actually see or otherwise know what they're listening to, that's the whole idea of blind testing.
You're right that a lot of subjective 'artistic' things cannot be tested in an objective manner. Is wine A better than wine B? Who's to say? We can say that there are differences in how they were made, what types of grapes are in them, when they were harvested etc, but none of this tells us if A is better than B. But here's the thing - ABX does not attempt to answer the question on whether A is better than B - it only attempts to answer the question of whether A is *different* from B. The former question is subjective, and the latter question is very much objective. In the same way, we can run a chemical analysis of the different compounds in wine A and wine B; if we find that every single molecule in wine A is completely replicated in wine B in the same composition and concentration, we are then forced to conclude that wine B is the same as wine A, and any question about whether wine A or wine B is better is moot. There is no way to disntiguish between them, and hence any perceived difference is completely illusory.
Both sides in the argument are right. Subjective sound is only one small part of the "HIFI experience" Visuals are important (valves that light up, big thick interconnects and mains cables, solid marble stands with gold trim), touching physical media (vinyl records, CDs, DVDs etc). I think the point at which you decide an interconnect sounds better is as soon as you make the purchase! If it enhances the whole listening experience then you have achieved an upgrade to your system or more accurately "an upgrade to your experience". Hifi is way more than just sound.
Hah, hah, hah!
Look - you either hear a difference or you don't. ABX testing is a perfectly valid method to determine that.
And so is blind AB
Real gem today. The comfort factor is EVERYTHING! I just got put through the ringer doing an upgrade project on my motorcycle because I became too unrelaxed. The frustration only created more petty stumbling blocks. Made more work for myself and pushed my completion date further. You lose all clarity when you let the stress get to ya.
"When I do a test, I don't want to feel like I'm being tested." What?
Think of it like going to an exam - many people underperform or fumble at exams, because they get nervous as fight/flight kicks in. Guessing it some degree of that he wishes to avoid. (not gonna argue with it, just explaining it).
Paul, on the subject of A/B testing:
Yesterday, you posting a video about the superior sound of DSD over PCM:
czcams.com/video/KMYBd7Ukcuc/video.html
Yet, you did not describe anything about how you conducted that test, other than saying that you record both formats at the same time.
Do you have duplicates of your microphones, analog-to-digital converters, interconnects, etc, all of the same revisions?
Did you use the exact same transport, etc, when you played back the DSD and the PCM files, for your artists to evaluate?
Did you switch boxes, cables, etc. between your two set-ups, to ensure that those identical boxes really do sound the same?
I want to believe you. But life experiences have taught me to be skeptical, even when dealing with well meaning, professionals (they make honest mistakes, too).
Just as The Absolute Sound does, and just as Stereophile does, would you please list all associated gear that you used for both your DSD recording and playback, as well as all associated gear for your PCM recording and playback.
Thank you.
The reality and the sad part is the DSD component may very well be better suited to those other set of components than the PCM component is. Too many variables.
One needs to test the whole setup and see if it works for him/her/it/they and the room it is played in.
It's called snake oil salesman
I think that people prefer the testing method that best confirms their biases. People who live by double blind testing tend to believe that human ears cannot be trusted to make objective evaluations. But ultimately human ears will do the evaluation. Just because a piece of equipment measures well or is created to A/B well against comparable products, it doesn’t mean that it will be enjoyable to listen to long term.
The problem is this believers cannot prove their beliefs all they can do is talk about them, try to convince others etc. If the differences between two amps are so miniscule how important are they? You cant even detect them when you are 'blind' !
Perfectly explained!
I think best way is to listen to the new component on its own to familiarise yourself with its sound for a while before doing any back to back comparisons. Then get yr peak db to 88-90 dbA at the listening position and listen to 30-45 second segments back to back. All the mumbo jumbo you read by some reviewers that this new amp/dac/speaker whatever it is "completely blows away" the one they heard 6 months ago is total nonsense. Aural memory is very short indeed, so trust yr own ears in yr system in yr room. Then do an extended listen to the new gear and ask yourself if the new equipment is more enjoyable, engging, moving and satisfying to listen to than the older one. If not save your money and buy more records or CDs instead.
Makes sense Paul. As you said, placebo (expensive cable alert!) does work, and that's the only risk behind not doing blind ABX testing. It's also important that the subject be relaxed, and as you said if the walls go up, and the listener feels like he or she is the thing being tested, that will absolutely taint results.
That’s an interesting way is saying that you don’t understand psychology testing.
Floyd Toole ……🤔
Great explanation, very good!!
People don't want to believe a lot of stuff that hurts their revenues or ego. Doesn't mean ABX testing is somehow flawed for audio. Selling things is all about manipulation, which is the aim of marketing. The real world however presents far more simpler challenges. The most common problem I see is people listening to music through expensive speakers in horrible spaces, negating the need for expensive speakers; or the second problem is listening to garbage, stuff that won't sound good on the best of audio equipment. The real world works on mass deception through the power of suggestion and shame. Works in politics, art, sound, auto sector, etc.
It is very simple, connect the cable A and B and say the result. There is no need for any emotional aspect in the environment of your home ?! You recognize black color in your house or in the laboratory. What this man claims is that you only see black in ideal conditions. :))) The ideal conditions are not to think with your head but to listen to him. It is easier to sell devices without any professional objective fair test. That’s why abx "doesn’t work" if you ask me..
What you mean is it doesn’t work for you!
Agreed Paul there's to many variables when where taking about emotion.
So if a system's Music stirs your emotions , that is not a good thing ?
@@larrywe3320 that's the point it mucks with your objectiveness when evaluating.
Great when your just enjoying music but not when doing a/b testing. Unless that is your basis for evaluation.
Paul, now that you’re in your 70’s, shouldn’t the critical, final decisions on sound be made by your chosen (younger) appointees? As Joe says, Come on Man.
This has to be the first time that anybody quoted Joe and it not being a joke or was it 🤔😉
I'll, try to be brief and respectful. Paul is wrong on this one, and always has been. ABX testing is NOT to tell which one is better. It's to tell of you can tell a difference between A and B. I will agree with him it's VERY hard and takes the emotion completely out of the process.
I have had the very controlled ABX test experience a number of times and that with 20 other I'll call them "Golden Ear" Audiophiles. IT'S REALLY FREAKING HARD. Especially using things like speaker cables. You can spend hours and hours doing it.
At the other extreme doing an ABX test, tubes vs solid state most of the time is a no brainer. You can easily identify if X is either A or B, not which is better, A or B.
I find power cables and interconnect plays a much bigger role.
@@allansh828 I don't agree with you on the power cable thing as our group did an ABx test on that one. Result, we could not decipher A or B from x. But the interconnects there was some Golden Ears who could successfully pick correct 70/30. Not great but some success.
@@wilcalint Bill, while I agree with most of what you’re saying, I’d say don’t waste too much more of your breath on this line of discussion.
@@allansh828 I find the air density to play a very emotional role.
@@fonkenful I was quite facinating at the time. And a lot of energy was put into it by the group.
When the stake is "i might have wasted tens of thousands of dollars on something meaningless", of course you can get nervous. :)
On the other hand, if you can tell something apart only when you aren't being scientifically observed, how good can that untestable opinion be? ;)
I agree if we’re talking about trying to objectify someone else’s a/b test and make a blanket statement out of it. BUT…if you were blind testing only for YOURSELF, that makes really good sense to me. Why not know the real truth for yourself? My only caveat would be to make the a/b tests long term tests, and not merely quick a/b ing back and forth; there are elements of sound you can’t hear when jumping back and forth.
Using any kind of switch box will color the sound so you can't hear the diferences. I once heard the difference between a switch box and no switch box. It colored the sound a lot.
But the most important point is, whether your equipment makes a sonically difference, at all! If you don’t do an ABX test, you can’t tell!
of course a test that will ruin your business model don't work. What a dumb question.
Comparing the sound from quality electronics such as CD transports or amplifiers is probably a waste of time and may be misleading. Unless there is a defective product then differences in audio may be very subtle. A may sound different to B, but what does this mean? Does this difference mean “better” or “worse”. This is highly subjective. Room acoustics and loudspeaker choice may have far more effect on audio quality and just moving speaker position can affect perceptions of the electronics.
Dear Paul, I don't always agree with your explanations. But this time you nailed, great video! Maby this video will help me with some annoying cable non-believers xD
I think Dr. Floyd Toole would disagree on this fact. Nobody had more experience with the field of psychoacousyics and how seeing the product influences what people think. Almost 20 years at the NRC doing some groundbreaking research on how we hear and what people like . Sorry Paul you're wrong about this one.
Agreed!
ABX tests YOU and not the equipment. If you cant spot a difference, that means YOU cant spot the difference, it doesnt mean there are no differences produced by the equipments.
Tests like these are made to fail you, even if 6 out of 10 or 7 out of 10 people get it right, people will still say that is very near to chance. They expect results to be 9 out of 10 or 10 out of 10 to shut them up. Everybody's sense of hearing is different, those 5, 6, 7 people that pass the test are put under and categorized as 'chance' which is pure bs. Its a game that you will never win.
You'd be right if your sample size is only 10 people, but what if it's 10,000 or 100,000? Not that that's realistic, but if it were theoretically performed and 5,000 out of 10,000 could not hear a difference over not just one track but 20 tracks, then it could be the difference is irrelevant to the human ear and not worth the money. Statistics is not a perfect science, but it can give you a degree of confidence when your sample is big enough
@@Chessmapling if 5000 out of 10000 people could not hear a difference, that means 5000 more COULD. Putting these actual humans under 'chance' is just ridiculous. Statistics are useful but dumb when used this way.
Correct Paul,most people don’t know about the emotions involved in listening 👂 witch I think 🤔 is the most important criteria in audio listening playback pleasure,and having long listening sessions
Now there's budding NASA engineer in the making. I just "feel" this circuit is better. Would you bet your life on that?
I promise I'm not being disrespectful but I'm genuinely trying to find a reason to not find this take anserine.. and can't. Even ignoring the fact that he doesn't understand ABX testing there's a problem when his argument against his misunderstood idea of ABX testing is "I would get defensive" which doesn't mean 'abx testing doesn't work' but at best "I'm unable/unwilling to do abx testing"..
Wild imo...
Lol at all the deafies in the comment section pleading at the altar to ABX testing, as if it is some perfect infallible method that can't possibly introduce biases of its own into the equation. It's alway such a coincidence the people who do this are also the people who do not own actual hifi systems.
Well, as long as you are doing blind A/B, your fine! And, being a Neuroscientist, I can tell you it's the same in Biology and Medicine. I think the introduction was not very clearly worded , and it confused some people into thinking you are not doing blind A/B testing. And of course, you are right, having the "X" added makes things more difficult, with no gains at all for your work.
Not at all. If you have a properly designed switching system so that A/B/X are clickless and quickly switched (time aligned, level aligned, etc) then you allow the use of short-term loudness memory, which is shown to be the most sensitive analysis of any kind of perception involving partial loudnesses.
@@jamesjohnston3643 Loudness should be exactly the same. Clickless is indeed better. All this has nothing to do with whether it is A/B or ABX.
@@AALavdas I refer you to the term "partial loudnesses" not "loudness", in other words, the detected loudness across the basilar membrane as a function of frequency and time.
With a proper ABX, one can most easily detect the DIFFERENT pair, by comparing A with X and B with X. This is still a pairwise comparison in a sense, the key being (as always) to listen for difference, rather than similarity.
A/B only offers you "same" or "not same", and does not give you the permanent A and B references (A and B should not change identity throughout a single test series, although they can move in another test series) and by doing so, removes the sensory anchor that makes hearing the difference an easier task.
@@jamesjohnston3643 Well, saying "across the basilar membrane" does not add anything to this sentence. And there is no way to know that, unless you have a probe on the basilar membrane. I agree that there are different merits in different approaches, but blind A/B is fine.
@@AALavdas Well, I didn't come here for an answer, but I didn't expect to encounter somebody who denies settled science. I would suggest that you check the works of Zwislocki, Greenwood, Zwicker, Allen (Jont, not Jon), Hall, and others as far as being able to estimate, at various levels, the excitation on the basilar membrane. You are completely incorrect in your dismissal.
Furthermore, you might learn what "partial loudnesses" are. I can create two very different signals that have the exact same loudness (in sones, phons, whatever you want) that will have entirely different partial loudnesses. Allen's work on self-training tests shows, conclusively, that proximate, clickless comparison at the subject's will provide the smallest dl's.
All hail Topping DAC's !!!! Ops, mine just broke.... It might have the best SINAD score but does it sound better? If perfectly flat frequency response is your thing, try to listen to a perfect 40 RTA car audio. I love my tone control and spices in my food thank you.
omg - - the comments are soooo predictable - - thanks Paul, this vid was certainly insightful, reassuring actually, at least at some level, simple listening tests are taken into account - - I'd bet you could spend an hour going over all of the "other" testing that must go on at PS audio - - so all those commenters that are falling off of their chairs to criticize, take a moment and try to imagine all of the "other" testing that of course goes on well before PS Audio brings any component to market. Ya think Amir is the only one out there with a high end analyzer and oscilloscope?
I agree that blind AB testing is useless. To really gauge a piece of equipment you need to live with it for a while not 5 minutes.
A more in your face sound will grab your attention in a quick comparison. After living with that sound for some time you may find that is ultimately hard and fatiguing. This is why most budget hifi sounds so awful. It is designed by the manufacturer to grab the attention so that it's the one you choose in a quick demo.
Well, but he just ended his explanation saying he uses blind AB testing, but not ABX. Scientific methods are quite useful in industrial environment. At home you use your emotions, in a factory you test the hell out of it.
Blind AB is absolutely necessary, and that's what he does. He does not do ABX
So Paul you basically admit that audio perceptions are due to emotional states which are not to be standardized operationally? Nice argument in order to circumvent the need to objectify claimed acoustic differences of hifi gear. Apart fr that you confuse a randomized control trial with ABX testing. Again you are arguing with anecdotal evidence and heuristics. The confidence needs to be derived from the perceptions. If that does not work for you, you admit you do not hear the differences. If you hear them, you will hear them in the ABX. Nice that you admit that. All I needed to know.
lmao the point of such testing is to test if a medicine is *more effective* than the placebo...
That is exactly what ABx testing is NOT. And that's where Paul is wrong.
LOL!!! You object to a test for being tested? That is sadly funny. The whole point of ABX testing is independence. You are right in that highend is emotional, and that is why so often people are fooled. Take the emotion out and you get to actual differences (or lack thereof). ABX or bust. Amir recently did a good video on ABX testing, supported with actual data. Now that is a good video.
Paul is just a salesman
No, he objects to a scientific test conducted unscientifically. You simply can't eliminate emotional response, or tastes, or preferences, from a blind test of audio equipment. ABX testing is inherently flawed where it involves humans expressing an opinion. You cannot take the emotion out of subjective listening, blind or otherwise. Data-driven testing is a completely different thing.
@@richardt3371 One can easily test for emotional preference. Which joke is more funny? Ask 100 people, and you will get an answer. Or the infamous Coke vs. Pepsi test in neutralized containers. You can see whether people can tell the difference, and then state a preference. And you can evaluate that statistically. No problem. The same applies to audio. Can you hear a difference between two cables/DACs, etc. and which one sounds better to you. The problem is, that most people cannot hear a difference, and that is a problem for sales.
@@danielgeiger7739 Sigh. You're missing the whole point of blind testing, which is designed to eliminate preconceptions or tastes, and which is why it is of no use whatsoever as soon as you introduce emotion. Asking someone if they prefer Coke or Pepsi, and testing them blind, gives you no useable information - you can't "evaluate that statistically", what you end up evaluating is human response, not whether one is actually better than another. It's an interesting experiment but it's also dead data.
@@richardt3371 OK: statistics 101. First you can statistically figure out whether one person can distinguish between Coke and Pepsi. Give them as much time/drink as they want to familiarize themselves with each, then do the blind test where they need to assign the sample to either coke or Pepsi.> 8/10 correct, they pass.
Re preference, that is a n =1 issue, of course. So you get 10 (or more) people who can distinguish the two, then determine their preference. Then you see whether there is a population preference for Coke/Pepsi. If there is a >= 8 vs 2 distribution, then there is a population preference. Same can be done for cables, DACS etc. If it is
Yeah I can understand your argument, and I find it valid. But it's sad to see a professional that aways use technical information to explain things suddenly warp words to try convince people of his argument, instead of just presenting it as is. Placebo work sometimes, yes, but never "more often than not". Especially when talking about medicines, you need to be way more responsible than that.
I clicked bec thumbnail
A very good explanation. Thank you Paul.
Love this answer, Paul - absolutely love it. The number of times I'm asked for ABX testing of the products we install and it really upsets clients to be told no, we don't do that. Fire-testing? Sure. Soundproofing? Sure. Those are scientific, quantifiable, data-led comparisons, and have validity. Does one sound "better" than another? With the back and forth: Now try that again? Now are you sure you're listening to modular type A or type B? Nope - it's too nebulous and too open to interpretation and preference.
Audio is emotion, it's emotional response. Getting bogged down in technical specifications is clinical and simply doesn't work. I hate it when posts on here scream for "ABX testing data please!" Nope.
Don't you see how conflicting this is coming from someone who sells expensive audio equipment? Of course emotion is highly important. We all know that but most products are sold with pseudo-scientific, emotionless oxygen-free quadruple-shielded false mambo-jambo.
Makes perfect sense to me. The concept of a double-blind test appeals to my scientific side, but as Paul relates his experience I can certainly identify with his ‘cognitive dissonance.’
It would mean nothing outside the clinical concept. "Double blind" means that both the patient and the investigator are blinded. There is no "double blind" outside clinical medicine. There is no patient here, just investigator. Even if there were 10 investigators, it would still be just "blind test" if they are all blinded.And that is enough.
@@AALavdas, it would be double blind testing if you called a bunch of your friends to come to you and listen to some equipment comparatively, and you yourself wouldn't know at which point which equipment is being listened to. If you - the organizer of the test - did now and only your friends - the listeners - didn't, then it would be simple blind testing (not double blind).
So if there is no way to know that you hear a difference you do hear it.. but if we can actually test it you get to stressed to hear the difference? That is very convenient. Just saying.