Blind Testing

[Clive]

There a few fora where blind testing has a fervent or even fanatical following. Some people seem to be quite rigorous about this whereas others only seem to ask for it where they don't believe differences exist.

Personally I see it as a hassle and indeed there's no one else in my household who would willingly assist the process. Also I find short-term A/B comparisons only tell me about gross differences but many differences that really count need longer auditioning; this requires living with the product for a few days at least. It would be possible to blind test over matter of days but it becomes somewhat impractical.

Blind testing, speakers, arm and cartridge combinations etc would seem problematic too though any obstacles can theoretically be overcome with money and effort.

For my part A/B sighted comparisons combined with longer term listening work well.

Which camp do you sit in?

Maybe we need a blind test poll but we can't vote if we can't see the screen.

[AlanS]

Aren't blind tests conducted by third parties, manufacturers, reviewers and allow a true freedom from coloration. influence of ones thinking, prejudices/favourites.

I'd love to do at least one just for the education it can provide. May golden eared persons have all sorts of reasons not to.

Why are you fussed?

Assess your new kit as best you can. Just avoid making it's amazing claims unless you want the chat

[Clive]

Aren't blind tests conducted by third parties, manufacturers, reviewers and allow a true freedom from coloration. influence of ones thinking, prejudices/favourites.

I'd love to do at least one just for the education it can provide. May golden eared persons have all sorts of reasons not to.

Why are you fussed?

Assess your new kit as best you can. Just avoid making it's amazing claims unless you want the chat

There seem to be a number of people insisting on blind testing for any statements made. I don't mean "wow this is amazing", I mean "this product is a little richer sounding" or whatever. Some of the comments on a few other fora essentially are "if you don't blind test then don't make any comparisons".

Why am I fussed; I'm not THAT fussed but these attitudes stifle discussion.

I do use my own method, as I described. It's not a tightly defined process but I've developed what works for me over 35+ years of building, buying and listening to music. I don't say this is what others must do, whereas the blind testers require others to follow their mantra. I don't find A/B testing (blind or otherwise) to be the only tool in the box though it has its place.

BTW, I tend to steer away from "amazing" claims, there's little that is truly amazing in reality. Everything has its place and price point.

[AlanS]

There seem to be a number of people insisting on blind testing for any statements made. I don't mean "wow this is amazing", I mean "this product is a little richer sounding" or whatever. Some of the comments on a few other fora essentially are "if you don't blind test then don't make any comparisons".

Why am I fussed; I'm not THAT fussed but these attitudes stifle discussion.

I do use my own method, as I described. It's not a tightly defined process but I've developed what works for me over 35+ years of building, buying and listening to music. I don't say this is what others must do, whereas the blind testers require others to follow their mantra. I don't find A/B testing (blind or otherwise) to be the only tool in the box though it has its place.

BTW, I tend to steer away from "amazing" claims, there's little that is truly amazing in reality. Everything has its place and price point.

If you qualify your posts with In my Humble Experience or in my system, and say how you came to your view. The thing that pisses me off are folk who make a simplistic statement about something without trying it anywhere else then become offended because you don't accept their word as truth for all.

[Gordon Steadman]

I find the discussion a bit pointless for me.

My only criteria for my hi-fi is, when I sit down to listen, do I get drawn into the music and unable to leave or can I walk away and start talking about the system.

I have heard many systems over the years and very few have held my attention for very long. One of the worst I ever heard was Krell amps driving Watt speakers. I couldn't get away from it fast enough, no doubt cost a fortune. I don't really care how they measure or compare in the micro differences. Its all in the music.

My amp changes tomorrow. I sat in front of the Quads with the Firebottle OTL inserted rather than the Leak and it just rocked. Bach was pure jazz and I could not get away until it finished. First change in twenty years. We shall see how I feel long term.

[Oldpinkman]

My wife is always asking me "what am I supposed to hear" or "did I get it right". Well, was. She has realised this stuff does sound great and is learning to trust her own judgement. I think there is truth in both camps. The real test is extended listening pleasure. I think back to my early (and surprisingly current) digital experiences. I could listen to CD's and AB test, and hear stuff - but I never sat down and listened to a CD all the way through. If I was listening on CD, I would always need to make a cup of tea, or phone a friend, or air the dogs. I could listen to vinyl albums back to back for hours. That amounts to my description of digital "hardness". It shows up on a soak test like that - not a quick AB. That said, blind testing is useful for exposing delusions, including your (my) own. It is very easy to hear what you see, or think you should hear. Blind testing helps distinguish real differences from prejudice where a true AB is possible. Of course, you need a few others to make it work. But it was how we did it at Pink Triangle. And if the (immediate AB) differences are real, then they survive a blind test. But its more a tool for manufacturers. As a listener, what matters is what you hear. And if you like it - its good!

[DSJR]

I try to use both - get a basically sound and well engineered design and then listen to music reproduced by it to see if the heart is engaged as well as the head. Good sounding products with obvious technical flaws tend to have been "voiced" around one particular parameter, the rest not being as good as this one particular aspect - IMO currently.. This is why good old gear can stand the test of time - decades in fact - and still do the business, because the design was a well balanced one, if a little more compromised than more recent ones made with up to date components. The Quad II is a classic case in point where EVERYTHING from the wires used to the values of some critical components was taken into account. Change these at one's peril as the (technical) balance goes and the results are readily audible.....

[Clive]

Some good points. I would concur that A/B testing, sighted or otherwise helps with identifying obvious technical flaws or inconsistencies, this relates to my head, not my heart. Longer term listening enables me to conclude whether my heart strings are tugged. Of course it's the music itself that moves us but the equipment has to allow it to happen.

[DSJR]

Thanks for saying what I cannot

[Eremite]

“Blinding” is a technique that is sometimes applied to an experimental design in order to control for various types of effects that would otherwise bias the results of the study. So, in order to answer the question of whether blinding is useful in a certain situation, it is important to be precise about what the question is in the first place!

If you have a question like “can people detect a difference between these two amplifiers” then there are a number of well-known effects that have to be controlled for in the experimental design. As examples, hearing is remarkably sensitive to amplitude effects - therefore levels must be matched very closely; auditory memory is very short term - therefore you need to use an ABX type design (or similar) that doesn’t stress the auditory memory too much; the sense of hearing does not operate independently of the other senses (or of higher order sense processing) – therefore you need to isolate the subject from these other sensory cues (using single blinding, for example); experimental subjects can be very sensitive to cues from the experimenter – therefore you need to isolate the subject from these experimental cues (by double blinding, for example).

However, there are other questions for which you positively don’t want the subjects to be isolated from certain effects. If you ask a question like “do you prefer this hifi setup to that one” then you’re asking about the complete experience, which includes all the sensory interactions that humans have. A pig ugly piece of equipment that sounds good may not be as acceptable as a beautiful piece of equipment that sounds just OK. Notice that we could, in principle, separate out the factors that contribute to the total experience; to do so would require a complex experimental design with some bits blinded and other bits not.

Furthermore there are some questions for which blinding is not possible. For example: “do you prefer the sound of this CD player or this turntable”. The sources can be reliably identified by the subject therefore they cannot be blind to them. This effect leads to some of the indirect experimental designs that are used to compare digital and analogue.

So, tl;dr version, whether or not you use blinding in an experimental design depends crucially upon the question you are attempting to answer. In some circumstances if it is not done, your experiment is worthless; in other circumstances it doesn’t matter at all.

So, in a way, being “fanatical” about blinding is neither here nor there; one either has a correct experimental design for the question in hand or one does not.