PDA

View Full Version : ABX/Double Blind Testing; an objectivist’s criticism.



Welder
19-08-2011, 15:36
I like a bit of measurement ;)
I have about as much faith in my hearing as I do in my eyesight and even less faith in the processing unit called the brain. Go picking mushrooms or get yer pharmaceutically supplied version and then come back and tell me you can trust your senses. The brain reacts to chemicals and the body produces different chemicals under different circumstances. They all effect your perception.

I’ve read stuff about the placebo effect and stress under listening tests and while I’ll grant they have their influence I don’t think that’s the problem here.

What seems to get overlooked here is we are often listening to subtle differences. Sure, swap out a Roberts Rambler radio with my current system and I’m pretty confident I’ll be on it. Swap something that produces a subtle change and it starts getting much more complicated.

Sit me in front of a strange audio setup and play me music I’m not necessarily familiar with and swap bits about, it wont matter if I’m valium calm and had my ears syringed out that morning, I’m unlikely to be confident about what I hear. How can I be? I don’t know what I’m listen to, or listening for :scratch:

Me and a few friends tended to descend on one of our group on a regular basis to play music and socialise. During a debate on the subject of double blind testing etc we came up with this scheme.

We decided that he should paint a half metre square section of the wall in his music room a different shade of yellow every day, or not, and we should all fill in a log stating whether we thought the colour had, or hadn’t, been changed that day. This went on for two weeks.
Given usually about six of us tended to gather there most days at some point, and we felt comfortable in the environment and on average spent a couple of hours either tweeting, parpping, banging or twanging one sort of instrument or another and listening to tunes we were hardly under stress and the “experiment” kinda blended into our normal routine.

There are many shades of black…
IbMqqtnvLTY

.erm, yello, especially if you’re a bit creative with a mixing pot.

At the end of the experiment it was quite apparent we had been guessing. Interestingly perhaps the only person who got anything like a positive result was the guys wife, largely attributed to the fact that she apparently had to clean up after the painter and prolly had plenty of time to study the colour in depth as she was the one who scrubbed the paint off the floor, kitchen sink, etc and had the advantage of a constant reminder by way of unusual nail varnish as we got reminded, often :lol:
Anyway, I digress.

It wasn’t until every colour change got painted side by side on the wall that the changes became more obvious; essentially no memory involved, instant comparison; something impossible with sound I suggest unless perhaps you play single tones through two sets of identical equipment bar the one component. Not a lot of point in that unless you listen to sine waves for pleasure. It hardly represents the complexity of music.

Hopefully most of us have a feeling for what our particular setups sound like. We acquire this feeling over time and experience. I know for example from my music server project that I couldn’t even begin to give a meaningful description of the changes I hear compared to say my laptop and if they got swapped about in an ABX style test, I wouldn’t be confident of picking one from the other. The same applies to many of the changes I’ve made to my system over the years, some I just prefer the sound of over time and if I revert to a previous configuration it just doesn’t sound right.

It’s a bit like that feeling when you open the door to your home and know that someone else is at home or a vehicle you drive a lot that isn’t performing as usual. In my experience these feelings are often right and there is a change from “normal” but you just can’t put your finger on what it is and if questioned you may well shrug that feeling off but, you’ll be under the bonnet or checking the house later.

By now the subjectivists may be joining hands and singing. But wait, I did say I didn’t trust my ears and it seems from the above the eyes aren’t too reliable either.
ABX style testing just doesn’t hack it as scientific measurement in my book. It failed miserably in our visual test and there were changes to be seen. It fails miserably for audio as well, certainly in the test conditions I’ve read about and the very basic experiments I’ve carried out with my mates.
That leaves measurement and until we can measure what we can’t atm there is only opinion and one is as good as another.

HighFidelityGuy
19-08-2011, 16:26
I may join in with this discussion later but for now I'll just link to this video which I think illustrates John's point perfectly: LINK (http://www.newscientist.com/blogs/nstv/2011/06/friday-illusion-can-you-spot-the-change.html)

I saw that video a few weeks ago and this thread just reminded me of it. It certainly proves that we can't always trust our senses for one of many reasons and it gives you something to think about.....

DSJR
19-08-2011, 17:39
I like to have a reference from which to judge, otherwise I have no idea where I'm at with audio. I'm not saying I have to LIKE this reference, but a known quantity can act as a benchmark I feel. One reason why I keep the AVI preamp for example, as I used the basic well designed neutrality of this in my limited tube rolling experiments on the Croft, which now sounds "neutral" but with that "extra" that Glenn's designs always seem to have, instead of bloomy and soft, as it did when I received it.

Ali Tait
19-08-2011, 18:10
The only way to really see for me is to try something new for a few weeks and then go back to what I was using before to see what the difference is.

Themis
20-08-2011, 06:17
I've come to the conclusion that ABX is of no use for the audio enthusiast that I am.

- Because it's too complicated to setup up properly the prerequisits for actually making a valid test
- Because it doesn't help me choose a better (to my pleasure) equipment. It merely can help me decide whether two bits are "different". Who cares ? I don't in most cases. Measures tell me more about if and where two bits are different. ;)
- Because it's statistically incomplete, and a "failed" session teaches me nothing. Life is too short to waste it this way...
- Because it obliges me to test equipments in a way that is the exact contrary of the way I listen to it in everyday use : therefore, any conclusion cannot be applied to my everyday listening.
- Because it is impossible to use when testing -say- 4 different equipment. Well, not actually *impossible* but abnormally long.

But, ABX experience is useful for learning a few fundamental things concerning audio comparisons :
- Correctly assigning "common mistakes" during comparisons (for instance: output level when making quick tests) :eyebrows:
- Determining the limits of short-term hearing memory
and, above all :
- Making the difference between "hearing sounds" and "listening to music"

I'm sure I've forgotten a few things, and I'm certain I treated some points too concisely, but that grossly sums up my feeling about ABX. ;)

Effem
20-08-2011, 21:09
I could never understand the logic behind attempting to conduct a test where an objective outcome is presumed, when in reality there can never be anything but an entirely subjective outcome :scratch:

Anyone for a game of table tennis?

Dr Bunsen Honeydew
20-08-2011, 21:36
I know what is better and what I like normally within 3 bars, and sometimes if its gross within 3 notes. I see zero point in anything else, even when I did about 7 years of regular blind test as part of the Hi-Fi Choice blind speaker test regulars, it still only took 3 bars. Paul Messenger could have just done it himself, but for some reason they (Hi-Fi Choice editorial) seemed to think blind testing gave them credibility. A waste of time, but fun taking the piss out of the marketing men who turned up for one sesh, and the pub lunches were good.

Marco
20-08-2011, 21:46
The only way to really see for me is to try something new for a few weeks and then go back to what I was using before to see what the difference is.

Aye, the true enthusiast's method - highly effective and successfully tried and tested over years! Bollocks to yer blind ABX testing. Keep it for the 'scope squad'...! :ner:

Marco.

sq225917
20-08-2011, 22:56
I'm pretty simple, if I can't hear gross differences immediately, sighted or otherwise then i'm not going to bother listening to a bit of kite any longer, the differences aren't worth worrying about. If they are that obvious then I'll try and blind test them to see which i really prefer the sound of.

I've done quite a bit of blind AB and ABX and the one thing I have found is that when it really gets down to the nitty gritty I'm only really any use in being able to tell fine changes apart if I can swap them over immediately, anything more than 5 seconds and it's gone. Some thing just aren't worth chasing.

Dr Bunsen Honeydew
21-08-2011, 02:01
When you are making changes to a product in the design process there is hierarchy of prominence amongst the changes. Quite often an improvement wont be heard until you have created other improvements first.

I call it the cleaning the window process. The more dirt on the window the less you observer through it, but until the gross is removed then the subtle is not noticed, so is said to not exist. But with a clean window even the most subtle of changes are noticed.

Blind doesn't help in this process as you need to know what you are doing, in fact it is probably a direct hindrance.

Themis
21-08-2011, 06:12
There's an example in psychology tests:

Male subjects are asked to compare two identical photos of a woman, determine the differences and say which one they prefer.
In fact the vast majority can't find any difference. The two photos are identical.
Nevertheless, a big (statistically valid) majority prefers one of the two photos.

At the end of the test, you find out that the photo you prefer has the pupil of the eye slightly larger, which is interpreted as a sign of desire.

When you are told about the difference, then you can easily see the the two photographs are not identical.
In this example, you can cheat the eye, but you can't cheat the brain. ;)

Marco
21-08-2011, 06:31
I like that one, Dimitri! :)

Marco.

Macca
21-08-2011, 08:37
I know what is better and what I like normally within 3 bars, and sometimes if its gross within 3 notes. I see zero point in anything else, even when I did about 7 years of regular blind test as part of the Hi-Fi Choice blind speaker test regulars, it still only took 3 bars. Paul Messenger could have just done it himself, but for some reason they (Hi-Fi Choice editorial) seemed to think blind testing gave them credibility. A waste of time, but fun taking the piss out of the marketing men who turned up for one sesh, and the pub lunches were good.

Hi Richard

I'd be interested to hear more about your experiences on the Choice review panel, you must have some good anecdotes to tell?

Themis
21-08-2011, 09:22
Hi Richard

I'd be interested to hear more about your experiences on the Choice review panel, you must have some good anecdotes to tell?
Yes, please, Richard ! :mex:

Dr Bunsen Honeydew
21-08-2011, 09:40
The team consisted of regulars invited by Paul and industry volunteers from companies providing product. They were the funny ones, and there were at least two at each sesh. The entertainment was having a laugh and taking the piss out of them without them realising it, and sometimes Paul would be sitting there trying to be serious and having great difficulty keeping a straight face. Terrible to mock the afflicted, but they wanted to come and throw their marketing bullshit around even though to man they hadn't a clue, all they were interested in was promoting and protecting their jobs, so they asked for it. They were also the ones who didn't even recognise their own companies speakers.

The fly in the ointment was Jason Kennedy, miserable git! who tried to keep everything in order, even though he was only there to change the speakers behind the black curtain.

Down to an old smugglers pub in the Thanet swamps for lunch and real ale, so the afternoon sesh could be a bit funnier than the morning one.

I had my Alfa Spider (old type) back then so part of the fun was the journey.

Anyway no names no pack drill. Though Guy Sergeant did a few, and he seemed to have difficulty taking it too seriously as well.

pure sound
22-08-2011, 16:12
I could participate with some detachment as we never had any speakers in the group under test. Naim & Snells would never have worked so we didn't bother to submit them. Sometimes the representatives of the major speaker companies were sales guys rather than r&d so they didn't always have a handle on what their r&d departments had been trying to do. I don't really have an objection to that kind of blind testing. Loudspeakers could vary quite alot. But even when amplifiers or cdplayers (or indeed cables) were done at Paul Miller's place, the differences were quite evident between 6 or 7 supposedly indistinguishable (measurements wise) products. It seems that it's the A vs B vs X type testing that seems more stressful & difficult to glean meaningful results from. Listen blind to a group of 5 or 6 CD players & characteristics that are repeatedly identifiable soon emerge.


I always enjoyed the occasions though particularly at Paul Messenger's and that pub did do a great lunch.

http://www.telegraph.co.uk/sponsored/foodanddrink/top-100-famous-pubs/8105544/The-Gate-Inn.html

Dr Bunsen Honeydew
22-08-2011, 18:11
I missed it for a while, but had fairly regular contact with Paul but we haven't been down there for two years now. I heard he started doing the blind tests again for Choice again and was mumbling about me going on the panel again, but the advent of forums has made me a bit of a poison chalice for the "normal" industry. Though if Choice closes as is rumoured I haven't missed much.

You may be interested that I got myself a pair of Advent Large speakers on ebay a week or so ago for a silly price. Always wanted to have a good listen to them as they sound my cup of tea, and for a pretty mint pair in solid walnut (though I bet the drivers need new surrounds) the price I paid was criminal :):eyebrows: I get the impression that the Snells had influence a bit from the work of Henry Kloss. I just have to get them picked up or send the seller some boxes.

http://www.ebay.co.uk/itm/290593601978?ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1439.l2649#ht_500wt_1156

Marco
22-08-2011, 18:20
I hope you enjoy them, Richard (I'm sure you will), but I had to giggle at what some people describe as "large" speakers! :eyebrows:

Marco.

Dr Bunsen Honeydew
22-08-2011, 18:25
They were only called the large speaker because they started to make a smaller one. Henry Kloss was a man after my own heart, a complete non bullshit merchant, his products spoke for him.

See here http://www.davidreaton.com/Advent.htm

Marco
22-08-2011, 18:29
They look interesting... I'll take you're word that they're good! :)

Marco.

Dr Bunsen Honeydew
22-08-2011, 18:36
As I said I am going on what I have been told, I haven't heard them as they are rare in the UK. I will report when I get them. If they are as good or better than AR3a's as many said at the time, then they are good.

Marco
22-08-2011, 18:39
I will report when I get them.


Please do, as I'd be interested in your thoughts. You certainly got them for a ridiculous price!

Marco.