STATS, STAT!
Were the results of my AAC Challenge below statistically significant?
Dave has theorized that the sample size was too small. But even with this sample size, it is possible to get numbers that statisticians can use. Of course, a larger sample size will give us greater accuracy... more decimal points.
Eric from Geek Farm - Professional Statistician - has this to say:
"By guessing randomly, you would have gotten 5 out of 10 right. We want to know the odds that if you were guessing randomly, you would have gotten 8 out of 10 right.
The odds of that are about .044 (4.4%), which is statistically significant. Ususally anything less than 5% is considered statistically significant. To give you a perspective, that (5%) is 1:20 odds."
I'm not a betting man, but I like them odds.
12 Comments:
Funny!
Fortunately, I used no chads in this test.
I'd still call this anecdote, not data.
I'd like to see a 500 person study.
Get us a grant, Dave, and we'll do the study.
Of course we can't fit 500 in the control room at S&M, so we'd need a larger facility, so the grant would have to be sizable.
Why don't you ask your precious Federal Government for a handout? Just say that it promotes audio "diversity" or something like that.
Dave, it doesn't matter what YOU would call it, it's still data.
The point of the study was not to see if "the average person" could tell the difference between an AAC and an AIFF. The point was to see if I could tell the difference, and according to a statistician I can.
It makes no sense to run the study with 500 people. Dolby did not say that a random selection of people can't tell the difference.
Hell, the average person can't tell the difference between an MP3 and an AIFF.
I am more apt to listen to someone who looks at data for a living. Sorry.
In any case, I could easily compare a larger number of tracks which would obviously make the data more robust. Maybe I'll do that. Maybe not.
Your handouts comment is funny, considering the amount of money the people you voted for hand out to their constituents every day.
Technically, it's a datum.
BTW, Dolby's claim was that "expert listeners" couldn't tell the difference. I was proposing a study of 500 expert listeners to make the data more robust, as you say.
I also look at data all day. Clinical trial data. And I would draw no conclusions from a study where n=1.
This is the problem with statistics.
Our "study" originally came up when Ken wondered if I could tell the difference between an AIFF and an AAC. When I did some reasearch on AAC's I found Dolby's claim.
Now, we don't know what data they used for this claim. Perhaps it was just a datum. I can't hope to dispute the veracity of this without knowing the details. Perhaps the whole thing was anecdotal.
So our "study" was based on n=1. We weren't trying to dispute Dolby (although that's certainly how I framed it when describing it, which didn't help) but to answer Ken's question. Could I tell the difference? And it looks like I could. Still, ten songs is a small amount.
Maybe Dolby did have 500 experts listen, and a statiscally significant portion couldn't tell the difference (I still find this unlikely, but it doesn't matter). But we don't know.
You are absolutely right that our "study" is not able to prove anything about Dolby's claims. We don't even know if they were testing music (it's easy to imagine plenty of audio sources that's limited in its frequency response and that would be identical after conversion to an AAC).
I certainly wouldn't trust our "study" to be anything more than what it is - looking at whether one person could hear a difference. It shouldn't be extrapolated to anything more.
I think we have hit on the problem.
I have no doubt that your sample shows that you are able to distinguish.
I was only talking about Dolby's bold statement.
Let's email them and ask for their trial protocol.
I was thinking the same thing.
Let's do it.
It's funny, every time a new piece of digital audio technology comes out the industry says it offers "perfect sound reproduction" (remember they said this about CD's, even as many audio engineers were complaining about the sound being significantly degraded, particularly with the older generation analog to digital convertors). Then a bit later they improve upon it... or replace it.
There is no such thing as perfect sound reproduction, of course. What are we reproducing? The sound of a flute from 1 foot? 10 feet? 100 feet?
Let's get back to boobies.
The reference to the chads made me chuckle.
I just had to say that.
Post a Comment
<< Home