The journey from there to here

I was speaking the other day with a rather intelligent friend on the issue of homeschooling. This friend is scientifically inclined (important in the context of this piece), and he commented that he had heard a piece on NPR about how homeschoolers are below their peers academically (he cited it as an "extensive study"). Now, his citing of NPR as a source immediately sent up red flags. NPR has a pretty consistently left lean, and the left has a recent history of virulently attacking homeschooling (ironic, as the modern homeschool movement was the product of LIBERALS disenchanted with the educational system, not of religious conservatives, as many believe).

Anyway, I stated immediately that I would need to get my hands on the hard data to speak conclusively on this study in particular. After all, a number of states have laws mandating standardized testing for homeschool students, and in EVERY ONE of these states, the homeschool students consistently outperform their peers. The "study" he referred to, if true, would be absolutely earthshaking as its implications would controvert all previously released data.

His response surprised me.

He accused me of being "defensive" and stated he was only relaying information he had heard. Outside of "NPR", he was unable to cite a source. As a scientifically minded individual, I would have thought he would have respected the fact that I wanted to look at hard data and not conclusions. As well, he should have been more ready to provide a source.

Thus, I came online. In searching NPR's archives, I was unable to find A SINGLE reference to this alleged study. I was equally unable to find such a reference through a google search or the Homeschool Legal Defense Fund's webpage (he cited it was done by a "large national homeschool organization", and that the evidence they found went against the evidence they were trying to find. Well, as much as I loathe the HSLDA, they ARE the primary national organization that could truly be considered "large", and I would have expected to have found at a minimum a rebuttal to the data).

It would seem to me that news as groundbreaking as this would be accessible in the NPR's archives. It is not, however, making me question its veracity entirely.

What disturbs me most, though, is this only goes to underscore the inadequacy of what we call research. We rely primarily on secondary and often tertiary sources without a thought of analyzing the raw data, which is almost always far more revealing than the "conclusions" (conclusions are often subjective; filter them through a couple of layers and they quickly become even more so); let alone that a scientifically minded individual would be offended when I would solicit said raw data.

It's a crazy world sometimes.

Respectfully submitted,

Gideon MacLeish


Comments
on Dec 06, 2004
It seems like your friend is not as "scientifically inclined" as you thought.

What disturbs me most, though, is this only goes to underscore the inadequacy of what we call research. We rely primarily on secondary and often tertiary sources without a thought of analyzing the raw data,


If a vague and passing reference qualifies as "research" then we are in big trouble. In a truly scientific or academic discussion, you friend's unsubstantiated claim would not be taken seriously. But it does seem like it was more of a 'friendly' discussion, which I think, points to an even more disturbing trend (for lack of a better word). That is, what passes for legitimate information anymore, being that we do rely on so many various sources, for even our most basic information. This can make having even the most regular conversation about current events like talking about fairy tales.
on Dec 06, 2004
NPR has a pretty consistently left lean


I would have to disagree on this one. Even though you could consider most people who end up talking on NPR as "damn dirty hippies", I listen to NPR instead of watching a lot of TV news shows because I find NPR just "presents things things that happen, and lets you decide" more than the heavily left-leaning TV news.
on Dec 07, 2004
You probably need to seperate research into Research, scientific research and peer reviewed publication.

Research covers any topic that any person spends some time looking at. It in no way gaurentees the accuracy of that research, or any impartiality or any acknowledgement of limitations.
Scientific research is a step beyond normal research as it uses scientific methods to attempt to provide some degree of accuracy and some indication of limitations.
Peer reviewed research adds an extra level of examination tightening up issues of accuracy, limitations and impartiality.

The biggest problem with peer reviews is that anyone not in the area of research has no way of knowing who the peeers are. So someone could publish research in a totally biased peer review journal and only those people who worked in the field would be aware of this bias. Others would be willing t obelieve the research was unbiased.

In your friends case, it sounds like he's spouting garbage. He's citing a second hand report as true which is totally poor form. he has no way of knowing how biased the NPR piece was of the original material and he seems to have no clue as to the level of scientific quality and peer review in the original data. If he stood up at a scientific conference with such poor knowledge of the facts he would be laughed off the podium.

Paul.