Sunday, May 20, 2012

how reliable are the social sciences?

NYTimes | Public policy debates often involve appeals to results of work in social sciences like economics and sociology. For example, in his State of the Union address this year, President Obama cited a recent high-profile study to support his emphasis on evaluating teachers by their students’ test scores. The study purportedly shows that students with teachers who raise their standardized test scores are “more likely to attend college, earn higher salaries, live in better neighborhoods and save more for retirement.”

How much authority should we give to such work in our policy decisions? The question is important because media reports often seem to assume that any result presented as “scientific” has a claim to our serious attention. But this is hardly a reasonable view. There is considerable distance between, say, the confidence we should place in astronomers’ calculations of eclipses and a small marketing study suggesting that consumers prefer laundry soap in blue boxes.

A rational assessment of a scientific result must first take account of the broader context of the particular science involved. Where does the result lie on the continuum from preliminary studies, designed to suggest further directions of research, to maximally supported conclusions of the science? In physics, for example, there is the difference between early calculations positing the Higgs boson and what we hope will soon be the final experimental proof that it actually exists. Scientists working in a discipline generally have a good sense of where a given piece of works stands in their discipline. But often, as I have pointed out for the case of biomedical research, popular reports often do not make clear the limited value of a journalistically exciting result. Good headlines can make for bad reporting.

Second, and even more important, there is our overall assessment of work in a given science in comparison with other sciences. The core natural sciences (e.g., physics, chemistry, biology) are so well established that we readily accept their best-supported conclusions as definitive. (No one, for example, was concerned about the validity of the fundamental physics on which our space program was based.) Even the best-developed social sciences like economics have nothing like this status.

Consider, for example, the report President Obama referred to. By all accounts it is a significant contribution to its field. As reported in The Times, the study, by two economists from Harvard and one from Columbia, “examined a larger number of students over a longer period of time with more in-depth data than many earlier studies, allowing for a deeper look at how much the quality of individual teachers matters over the long term.” As such, “It is likely to influence the roiling national debates about the importance of quality teachers and how best to measure that quality.”

But how reliable is even the best work on the effects of teaching? How, for example, does it compare with the best work by biochemists on the effects of light on plant growth? Since humans are much more complex than plants and biochemists have far more refined techniques for studying plants, we may well expect the biochemical work to be far more reliable. For making informed decisions about public policy, though, we need to have a more precise sense of how large the difference in reliability is. Is there any work on the effectiveness of teaching that is solidly enough established to support major policy decisions?

The case for a negative answer lies in the predictive power of the core natural sciences compared with even the most highly developed social sciences. Social sciences may be surrounded by the “paraphernalia” of the natural sciences, such as technical terminology, mathematical equations, empirical data and even carefully designed experiments. But when it comes to generating reliable scientific knowledge, there is nothing more important than frequent and detailed predictions of future events. We may have a theory that explains all the known data, but that may be just the result of our having fitted the theory to that data. The strongest support for a theory comes from its ability to correctly predict data that it was not designed to explain.

While the physical sciences produce many detailed and precise predictions, the social sciences do not. The reason is that such predictions almost always require randomized controlled experiments, which are seldom possible when people are involved. For one thing, we are too complex: our behavior depends on an enormous number of tightly interconnected variables that are extraordinarily difficult to distinguish and study separately. Also, moral considerations forbid manipulating humans the way we do inanimate objects. As a result, most social science research falls far short of the natural sciences’ standard of controlled experiments.

18 comments:

umbrarchist said...

Are all of these subjects going to be biased by the over inflated ego of European culture?

Economics that can ignore Demand Side Depreciation for SIXTY YEARS?  How weird!
 

CNu said...

What does the inflated ego of European culture mean?

and

Without exception, economics sole and exclusive purpose is to narratize and rationalize the profit-seeking of those who extract more from the culture and the ecology than they put into it.

Big Don said...

Right.  When it comes to downright evil, those economist-driven profit-seekers rank right down there with urban IQ-75 Flash-Robberz...

CNu said...

Economists aren't driving a dayyum thing bighead full-o-mush!!!

read slowly as an aid to comprehension....,

the economists propagandize and rationalize the profit-seeking predations, period.

remember that fact should you live long enough to see all those entitlements you've long imagined yourself as having "earned" stripped away from you and many another in your generation.

Big Don said...

The point was, you shouldn't be criticizing a system wherein profits are extracted under a civilized legal framework.  There are much more deserving targets for your wrath...e.g., blatant cold-blooded criminals.

umbrarchist said...

 The White Man's Burden and Social Darwinism rationalization for the dominance of European culture on the planet.

CNu said...

Memo to Don.

Assertion of privilege (private-law) and courteous behavior (you know, those exaggerated manners and affectations contrived to keep privileged european steel/steal sheathed instead of drawing other privileged european blood) - IS PRECISELY THE COLDEST BLOODED CRIMINALITY THAT THERE HAS EVER BEEN IN THE ANNALS OF RECORDED HUMAN HISTORY!!!

But you needn't take my word for it BD, Fr. John Romanides recounts the rise of criminal feudalism very nicely, as well as, the system of "theological" propaganda concocted to narratize and rationalize its psychopathic criminal predations.

makheru bradley said...

The study assumes that those HVA teachers were not using the eraser to raise those test scores.

It appears that those ATL educators are in for some severe punishment.

http://theholecard.blogspot.com/

Big Don said...

@CNu - "Economists aren't driving a dayyum thing"


...A lot of major investment decisions are made as a result of what economists like Krugman, Roubini, et. al. have to say...

CNu said...

You seem to have completely missed out on the past 30 years of financial industrial rule by quants and now lately by hackers implementing flash trading schemes...,

Tom said...

Are you kidding me?  Only complete amateurs are going by what some economist they don't even know writes in the newspaper.

Big Don said...

Didn't say one economist.  Info from many economists, big names and small names, combined with direct investor observations, is synthesized into decisions that make sense for each particular investor.  Furthermore, what's going on in any observed large corporation is driven to some extent by their *own* economists.  So to say that,  "Economists aren't driving a dayyum thing"  just isn't a very accurate picture...

Hey, give BD a break.  We could have really unloaded on that IQ-75 school cheating thing...

Tom said...

Maybe to some extent, but (a) economists are full of crap, and (b) lots of people know that, not just us six people here.

Big Don said...

IIRC at least some of Martenson's Crash Course was originally posted here on Subrealism...

CNu said...

rotflmbao..., I always thought your "interests" had fetishistic underpinnings

CNu said...

ahem..., there are 40 daily ritual habituals and between 600 and 800 daily unique visitors. on special days (if a topic is a really hot search) we occasionally spike past 2000 unique visits in a day.

CNu said...

Economics is the publishing of political agendas that are hidden within
known-false assumptions. If one accepts these assumptions then one accepts the
hidden agendas.

Tom said...

 ...  my bad ...