May 19, 2011

A Pop Science Fiasco (Including Some All-New Steven Pinker Bashing)

If you follow my writing at the Good Men Project, you'll know that I recently reviewed A Billion Wicked Thoughts, a new pop psychology book about sexual desire by Ogi Ogas and Sai Gaddam. Basic summary: biologically, men are more interested in sex, and women in emotions, because it was advantageous for us to have evolved this way.

I mowed down the book at GMP without doing much background research, mainly because I didn't have to — on the basis of what I know about experimental design and evolutionary psychology (henceforth EP) I had plenty of ammunition to be getting on with. Ogas and Gaddam's argument is just plain messy.

But I dropped the ball by not digging deeper because, man, there is some seriously screwed up history to this research project that is a lot more troubling than its dubious EP conclusions.

The story begins in the first half of 2009, when Ogas and Gaddam signed a book deal with Dutton. I can't find an exact date, but given that their agent's blog mentions the deal in July 2009, it can't have been any later. (At that point the book was called Rule 34 and slated for release in 2010.)

Here appears the first ethical murkiness: in August 2009 — i.e. after signing their contract — Ogas and Gaddam allegedly began contacting fan fiction authors across the web asking them to take part in a survey for the book. [Update: if you look at the timeline they present in the book, they actually didn't start doing ANY research until after the book deal.] Now, for certain kinds of nonfiction there's nothing wrong with doing research after you have a publishing deal. Who wants to spend weeks holed up in an archive somewhere if nobody's going to print your findings?

But for scientific studies there's obviously a big problem with it: conflict of interest. If a publisher is agreeing to print your book at a certain time — and, probably, has already paid you some money for it — well, you have to come up with some impressive results, fast, and that kind of incentive seriously jeopardizes scientific disinterest. (This assumes, of course, that you're not a fire-breathing cultural studies leftist who doesn't believe science can be disinterested to begin with).

Anyway, that's problem number one, which on its own would probably scupper the project in an ethics board review. But according to some of the people Ogas and Gaddam allegedly contacted, the survey had far deeper problems: it used human subjects without proper release or protection, it asked needlessly invasive questions, it made no real attempt to ensure subjects were of legal age, etc., etc.

That's probably why, when contacted for comment, the Institutional Review Board at Boston University, where Ogas and Gaddam received their PhDs, allegedly told offended subjects that the study was not affiliated with the institution and had no IRB approval.

Ogas and Gaddam don't actually deny this:"we intentionally conducted our research outside of academia, without federal funding," they said in an interview, and hence did not technically require IRB approval. Their explanation for why they chose to do so sounds plausible, I suppose, but strikes me as disingenuous since a survey like this would probably never get the go-ahead, at least not within their desired parameters or the timeframe demanded by their publisher:
We intentionally conducted our research outside of academia, without federal funding, in order to remain independent from the fierce tempest of ideological, social, and political pressures that besets the contemporary study of sexuality.
They also add that "we did not directly study human subjects," that "there is no original survey data in our book," and that "we neither requested nor received identifying information about individuals." One wonders about the ontological mechanics of not requesting things from individuals you never surveyed to begin with.

In any case, none of this jibes particularly well with the fanfic authors' accounts, detailed here, which claim that Ogas and Gaddam at the very least attempted to generate original survey data gleaned directly from human subjects, regardless of whether or not it actually appeared in the book. And if that makes their explanation sound like legalese wrangling, the acknowledgements section in A Billion Wicked Thoughts gives a possible reason why:
We'd like to thank the publishing professionals who made this book possible, including attorney Gary Mailman.
Since when were attorneys listed as "publishing professionals" before copy editors and editorial assistants?!

Of course, the nature of the internet makes it difficult to corroborate the fanfic version of events. The LiveJournal account allegedly belonging to Ogas has been deactivated, and all the evidence of the survey now exists in the form of screen grabs and copy/pastes archived by other LiveJournal users who were offended by the survey and clearly have some motivation for assassinating Ogas and Gaddam's character.

But the sheer depth of some of this documentation makes me wonder why anyone would go to the trouble of fabricating it, and good grief, if it's legit it's horrifying. Ogas and Gaddam allegedly based the survey on methodologically dubious research questions like:
How is straight female interest in slash fiction like straight male interest in shemale models?
Their survey questions, meanwhile, were allegedly altered halfway through the study, and in any case were just as methodologically dubious, e.g.:
Do you enjoy fan fiction that involves a character changing his or her gender?
[Leading question; more meaningful data would be generated by asking "What do you enjoy about fan fiction?"], or:
What proportion of of the fanfic that you read contains explicit sexual content?
[Unreliable question; no sound empirical basis for comparing user-reported "proportions", and in any case definitions of "explicit" will change from respondent to respondent.]

When respondents attempted to address these shortcomings with Ogas and Gaddam, the researchers apparently became insulting and abusive and ultimately shut down the survey altogether.

Again, much of this is difficult to corroborate since it comes down to a "he says"/"she says" argument about survey materials whose originals no longer exist in the public domain. But here's what's on the record:

1. They signed a contract agreeing to publish their results before all any of their research had been conducted.

2. They deliberately conducted said research without institutional affiliation or IRB approval.

That would be fine if the book weren't presented as scientific research, but it is, and as is often the case with "research" about sex, A Billion Wicked Thoughts has been gleefully reported on by media outlets everywhere as if the book's conclusions are Hard Fact.

But that simply isn't true, and I wonder why the scientific community is going along with it. None other than scientific media messiah Steven Pinker graces the cover with his praise:
In a stroke of ingenuity, Ogas and Gaddam circumvent the deepest limitation of standard psychological surveys: that they merely tap undergraduates’ socially acceptable responses, a flaw nowhere more damaging than in the touchy realm of sexuality. A Billion Wicked Thoughts is a goldmine of information about this hugely important topic.
Since the main thing Ogas and Gaddam circumvented about standard psychology was the very framework designed to make it reliable and ethical, and since the "goldmine of information" thus rendered is empirically suspect, it seems like Pinker must either not have scrutinized the book's research very closely, or scrutinized it but decided that there was nothing wrong with publishing non-institutional research under the aegis of science.

Either way, that seems to imply a sort of two-tier system of research: the "real," institutional research that only "real" scientists need to know about, and the "pop" research that keeps the general public distracted and so doesn't need to be that rigorous or purged of ideological motivation.

And either way, that strikes me as deeply problematic. What gives, Pinker?

1 comment:

Dye! said...

(1) I find your attitude toward the contribution of lawyers to the publishing industry deeply disturbing. While I personally find the payment of hundreds of dollars per hour the only reward I need, what harm is there to acknowledging that, without lawyers, nary a subject would be studied nor a word written?

(2) What have you got against pseudoscience? Sure, it lacks the "rigour" of "real" science, but how else would learn about how Chinese aliens built the pyramids?

Post a Comment