Tuesday, June 24, 2008

Can you Trust a Survey?

Recently the Pew Forum on Religion & Life issued its SecondReport on major Religious trends in the United States. Understandably, many Evangelical leaders were upset to discover 57% of the people who identified themselves as “Evangelical” thought eternal life is possible through other religions.

The survey is being (slightly) lauded as demonstrating increasing tolerance within America. Before we become alarmed (if you are an evangelical) or encouraged (if you think Americans are becoming more tolerant) I question how the survey was conducted.

See, the survey also concluded 21% of people who identify themselves as atheists believe there is a god. Yep, that’s right. 1 in 5 atheists (according to the survey) say, “I am an atheist. I believe there is a god.” Something doesn’t add up.

And no, “atheist” was not a catch-all phrase. The agnostics were separated out, of whom 55% believe there is a god, and the “secular unaffiliated” were also separated out, of whom 66% believe there is a god.

To further demonstrate how odd the results appear, 6% of people who say they are atheists, and 14% of people who say they are agnostic, stated they believed in a personal god! Not only that they believe in some general notion of “god,” but a particular description of one! Now, it may seem as if I am dangerously approaching the “No True Scotsman” fallacy. We are all quite familiar with “No True Christian.” I don’t want to say these folks are “No True Atheist”—but I would sure like an explanation of what they think the word “atheist” means by saying they are atheist and believe in a personal god!

So I looked at how the question was asked. Enlightenment:

Q.16 What is your present religion, if any? Are you Protestant, Roman Catholic, Mormon, Orthodox such as Greek or Russian Orthodox, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, something else, or nothing in particular?

INTERVIEWER: IF R VOLUNTEERS“nothing in particular, none, no religion, etc.” BEFORE REACHING END OF LIST, PROMPT WITH: “and would you say that’s atheist, agnostic, or just nothing in particular?”
[emphasis in original]

From Data (Note: PDF File.)

You can see exactly what was happening. The person being interviewed was getting bored with the survey and launched out the quickest answer. We can imagine the conversation:

Interviewer: What is your present religion, if any? Are you Protestant, Roman Catholic, Mor--
Interviewee: [impatiently] Oh, I am not one religion in particular.

Interviewer: [following instructions] “and would you say that’s atheist, agno--”
Interviewee: [impatiently] Atheist, I guess.

The person was marked down “atheist” and later, when the question of what type of god they believed in came up, they said, “a personal god.” No one ever thought to ask, “Does this line up?”

In a local Judge’s chamber there is a picture of a cake from a certain bakery. On the cake is the following message:

“Good Luck, Betty
“Underneath that
“Best Wishes from all.

Likewise we can imagine THAT conversation:

Person: We would like to order a cake.
Bakery: What do you want on it?
Person: ‘Good Luck, Betty,’ and underneath that, ‘Best Wishes from all.’


  1. Good of you to wish to avoid the One True Scotsman fallacy, but there's a distinct difference between shifting around your definition of something to ensure that it always excludes certain people, no matter what, and an absolute contradiction to the core definition of the word.

    It may be that there are folks who call themselves Christian, without believing in a literal Christ; but there are no atheists who believe in a literal, let alone personal, God. I don't have to shift the definition to know that: the definition already covers it!

  2. Though, note that the 21% figure is for the belief in God or a "universal spirit", and that of that 21%, 12% believe in an "impersonal force", and 3% gave "don't know what kind of force/God". So, this would leave just the 6% of "atheists" who believe in a personal God. Still, that means there were 30 persons (out of over 35,000) who claimed to be atheists who believed in a personal God.

    So, the presentation of the results should get a fair amount of blame, too, since most of those atheists were claiming belief in a "universal force", and not the idea of God.

    I recall noted atheist video-blogging comedian Pat Condell as having mentioned a believe in a "universal lifeforce" or some such.

  3. The other half of this is one of my personal pet peeves--the "nondenominational" Christian.

    You know the type: the fundamentalist, Pentecostal, evangelical Protestant who refuses to identify as a fundamentalist, a Pentecostal, an evangelical or even a Protestant (usually out of an aversion of labels, at least applied to themselves. Others are fair game). Often, they will tote out the old "Bible-based" tag, saying that they essentially picked up a Bible one Thursday afternoon never having been exposed to one before, and then miraculously happened to build a church based on Trinitarian doctrine. It's a pretty cheap ploy to build up a supposed ideological purity, but it also IMO represents a streak of anti-intellecualism and anti-introspection on the part of this particular brand of Protestantism.

    The Church ignores its own history and its own doctrine at its peril.

    I wonder how many "nondenominational" Christians listened to the pollster's question this way:

    "Am I Protestant? No. Roman Catholic? No. Greek or Russian Orthodox? No. Mormon? No. Jewish or Muslim? Hell no! Hindu? No. Atheist? No. Agnostic? No. Something else? [insert flush of self-satisfaction] I'm a CHRISTIAN!"

    And how does the pollster respond? What box gets ticked?