How do you prove that something like Scientology auditing “works?” I’ve had many discussions with former Scientologists who say that their lives changed for the better because of Scientology auditing, and that I should not treat it with such contempt.
I believe the stories of people who have life-changing “wins” from auditing yet, at the same time, I remain resolute in saying that auditing is not proven to be useful. How do you resolve the apparent paradox in my view?
Auditing, using the e-meter (below) to attempt to divine a member’s innermost thoughts, is perhaps the cornerstone of Scientology’s self-help practices. The e-meter is a simple century-old electronic circuit dolled up in a fancy package that functions as an oversimplified lie detector. Scientologists pay hundreds of dollars per hour for an “auditor” to interrogate them until the needle shows some indicator that’s imagined to be a specific troubling thought or emotion disappearing.
The issue of how to prove or disprove something like “auditing works” is not inherently obvious. And the fact that individual stories can be positive while the general phenomenon remains unproven can be seemingly problematic. There’s an important distinction between what happened to one person’s experience in a particular circumstance versus a finding that something is true over a long period of time in the population as a whole. I’ll explore the general concept of statistical validity and then I’ll show how this applies to auditing.
The Plural of “Anecdote” is not “Data.”
Many of the people I’ve talked to about auditing have told stories of life-changing “wins” that came out of an auditing session, whether it was their own work or when they served as someone else’s auditor. While each of those stories may be true in the sense that the person is not intentionally trying to deceive me about what happened to them, even an infinite number of such stories cannot prove that auditing works. You have to approach proof another way.
Scientists have a saying: “the plural of anecdotes is not data.” Success stories alone are not actual proof of anything. They are especially not proof that something causes something else. To determine causality, you have to use very specific statistical and analytical techniques in a sample of cases that span all the possible inputs and outcomes.
Understanding causal relationships are very important to predicting what might happen, and thus to making good decisions. Anecdotes can be intentionally or unintentionally cherry-picked to distort the truth. They tell only part of the story.
Here’s a cartoon that does a great job of illustrating the problems with anecdotes, plus other bad science.
An Example of Where Anecdotes Prove Nothing
Here is where anecdotes don’t work: suppose a medical quack like Dr. Oz says “drinking a quart of fresh-squeezed orange juice daily can cure ovarian cysts.” Oz is careful to point out that only fresh-squeezed works. In the next year, millions of women drink lots of orange juice, enriching the orange juice industry considerably in the process, and thousands of them discover at their next OB/GYN checkup that they were free of cysts. Some number of those with cysts find they’re gone. They write Dr. Oz glowing letters of thanks. Being the pseudoscience hack that he is, Oz reads a few aloud on his show, showing a whole mailbag full of correspondence to “prove” the point.
Did drinking OJ really cause their cysts to remit? Well, it may have in some cases. But it turns out that benign ovarian cysts spontaneously remit quite often, even large ones or in cases of severely polycystic patients. You can’t establish causation until you also measure things like:
- The number of people who drank orange juice and did not see remission
- The number of people who didn’t drink orange juice but saw remission
- The number of people who drank no orange juice and did not see remission
- In each case where a patient saw remission, you rule out other factors (like being on some particular drug) that could have caused cysts going into remission.
- The rate of remission for people who drank frozen orange juice because they couldn’t afford fresh.
When you do it right, you might find something very different from a causal relationship. Perhaps, since fresh-squeezed OJ is very expensive, everyone that saw remission was rich and thus lived in conditions where cysts are less likely to occur (fewer carcinogens, less stress from a more fulfilling job, etc.). Or you might find that there is a causal effect, but orange juice is only fractionally more effective than doing nothing, so $5 a day someone spends on fresh-squeezed OJ is mostly wasted.
Anecdotes are like votes. There are some things you can vote on, but there are things you can’t, like the law of gravity.
Need to Rule out Alternative Explanations
People pushing anecdotes tend to jump to a particular cause for a phenomenon and usually ignore alternate causes. This is a peril not just of thinking anecdotally, but in doing actual scientific experimentation. Here’s a contrived example:
Suppose I get data that say that 99.7% of Scientologists in Mexico City report feeling significantly more energized after going into their org for an evening of study and auditing. When you have that much of a correlation, it’s especially important to isolate other factors that may cause the result.
Why did I use Mexico City in the example? It’s high in the mountains and has a lot of nasty particulate matter in its smog. Suppose the local org has a high-quality HEPA filter that nearly eliminates particulate matter of all sizes. In fact, the filters are so good that they release a certain amount of ozone into the building’s air. It could easily be that the cleaner air is providing the emotional boost, not the auditing. And people can feel the cleaner air in part because of the small elevation in ozone, which “tickles” your nose and makes you feel a little more alert. You would have to isolate the effects of auditing from the effects of better air quality.
If You Run Out of Counterarguments, Is Your Hypothesis Valid?
Another fallacy that sometimes crops up in attempting to think about proper use of evidence is what I call the “UFO fallacy.” This is essentially making the mistake of believing that because the counterarguments aren’t sufficient to disprove your hypothesis, then what you are trying to prove must be true. This typically comes up in UFO-related discussions.
Consider a hypothetical report of unexplained aerial phenomena such as bright lights racing overhead at high speed, noiselessly buzzing a car in the desert. Scientists investigate and can’t come up with a definitive explanation for that particular sighting from any of the long list of explanations that have accounted for unexplained aerial phenomena. A scientist would say that the cause is undetermined. But a UFO fan would say “because you can’t explain it as anything else, that sighting therefore must be an alien intelligence piloting a spacecraft, searching for people to subject to an alien anal probe.”
Ummmm, no.
Just because there isn’t an affirmative explanation doesn’t mean the one that has no other evidence for it must be correct. Yes, it might be an alien intelligence, but it is more likely explainable by a different scientist with more experience, it might be due to some scientific phenomenon that’s not well understood right now, or it might be due to some undetected issue with the report. There has to be affirmative proof for a new and previously unseen resolution of a problem.
And just like the existence of unexplained phenomena doesn’t allow you to claim that space aliens are real, so too does establishing correlation not allow you to jump up to claim causation. So even if all your friends got amazing wins from auditing and you don’t remember ever having had a bad session, it doesn’t mean that auditing works.
Anecdotes Have Value… But Only in the Right Place
There are a lot of places where “anecdotes” may lead to incredible discoveries. Sometimes, a collection of anecdotes suggest something interesting may be happening that deserves further study, leading to a major new discovery.
A medicine for pulmonary arterial hypertension code-named UK-92,480 was failing a Stage 3 trial. Pfizer pulled the plug after investing $1 billion because the drug was no more effective than much cheaper existing drugs, so nobody would buy it. Standard practice at the end of a trial is to collect the unused pills because you obviously can’t leave an unapproved drug out there in the hands of the public. But a surprising number of the men in the study refused to give it back. Eventually, Pfizer figured out why. And so was born Viagra.
The anecdotes coming back from the clinical research organizations running the study eventually led investigators to figure out what was going on. But Pfizer couldn’t just start selling Viagra for erectile dysfunction based on anecdotes from people taking it. They had to conduct additional trials to prove the extent to which Viagra actually caused erectile dysfunction to disappear.
In other words, the scientists at Pfizer used the anecdotes about people keeping the pills to point them to an area for inquiry. Once they began to suspect something was going on, they did a full statistical evaluation to prove that the drug actually cured what the people hanging onto the pills said it cured. Unfortunately, Scientology will never allow auditing to be subjected to a proper controlled trial, in fear of the results. After all, the one paper that evaluated Scientology “tech” in the 1950s roundly trashed it in extremely contemptuous language (I can’t find a copy of this paper; if any readers could post a link in the comments, I’ll update this).
Anecdotes Versus Statistics in Evaluating Scientology Auditing “Tech”
So how does this relate to Scientology? It’s the central question: does any Scientology “tech” actually help people? Lots of people claim anecdotally that Scientology “helped them,” but there’s nothing measurable in the real world. Some people have had life-changing wins as a result of auditing, such as the realization that they are “out 2-D” and needed to get a divorce. And in some cases, auditing was the one discipline they were practicing that could have gotten them to realize that and then helped them to take action.
But when you have a case of a real-world effect of Scientology like realizing that you need to get out of a horrendous marriage, you have to look at it on a population basis. First, what percentage of the general population figures out by themselves that they needed to get a divorce, without any input? Second, what percentage of the population pursuing other self-help methods (reading books, seeing therapists, doing yoga, going to Landmark Forum, going to Alcoholics Anonymous, praying to a giant Styrofoam grasshopper, watching Anthony Robbins videos, etc.) came to the same conclusion?
And finally, what percentage of Scientologists who did auditing failed to get any measurable value out of their experience? For that last, you’d have to look at anyone who ever “squeezed the cans.” You can’t simply look Scientology’s current membership. You would literally have to look at 60 years of history. You’d have to include a representative sample covering everyone from tire-kickers who spent a few minutes trying it out at some county fair and came away unimpressed all the way to people who drank the Kool-Aid up to OT VIII and everyone in between.
You’d find that only a small percentage of people who ever got any auditing found it worthwhile. Even if you restricted the sample a bit, to, say, anybody who did 10 or more hours of auditing, you’d still have a lot of people who said either at the time or in retrospect years later, that it was worthless.
That’s not to say that auditing is completely worthless. It’s as unlikely that auditing would work 0% of the time as it is unlikely that auditing would work 100% of the time. That is, of course, one of the big lies in Scientology: that “the tech works 100% of the time when applied standardly.”
We’ll never quantify the actual effectiveness of auditing. There’s no value in doing a study of a technique ever less practiced by the shrinking membership of a failing cult. Nobody will front the cash for a full clinical trial. We can hypothesize that auditing will have some effectiveness, and we can speculate that the reasons that auditing works in the cases it does may ultimately have little to do with the Hubbard’s beliefs about how the mind works. But we can be absolutely certain that any number of positive anecdotes, no matter how well validated, won’t prove its effectiveness.
