Dev Null (Hypothesis)

So, Jason Thibeault is talking about the null hypothesis in relation to sexual assault claims, in response to Twitter comments that seemed to say that the null hypothesis wrt a claim that someone was raped is that she made it all up, or that she’s lying. I’m not going to go and try to find the specific comments and conversation to make sure that this is indeed what was said — because it being stated that boldly isn’t that likely and everyone summarizes longer conversations into soundbytes — but I do want to address Thibeault’s take on the null hypothesis, which seems to reflect an overarching problem in skepticism and in general: attempting to take a concept that applies in some specific situations and contorting it to be a concept that applies in general, even if it doesn’t really fit.

Before getting into that, I want to highlight this part of his summary:

… if you’re willing to be skeptical of a claim without evidence, be equally skeptical of a counterclaim without evidence …

What most skeptics fail to realize is that a strong atheist claim — a claim that God or a specific theist God does not exist — is, in fact, a counterclaim. By this reasoning, we should be as skeptical of a strong atheist claim as of the theist’s claim that God does exist. And yet, most atheists who subscribe to skepticism claim that the atheist, even the strong atheist, has no burden of proof, in that in some sense the “null hypothesis” of that claim is that God does not exist. So, then, most skeptics of that sort will be rejecting skepticism if they maintain that one can hold the strong atheist claim without sufficient evidence … as they do when they insist that the burden of proof is on the theists, not the atheist. Theists and strong atheists would have an equal burden of proof by this reasoning.

No, onto the null hypothesis. The null hypothesis, in its most direct and simplest form, simply says that if you are going to posit that something has an effect, then what you need to do is demonstrate that introducing that thing into the proper environment has that effect to a degree greater than can be accounted for by random chance. Control conditions on experiments provide the clearest example of how the null hypothesis works. What you do is create two experimental conditions that are identical except for the thing that you want to measure. You introduce that thing to one case, and measure for the effect you claim it will have. If the experimental case where you introduce the factor shows the effect to a significantly greater degree, then you’ve overcome the null hypothesis — where that factor isn’t present — and therefore can conclude that the thing you think has an effect likely does.

What the null hypothesis does not do is advocate between two competing hypotheses. If you want to claim that thing A will have a certain effect, and someone else claims that A will have a different effect, then there is no real relevant null hypothesis. Sure, you might want to test to see if it has any effect at all, at which point the null hypothesis might come into play. But you don’t use any sort of null hypothesis to determine which of the two hypotheses is more reasonable. When comparing them, you don’t appeal to a null hypothesis at all, but simply to what the hypotheses themselves predict. As I said, comparing to the null hypothesis might prove both wrong … but it won’t prove one of them better than the other without comparing their predicted results. Some might claim that Occam’s Razor is a version of the null hypothesis when comparing hypotheses, but you can only argue that appealing to the “additional effect” idea, to the idea of overdetermination. If two hypotheses predict the same results but one has an additional entity, then we might be able to say that the addition of the additional entity seems to have no effect, which ties right back into the null hypothesis test. But when two hypotheses predict the same effects but the one with the additional entity isn’t just adding it on, but is in fact completely different and because it is different that new theoretical entity is required, then you can’t compare the two using the null hypothesis; whether the entity is necessary or not or exists or not depends on which hypothesis is correct, and so you can’t simply say that it is not necessary. But you might still be able to use Occam’s Razor to compare them.

So the important thing to note about Thibeault’s argument is that he is, by his own admission, comparing two hypotheses. At that point, the null hypothesis doesn’t apply. Either one hypothesis or the other is adequately demonstrated … or neither of them are. So, in the case of a sexual assault case where one says that she was raped and the person she’s accusing of the rape says that he didn’t, then we don’t have a null hypothesis … a fact that Thibeault makes clear by trying to create one:

The null hypothesis here is actually something more like “we can’t get to the truth of these dueling claims but can take no action in absence of hard evidence”.

A null hypothesis is, in fact, a hypothesis, not a recommendation on how to proceed. The null hypothesis is also always something comparable to the hypotheses, which is why you can say that you can overcome it. His purported “null hypothesis” is in fact a statement of epistemic uncertainty, not a null hypothesis.

A big part of Thibeault’s post relies on comparing the new hypothesis to the expected or proven cases. This usually isn’t a null hypothesis either, unless the new hypothesis claims that in a new circumstance or with a new effect something different will occur. Otherwise, all you are doing is challenging an existing hypothesis that has been proven and demonstrated, and you do have to provide evidence sufficient to overturn it … but again, that’s because it has been demonstrated, not that it is to be treated as a null hypothesis.

So how does he apply the null hypothesis to sexual assault claims?

So, between that, and the fact that we absolutely know that rape is drastically underreported, we can then presume that the majority of rape claims are actually true (or at least true to the extent that the woman was in fact raped), because the number of rape claims is significantly lower than the number of rapes. Because this is a common occurrence, and because there are pressures against admitting that you’re a victim of such a common occurrence, the majority of such claims are true. The null hypothesis when confronted with a woman claiming to have been raped is that yes, she very probably was raped, because it is not an extraordinary claim, and ordinary claims require only ordinary evidence. Testimony absolutely counts as ordinary evidence.

It’d be hard to find a better example of how the notion of a null hypothesis can be contorted to try to fit cases where it clearly doesn’t fit. Thibeault here talks about the null hypothesis in the case where someone claims to be raped to be, in fact, that she was raped … because it is a common occurrence and so you don’t need extraordinary evidence to belief that. Essentially, all he’s claiming is that the testimony of a woman who says that she was raped is sufficient evidence to at least believe that she was. That has no relation to a null hypothesis … and even answers the question that Thibeault is trying to answer. The challenge around the null hypothesis claim was for her to give evidence, and Thibeault’s answer here is that testimony of first-person experience is evidence, that essentially that at that level we don’t need any additional evidence just to believe her.

But this isn’t enough for all claims:

That is not to say that the person claiming to have been raped knows enough of the assailant to accurately make an accusation, or has enough evidence to get that person put in jail, and those cases where the woman was indeed raped but cannot level an accusation that sticks are all often counted against the “false accusations” statistic.

I will further note here that because most individuals are not rapists, any specific claim against a specific person requires extraordinary evidence.

Now, it is obvious that in almost any case where someone is making a claim that they were raped, there is indeed a specific claim being made. So in almost all of the sorts of rape cases that Thibeault is talking about, extraordinary evidence would be required … at which point that she was raped would no longer be a null hypothesis by Thibeault’s own logic.

Essentially, this is what seems to be epistemically justified: if a woman says to you that she has been raped, then rape is common enough that you should believe her unless you have reason not to, especially considering that this is not something that people will mention casually. However, as soon as she claims that she was raped at a specific place at a specific time by a specific person, this is a claim that goes beyond that and so you have to consider the context of the claim in determining whether she should be believed or not … which includes how trustworthy she is and how trustworthy you consider the person she accused to be.

None of this makes any reference to a null hypothesis, directly or implied. And it does not support claims that by default you should believe that she was lying, which is indeed a claim in and of itself. But it is no surprise that skeptics are fighting over what actual claim should be accepted by default in these cases, since as the start of the post demonstrated many of them have been using that sort of tactic to win arguments over religion. Ultimately, the only sort of skepticism worth having is one that takes a consistent idea of how much and how objective of evidence you need in order to believe something, and the modern Internet skepticism lacks the philosophical maturity to be that consistent. Hence, contortions of valuable principles in order to fit a “one-size-fits-all” epistemology.