Gabbin’ ‘Bout Cause …

So, David Colquhoun at Improbable Science weighed in on hy he doesn’t think philosophy has much to offer sciencew, and he uses a fairly peculiar example of causation linked to a diet study to talk about an error that he thinks some philosophers have made. He says a bit about it and I’m probably oversimplifying his view, but he isn’t all that careful so I think it’s fair to call him out on something that he says repeatedly, summed up in his conclusion:

Of course RCTs [Randomised Controlled Trials] are not the only way to get knowledge. Often they have not been done, and sometimes it is hard to imagine how they could be done (though not nearly as often as some people would like to say).

It is true that RCTs tell you only about an average effect in a large population. But the same is true of observational epidemiology. That limitation is nothing to do with randomisation, it is a result of the crude and inadequate way in which diseases are classified (as discussed above). It is also true that randomisation doesn’t guarantee lack of bias in an individual case, but only in the long run. But it is the best that can be done. The fact remains that randomization is the only way to be sure of causality, and making mistakes about causality can harm patients, as it did in the case of HRT.

So, essentially — and he says this repeatedly — to get at causation you have to do randomized controlled trials, and the key is the randomization. Otherwise, you can’t get causation. This is so out of whack with both philosophy and science that it’s really hard for me to decide which of my backgrounds I’m bringing to bear here. Randomization isn’t the only way to be sure of causality, and it isn’t even the best way. Its main benefit is that sometimes it’s the only thing you can do.

Let’s start by talking about the best way to find out about causation, straight from science: decide which factor you want to test for its effect, control for all other possible factors, and then alter that factor. So if you want to see how, say, a temperature increase affects the volume of a gas, you control for things like pressure, then change the temperature and see what happens. Then you can conclude that temperature increases cause the volume of a gas to expand, and voila … you have causation. And note that there was absolutely no randomness involved, and any introduction of randomness would spoil the whole experiment.

This carries on into common sense causal determinations. If I want to find out why my car won’t start, I go through and alter things while keeping everything else the same until the car will start. So I first, say, clean the battery cables, and try it. Then check the spark plugs. And so on and so forth. This is basically just second-nature to us, and this is how we do, in fact, find out about causation.

Now, there are indeed problems with this method. The first is that you have to be very careful that previous examinations don’t actually change your state. So, for example, if I don’t put the battery cables back on properly I’ve now created a new state and a new problem and so won’t actually determine the cause of my car not starting even if I do make the appropriate change, as I’ve introduced another cause (I do this a lot when fixing bugs while programming). A controlled science doesn’t have to worry about this too much, since it at least would identify what the new state will be even if it can’t quite reset it. But there is another problem that is quite relevant for the discussion Colquhuon is having: we often do not know or are unable to control for all of the potential confounds when we start the experiment. The field this is most relevant for is actually psychology, since not only do you have a massive amount of potential interactions and confounds those confounds can include the subjects changing their behaviour due to the test conditions.

Enter randomization. It allows the experimenters to not have to know what all the confounds are by presuming that if there are potential confounds they will be more or less equally distributed amongst the participants, especially with a large number of runs. But this isn’t some kind of excellent or precise methodology that works well at establishing causation. This is simply a workaround for the fact that we don’t know and can’t control for all the factors.

So, in terms of determining causality, it’s inferior to the normal practice. It requires more subjects and more trials than normal practice, and it also can’t detect as small a difference as normal practice because it has to always correct for statistical anomalies and the potential presence of another factor that is actually the cause of the change but which is only slightly unbalanced in your test set of participants.

So, the ideal way to look for causation is:

1) Control explicitly for every possible thing you can think of and can control.
2) Randomize to try to get rid of the things you can’t control for or didn’t think of.

But if you’ve exhausted all possibilities in 1), you don’t need to do 2). You do 2) only because you have no choice, not because it is some kind of Gold Standard for determining causation. 1) is the Gold Standard; 2) is the inferior workaround. God help you if you skip 1); you’re likely to screw up big time because there are just too many factors involved to get any sort of answer, and so you’ll miss finding the causations that are there and you were hoping to find.

Presumably, Colquhoun is very much aware of this, but if anything I’d suggest that philosophy could be of aid to him in making sure that he understands exactly what he’s saying when he says things like “randomization is the only way to be sure of causality”, which is, in fact, patently not true, as it’s a) not the only way and b) doesn’t actually let you be sure of causality anyway (due to potential confounds that you just missed).

Advertisements

2 Responses to “Gabbin’ ‘Bout Cause …”

  1. David Colquhoun Says:

    On the contrary, reading RA Fisher is infinitely more useful than reading any philosopher in my experience. It was surely obvious that I was talking mainly about clinical trials where causality is a huge problem. It really doesn’t help at all to bring up the old chestnut about RCTs for parachutes.

    I can’t think why you say diet and health is a peculiar example. There is scarcely any topic on which more is written on the basis of inadequate knowledge.

    I’m sorry, but I think your last section is plain wrong, both in principle and in practice. There is never any way to ” Control explicitly for every possible thing” because you never know all the things that might be relevant. I do get the impression that you are talking about principles and have never actually done an experiment yourself. It’s harder than you think.

  2. verbosestoic Says:

    David,

    I think you’ve taken my first statement too strongly, which is in part my fault for not making it clearer. The idea is that you sit down and think of all the things that you can think of that could impact it, and control for those. Then if it is possible for there to be other factors, you randomize to hopefully control for them. How easy it is to control for all factors depends on field. Psychology, as I said, has to randomize because it’s really hard to control for all of those factors, and health studies are likely to be the same way. Physics, chemistry, and biology, on the other hand, can mostly control for most of their factors, and if they find another factor that might have an impact they simply control for that one and re-run the test.

    The whole point of that part was to insist that even if you are doing randomization, you really should do 1) first. Do you disagree with that?

    As for why I talked about it, it is, as I said, just an exercise in making sure that you really do say what you mean. To claim that the only way to find causation as if it’s some kind of global best principle — as you do — is problematic. Even in the soft sciences of psychology and health, what they really want to do is determine causation the way things like physics do. The problem is that they simply can’t due to their subject matter, and it seemed to me that you didn’t make it clear that the details of the subject matter is what matters, and that the randomization approach is only used when it’s all you can actually do. Again, I think you do get that, as I said, but when you’re going after philosophy and what they’re saying making sure that you aren’t overextending your position may indeed be one reason why you’re getting the philosophical backlash.

    And I found it an odd example because other than work on general causation philosophy generally says little about diet … and may not even be wrong in the criticisms it says here (how they’re used is another matter).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: