Theory and Experiments

 

This post was prompted by the most recent episode of Hidden Brain Putting Our Assumptions to the Test in which Shankar Vedantam talked to Abhijit Banerjee about the use of experiments in economics, specifically randomized controlled trials RCTs to study development economics.  I like the Hidden Brain, and I always enjoy listening to Banerjee. There was, however, one thing about the episode that bothered me. I thought Vedantam made it sound like Banerjee's work reflected a contest between theory and experiment, rather than emphasizing the importance of both.

The problem with this approach was most glaring when they talked about an experiment that Banerjee conducted in India to study policies to encourage vaccination. They first discussed a number of relatively unsuccessful interventions that had been made to encourage vaccination. Then they turned to Banerjee’s successful experiment. Families were randomly selected to receive an offer of a bag of lentils (1 kilo) if they got their children vaccinated. Families that received the offer were significantly more likely to have their children vaccinated. Vedantam stated that they had gained this knowledge from an experiment, not a grand theory about why poor people do not get vaccinated. But there was a grand theory, and it was clear when Banerjee talked about why it worked. He pointed out that going to get vaccinated had significant costs in terms of time and effort, and the usual unpleasantness of having small children vaccinated. The lentils, while seemingly a small thing relative to not having your children get sick, provided an immediate observable benefit to help compensate for those costs. The grand theory is that people make choices by weighing the costs and benefits. If you increase the benefits of something people will be more likely to do it. If you increase the costs, they will be less likely to do it.

In randomized controlled trials the selection of who will receive the treatment is random; the selection of the treatment is not random. Banerjee did not randomly assign people to a group that would have people pray for them to get vaccinated or to a group that would not receive prayer. He randomly assigned them to a group that would receive an incentive to get vaccinations or to a group that would not receive the incentive. The experiment was based on the theory that increasing the benefits of vaccination would make people more likely to get their children vaccinated.

Vendantam several times made comparisons to the use of RCTs in medicine and the advances in medical knowledge arising from them. But the situation is no different in medicine. People do not go to the expense of running an RCT on randomly selected drugs, nutrients, activities, etc. They have selected a treatment based upon some theory that suggests that it might work. A science with experiments absent any theory would not be possible. There are too many possible treatments.

Although the discussion on the podcast was about experiments, the argument that some theory is necessary to guide the empirical research applies to other kinds of empirical research as well. There is too much data in the world to not have some means of identifying what is useful.