How to Explore the Effect of Doing Something? (Part 2)

Applied Causal Inference 101: Non-Experimental Data

Photo by JJ Jordan on Unsplash

In the first part of this article, we discussed some of the core ideas of identifying the causal effect of doing something (such as receiving a treatment, implementing a policy, etc.). If you missed it, here is the link to the first part:

How to Explore the Effect of Doing Something? (Part 1)

In this second and final part, we will discuss how to identify the causal effect of doing something using non-experimental data. Before we start, we will review the basics one more time:

#1. The causal effect of doing something (e.g., receiving a treatment X) on an outcome (Y) is the difference in the value of the outcome (Y) when we do X and the value of the outcome (Y) when we do not do X, keeping all other things constant.

#2. Observing the outcome Y in this real world is NOT sufficient to identify the causal effect of doing X on Y. We have to know what would be the outcome Y in a counterfactual world, where everything else stays exactly the same as in the real world, except we do not do X. Ideally, we consider the difference in the outcome Y in the two worlds as the causal effect of doing X on the outcome Y.

#3. There is no reliable way to know the causal effect of doing something on any particular human being. However, a properly conducted randomized experiment helps us identify the causal effect of doing X on some outcome Y for the typical (average) participant from a target population.

#4. The typical (average) participant of the treatment group and the control group in a randomized experiment can be considered counterfactuals of each other. In simpler words, randomized experiments help us make an apples-to-apples comparison. If randomization works as intended, we consider the difference between the treatment group’s average outcome and the control group’s average outcome as the causal effect’s estimate (also known as Average Causal Effect).

#5. Three essential requirements for making a valid causal inference are 1) apples-to-apples comparison, 2) no interference, and 3) no two versions of the same treatment.

Experiments are great, except sometimes they are unethical, if not impossible, to conduct. For this reason, social, behavioral, and health scientists often try to make causal inferences using non-experimental (also known as observational) data.

The key issue with using non-experimental data is that because the treatment is not randomly assigned, some people are more likely than others to select into the treatment. In this sense, in the context of observational data, the treatment and control groups are usually not apples-to-apples. Suppose, there are certain factors/characteristics that affect both the treatment and the outcome. In that case, comparing the outcomes of the two groups will result in a biased estimate of the causal effect.

Using a hypothetical example, we will contextualize this critical problem and explain the fundamental concepts behind two methodological approaches to tackle it!

Problem Context

A grocery store XYZ offered all its customers a “shopper’s card” in September. Some of the customers accepted it, and others did not. The marketing analyst is trying to understand whether accepting the shopper’s card led to higher spending by customers in October.

As “lead to” is a causal phrase, fundamentally, the analyst is asking: Does the acceptance of a shopper’s card cause an increase in spending by the customers at the grocery store?

She defines the treatment as a binary variable: whether or not a customer accepts the shopper’s card. And the total amount of money a customer spends at the grocery store in a month is her outcome of interest.

For the sake of simplicity, let’s pretend the analyst does some background research and finds that “income” is the only factor that differentiates the customers who accepted the shopper’s card and those who did not. In other words, customers with a higher annual household income were more likely to accept the shopper’s card compared to customers with a lower annual household income. Moreover, intuitively, the analyst believes that higher-income people are more likely to spend more on grocery shopping. Adding everything together, in this example, income is the only factor affecting the treatment (i.e., whether a customer accepts the shopper’s card or not) and the outcome (i.e., the total amount of money a customer spends at the grocery store in a month).

Figure 1: Confounding Variable

What if the analyst takes the simple difference between the monthly spending by customers who accepted the shopper’s cards and who did not? This naive estimate (i.e., the simple difference) of the causal effect can be broken down as the following:

Naive Estimate = Causal Effect (the green arrow) + Spurious Correlation (induced by the red arrows)

It should be evident that we need to remove the spurious correlation to recover the causal effect. If we can successfully do that, we are more likely to get an unbiased estimate of the treatment effect.

Conditional Apples-to-Apples Comparison 🍎v🍎

We can remove the spurious correlation by employing a conditional apples-to-apples comparison method.

For simplicity, let’s further pretend that only three income groups exist among the customers. These groups are customers with an annual household income of 1) $50,000, 2) $75,000, and 3) $100,000.

If income is the only factor differentiating our treatment and control groups, comparing customers within the same income group will be a“apples-to-apples” comparison. Why? Because as the analyst figured out, based on her background research, other than income, on average, there is no other factor/characteristic which is different between the two groups.

Next, we compare the monthly spending between the treatment and the control groups within the three income groups. By doing so, we get three different numbers. Finally, we take a weighted average of these three numbers to get our estimate of the average causal effect. Figure 2 shows the procedure using a toy example. Also, it illustrates the “bias” in the naive estimate.

**Weight=proportion of the total customers within each income group**

Figure 2: Calculation of the Naive Estimate and the Average Causal Effect. A naive estimate means the simple difference between the average spending by the treatment group (customers with shopper’s card) and the average spending by the control group (customers without the shopper’s card). For calculating the Average Causal Effect, first, we calculate the difference in average monthly spending between customers with and without shopper’s cards within each of the three income groups. Then, we take a weighted average of these three numbers to estimate the Average Causal Effect. Note that, in this toy example, the weight of each of the three income groups is 6/18 because we have 18 customers, of whom 6 are from $50k, 6 are from $75k, and 6 are from $100k income groups.

Comparing Groups with Apples-to-Apples Trends Overtime 🍎v🍎📈

For the conditional apples-to-apples approach to work, we need to know which variables to condition on (i.e., do a within-group comparison of the average outcome of the treatment and the control groups). In our example, we assumed that income is the only factor that affects both the treatment and the outcome. Therefore, conditioning on only income was enough to get an unbiased estimate of the average causal effect.

What if the reality is far more complicated than we have considered? For instance, maybe other than income, a customer’s materialism, impulsivity, and other factors affect the treatment and the outcome. What if we do not have data on all these variables and cannot condition on them? In other words, how should we proceed when we know there is no way to turn the treatment and the control group into apples-to-apples?

In such a scenario, we gather data from a treatment and a control group over an extended period of time before the treatment group received the treatment. Also, we gather data from both groups for at least one post-treatment time period.

Most importantly, we have to make a critical assumption that “had the treatment group not been treated, in the post-treatment period, their average outcome would have evolved in parallel to the average outcome of the control group.”

In our example, the analyst gathers monthly spending data for the shoppers who accepted the shopper’s card (treatment group) and those who did not (control group) before and after the grocery store offered the card.

Figure 3: Comparing the difference in average outcomes before and after the treatment was assigned for the not-apples-to-apples treatment and control groups (with apples-to-apples trends)

Let’s try to break apart the story in Figure 3 step-by-step:

1. The grocery store offered the shopper’s card in September. The analyst gathered average monthly spending data for both groups from May to October (i.e., before and after September).
2. Before the treatment was assigned (i.e., the shopper’s card was offered), the average monthly spending of the two groups was evolving in parallel. Please note that the average outcomes in the two groups were not exactly the same. This is expected because these two groups are not-apples-to-apples. People spending more anyway were more likely to accept the shopper’s card.
3. Most importantly, the average spending over the months in the pre-treatment period was moving in parallel — this is the key to causal inference! We can make a “fair” argument that given the existence of the parallel trends in the pre-treatment period, in a counterfactual world where everything else stays the same except the grocery store does not offer the shopper’s card, the average outcome of the treatment group would have evolved in parallel to the average outcome of the control group (the dotted green line shows this).
4. Given that the above argument is sound, the distance between the solid green line and the dotted green line estimates the treatment’s causal effect on those who received the treatment.
5. Mathematically, we estimate:

Average Causal Effect on the Treated = [Average outcome in the post-treatment period for the treatment group − Average outcome in the pre-treatment period for the treatment group]−[Average outcome in the post-treatment period for the control group − Average outcome in the pre-treatment period for the control group]

**This approach is called a difference-in-differences method**

To conclude, if we are working with non-experimental data, for making a valid causal inference, we can try to make either a comparison of conditional apples-to-apples groups at a specific point in time or a comparison of not-apples-to-apples groups (with apples-to-apples trends) over time. In either case, two other key assumptions of causal inference — no interference and no two versions of the same treatment — must hold.

There are several other methods of causal inference with non-experimental data. But we will leave that discussion for another day.

Thanks for following along!

If you are interested in reading more beginner-level articles on the fundamental ideas in causal inference, read some of my previous posts.

Confounding Variable and Spurious Correlation: Key Challenge in Making Causal Inference

Endogenous Selection Bias: Another Key Issue in Causal Inference

Regression and Causal Inference: How Causal Graphs Helped Me Overcome 3 Key Misconceptions

Regression and Causal Inference: Which Variables Should Be Added to The Model?

*For drawing the figures, I used vector images from Publicdomainvectors.org, which offers copyright-free vector images in popular .eps, .svg, .ai and .cdr formats.