Interaction Detection

Learn how to detect interactions between overlapping experiments and understand their impact.

What is Interaction Detection

When you run overlapping experiments, it is possible for them to interfere with each other. Interaction Detection lets you pick two experiments and evaluate them for interaction. This helps you understand if people exposed to both experiments behave very differently from people who’re exposed to either one of the experiments. Essentially you want to answer this question: Does being in both experiments change the effect on a metric in a way that’s different than just summing the individual effects?

Should I worry about it?

Our general guidance is to run overlapping experiments. People seeing your landing page should experience multiple experiments at the same time. Our experience is echoed by all avid experimenters (link). Teams expecting to run conflicting experiments are typically aware of this and can avoid conflicts by making experiments mutually exclusive via Layers (also referred to as Universes).

How to use it

You can initiate an Interaction Detection analysis as a special type of custom query on top of an Experiment (Experiment -> Results -> Explore -> Interaction Effect Detection)

You can pick a second experiment to analyze (along with metrics) and we will show you a quick summary.

In each section:

• By Groups: gives you the unit counts based on the collective slice of group assignment
• Metric Summary Results: shows you the estimated intervals of difference in metric lift
• Overlapping Unique Users: offers an overview of the general traffic intersection of the two experiments

See more examples in this article.

Methodology

Assume you have two experiments A & B, both have two experiment groups control & test. The interaction effect will calcuate the overlapping impact on users who are exposed in both experiment A & experiment B.

$$\Delta_{\text{treatment effect}} = (\overline{X_{testA \cdot testB}} - \overline{X_{testA \cdot controlB}}) - (\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}})$$ $$\Delta_{\text{treatment effect}}{\%} = \frac{\Delta_{\text{treatment effect}}}{(\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}})}$$ $$variance = \frac{Var(X_{testA \cdot testB})}{n_{testA \cdot testB}} + \frac{Var(X_{testA \cdot controlB})}{n_{testA \cdot controlB}} + \frac{Var(X_{controlA \cdot testB})}{n_{controlA \cdot testB}} + \frac{Var(X_{controlA \cdot controlB})}{n_{controlA \cdot controlB}}$$

Intuition

• The part $\overline{X_{testA \cdot testB}} - \overline{X_{testA \cdot controlB}}$ tells you: “If someone is in A’s test, how much does changing B from control→test change the metric?”
• The part $\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}}$ tells you: “If someone is in A’s control, how much does changing B from control→test change the metric?”
• By subtracting those two, you see whether the effect of B differs depending on whether you’re in A’s test or control group → that difference is the interaction effect.

If that result is not statistically signficant, then the effect of B is the same no matter which group you’re in for A (so no significant interaction). If it’s statistically signficant, then the effect of B depends on A’s assignment (so there is a significant interaction).

Directionality

The direction (positive or negative) of the interaction effect tells you whether the combination of the two treatments amplifies or dampens each other’s effects.

Think of the interaction effect like this:

$Interaction Effect=(Effect of B when A=Test)−(Effect of B when A=Control)$

So:

• Satisitically Positive interaction → B’s effect is stronger when A is also “on”
• Satisitically Negative interaction → B’s effect is weaker (or reversed) when A is “on”

Magnitude

The magnitude (absolute size) of the interaction effect indicates how much the combined impact deviates from simple additivity.

For example:

• Experiment A produces a +5% lift on your metric of interest
• Experiment B produces a +5% lift on your metric
• The interaction effect between A and B is –3%

This means that when both experiments are rolled out together, their combined impact is expected to be about 3% lower than the sum of their individual effects — in other words, the features interfere with each other.

To obtain the most accurate estimate of the true combined impact, it’s best to run a new experiment that includes both features together. Because experiments A and B may not have run over exactly the same time period or under identical conditions, differences in timing and seasonality can slightly influence the measured interaction magnitude.

When an experiment includes more than two groups, interaction effects are evaluated pairwise between groups. You can view the interaction effect for specific group combinations by selecting the desired groups in the “Select Comparison” dropdown, as shown in the UI above.