Marketing Incrementality Testing with Python: A Practical Guide

Introduction

Running your first marketing incrementality test? You’re not alone.

As a growth marketing freelancer, I’ve seen how hard it’s become to measure true marketing impact. With cookies fading out, privacy rules tightening, and users jumping between devices, classic attribution models just don’t hold up anymore.

That’s where incrementality testing steps in. By comparing results between those exposed to a campaign and those who aren’t, you can estimate how much of the outcome was actually caused by your ads, not just correlated.

In this guide, I’ll break down what incrementality really means and how to run a simple test using pandas.

What is Incrementality?

Incrementality is a way to measure the true impact of your marketing campaigns by asking a simple but powerful question:

What would have happened if we had done nothing?

In practice, this means comparing the behavior of a group exposed to your ads (test group) with a similar group that wasn’t exposed (control or holdout group). The goal is to isolate what actually changed because of the campaign — not just what happened during it.

Unlike basic performance metrics (like ROAS or conversions), incrementality doesn’t assume all results came from your marketing. It separates caused outcomes from coincidental ones — filtering out noise from seasonality, organic traffic, or competitor activity.

An incrementality test helps answer:

• How many conversions were generated only because of the campaign?
• Would those users have converted anyway?
• Is this channel/campaign worth scaling?

In short:
Incrementality is about causality, not correlation.
It tells you if your marketing actually moved the needle.

Why is incrementality getting important today?

In a privacy-first landscape, traditional attribution models are losing their reliability. Tracking customers across platforms is no longer feasible due to a combination of regulatory, technical, and behavioral changes.

In a world where tracking is fragmented and consent is harder to obtain, incrementality testing offers marketers a cleaner, more honest answer to the question: “Did our marketing move the needle?”

The decline of third-party cookies, increased restrictions on mobile identifiers like IDFA (iOS 14.5+) and GAID (Android 13+), and strict privacy laws such as GDPR and CCPA have made it harder to follow users across channels and devices. This has left marketing teams with blind spots, especially in multi-touch journeys and upper-funnel activities.

As a result, click-based attribution models (MTA) have become increasingly less accurate. Since iOS 14.5, most users (~70–80%) opt out of tracking. In response, Meta formally introduced incrementality measurement across Europe in July 2025, recognizing the need for a more privacy-resilient metric.

Incrementality compares aggregated outcomes between exposed and unexposed groups, isolating the net effect of your marketing.

Importantly, incrementality is not attribution. It’s about determining if your marketing actually caused a conversion to happen, or if it would have happened anyway.

Incrementality compares aggregated outcomes between exposed and unexposed groups, isolating the net effect of your marketing.

In a world where tracking is fragmented and consent is harder to obtain, incrementality testing offers marketers a cleaner, more honest answer to the question: “Did our marketing move the needle?”

Different Types of Incrementality.

When measuring marketing effectiveness, incrementality can be applied at different levels depending on your goal and data setup. Below are the most common types:

1. Channel-Level Incrementality

This approach isolates the impact of a single marketing channel — for example, paid search or Meta ads, to assess how many additional conversions are driven by that channel alone.

2. Media-Level Incrementality

You’re comparing the effectiveness of multiple media channels or formats, like Meta vs. TikTok, or social ads vs. email. The goal is measuring uplift from specific ads or campaigns, for example, you’re deciding how to redistribute budget across channels or creatives.

3. Campaign-Level Incrementality

This is the highest level: it measures whether an entire multi-channel marketing campaign drove value beyond what would’ve happened anyway.

How to calculate Incrementality?

There are multiple ways to measure incrementality, each with different levels of complexity, accuracy, and resource requirements. The core idea remains the same: isolate the true impact of your marketing activity by comparing a group exposed to the campaign with a group that is not.

Some of the most widely used approaches include: A/B Test, Conversion Lift Studies, Randomized Controlled Experiments, Geo-based Testing, Synthetic Control Methods.

For this guide, we’ll focus on A/B testing as a practical and widely applicable method.

1. A/B Test or Holdout Design

The most direct way to calculate incrementality is by comparing a test group (exposed to ads) vs. a control group (not exposed). A/B Test or Holdout Design in incrementality typically refers to a randomized A/B test where users are randomly assigned to:

• Group A: Users who saw your ads
• Group B: Similar users who didn’t

2. Geo-Based Lift (GeoLift)

If you can’t split audiences directly, you can compare locations (ie, cities, regions): for example, run ads only in the UK and hold out Ireland. You measure the difference in outcomes between geographies before and after campaign launch.

Geolift is an open-source Python library developed and released by Meta (Facebook) in 2022. It is important to remember that Geo-Lift testing is often positioned as a reliable alternative to a real A/B test, but in reality, it’s different and with weaknesses because it relies on a method called “Synthetic Control”. Geo-lift doesn’t randomize users.

3. Time-Based Comparison

If a true control group isn’t available, you can compare performance before and after a campaign but it’s hard to isolate effects due to seasonality or external events.

You should use it only when you have no way to isolate audiences or regions.

Step by Step Code in Pandas in Python

If you are a marketer aiming to learn and implement the incrementality test on pandas you can analyse this very simple code as first example. I use Jupyther notebook for this.

import pandas as pd
from scipy.stats import ttest_ind

# Step 1. Export your campaign Google Ads data as csv
df = pd.read_csv("your_campaign_data.csv")

# Step 2 – Clean numeric columns if needed & EDA analysis
df["Date"] = pd.to_datetime(df["Date"])
df.describe()     # check means, std, min/max 

#Expected dytpe
df = df.dtype
Date              datetime64
Campaign name     object
Location          object
Clicks             int64
cost             float64
CTR              float64
Impressions        int64
ROAS             float64
Conversions        int64
CPA              float64
AOV              float64
dtype: object

# Step 3. Define holdout vs. test
holdout_regions = ["Wales", "Scotland"]
df["Group"] = df["Location"].apply(lambda x: "Holdout" if x in holdout_regions else "Test")

# Step 4. Select time window (e.g., last 3 months)
df_filtered = df[(df["Date"] >= "2024-05-01") & (df["Date"] <= "2024-07-31")]

# Step 5. Split groups
test = df_filtered[df_filtered["Group"] == "Test"]
holdout = df_filtered[df_filtered["Group"] == "Holdout"]

# Step 6. T-test on conversions
t_stat, p_val = ttest_ind(test["Conversions"], holdout["Conversions"], equal_var=False)

print(f"T-statistic: {t_stat}")
print(f"P-value: {p_val}")

If the p-value is below 0.05, your campaign likely had a statistically significant impact on conversions.

Otherwise, you might be spending without incremental lift.

Conclusion

In conclusion, traditional attribution is becoming less reliable due to privacy restrictions and incomplete tracking. This is where incrementality testing, marketing mix modeling, and A/B testing become essential. These methods help identify what truly drives performance by cutting through assumptions and surface-level metrics. When applied with the right tools such as pandas or GeoLift, they support smarter decisions about where to invest and where to reduce spending for funded startups. This approach shifts marketing from guesswork to measurable impact, offering clarity on what is actually working.

Are you a Startup Founder or Head of Growth? Get in touch now!

 

---

Bibliography

Adjust. (2023, June 9). What is incrementality?. Adjust GmbH.

Improvado. (2023, April 25). Incrementality testing: The complete guide. Improvado.

Makeup & Breakup. (2020, August 7). A/B Testing & Cohort Analysis. DIGITAL MARKETING – Connecting the Dots

Govindan, G., Baig, M. R., & Shrimali, V. R. (2021). Data Science for Marketing Analytics: A Practical Guide to Forming a Killer Marketing Strategy Through Data Analysis with Python. Packt

Meta Open Source. (2022). GeoLift: Methodology. Facebook Incubator.