Geo Holdout Tests: Measuring What Your Ads Actually Cause, Without a Data Science Team

Ad platforms report ROAS that includes sales you would have made anyway. A geo holdout test run on your own order data measures the sales your spend actually causes. Here is how it works, how to design one for a WooCommerce or Magento store, and how to turn the result into a budget number.

Every ad platform reports a return on ad spend, and every one of them counts some sales that would have happened anyway. A customer who already had your store bookmarked clicks a retargeting ad on the way to checkout and the platform takes the credit. Add up the conversions claimed by Meta, Google and your email tool and the total is often larger than your actual order count.

The question that matters for budget is narrower: how many extra sales did this spend cause? That is incrementality, and the reliable way to measure it is an experiment. Ad platforms have made their own lift studies cheaper, but they still have floors that many independent stores do not reach. A geo holdout test, run on your own order data, has no such floor. This guide explains how to design one for a WooCommerce or Magento store.

Why not just use the platform’s lift study?

Sometimes you should. Google says that in 2025 it lowered the cost of an incrementality experiment from a previous high of $100,000 to a minimum of $5,000. When Google opened Conversion Lift to Search and Performance Max in October 2026, eligibility required at least 1,000 observed conversions and a minimum campaign budget of $5,000. A store with 300 orders a month does not qualify, and a platform study only measures that platform. A geo test measures your total sales, whichever channel you change.

How a geo experiment works

The method is old and well documented. In Google’s 2011 paper, non-overlapping geographic regions are randomly assigned to a control or treatment condition, and each region receives its condition through geo-targeted advertising. You change spend in some regions, leave the rest alone, and compare what happens to sales.

Comparing raw sales between regions does not work, because regions differ in size and seasonality. The comparison is instead between what the test regions actually sold and a counterfactual: an estimate of what they would have sold without the change, built from how they tracked the control regions before the test started.

Holdout test: actual sales against the counterfactual Line chart over time with three periods: pre-period, test period, cooldown. Before the test, the test regions’ sales and the counterfactual built from control regions overlap. During the holdout, actual sales fall below the counterfactual. The gap is the incremental effect of the paused ads. Pre-period (fit the counterfactual) Test: ads paused Cooldown Shaded gap = sales the paused ads were causing Test regions, actual daily sales Counterfactual from control regions Daily sales
Illustrative. The pre-period shows how well the counterfactual tracks reality. If the two lines do not overlap before the test, the result during the test cannot be trusted.

Several open-source tools build that counterfactual. Meta’s GeoLift uses synthetic control methods and includes a market selection step that simulates candidate test regions before you spend anything. Google’s time-based regression was designed for situations with few geographic units, including single test-versus-control market comparisons. These are not interchangeable: a simulation study by Recast that ran 32,000 synthetic experiments found that four common open-source geo tools recover the true effect with different accuracy depending on the scenario.

Choosing a design

DesignWhat you changeGood forWatch out for
Holdout (go dark)Pause a channel in test regionsChecking whether an existing channel earns its budgetLost sales during the test; keep the test regions small
Scale-upRaise spend in test regionsDeciding whether to spend moreDiminishing returns; the answer applies to the extra spend only
Randomized geoRandomly assign many regionsU.S. states or DMAs, large countriesNeeds enough regions with steady sales
Matched marketOne or a few test regions against weighted controlsCountries with few regions, such as Canadian provincesWider uncertainty; depends heavily on pre-period fit

Running one on a WooCommerce or Magento store

Step 1. Build a daily sales series by region

Use the shipping address on paid orders, not website analytics, so consent and ad blockers do not affect the outcome. In WooCommerce with High-Performance Order Storage, addresses sit in their own table:

SELECT DATE(o.date_created_gmt) AS day,
       a.country,
       a.state                  AS region,
       COUNT(*)                 AS orders,
       SUM(o.total_amount)      AS revenue
FROM wp_wc_orders o
JOIN wp_wc_order_addresses a
  ON a.order_id = o.id AND a.address_type = 'shipping'
WHERE o.type = 'shop_order'
  AND o.status IN ('wc-processing', 'wc-completed')
GROUP BY day, a.country, a.state;

In Magento, join sales_order to sales_order_address where address_type = 'shipping' and use region or a postcode prefix. You need history: Recast’s guide to GeoLift suggests at least six months of daily data or a year of weekly data, broken down by geo.

Step 2. Check power before you start

Pretend the test already happened in last year’s data, with no real change, and see how large a fake effect the method would report. That spread is your noise floor. If the channel you plan to pause plausibly drives 3 percent of sales and the noise floor is 8 percent, the test cannot answer the question; lengthen it, enlarge the test regions, or test a bigger change. The same logic as sample size planning for A/B tests applies, which QuantumLayers covers in A/B Testing Statistics: Sample Size, Significance, and the Mistakes That Produce False Winners.

Step 3. Run it cleanly

  • Run three to six weeks, covering whole weeks so weekday patterns cancel out.
  • Change one thing. No sale, email blast or new product launch that hits test and control regions differently.
  • Exclude the regions that hold your warehouse or a physical shop if local effects could distort them.
  • Keep a cooldown of one or two weeks after the test and keep measuring; some ad effects arrive late.

Step 4. Turn the result into a budget number

A hypothetical example. A store spends $9,000 a month on Meta ads and pauses them for four weeks in regions that normally produce 30 percent of its revenue. That removes about $2,700 of spend.

Counterfactual revenue in test regions$45,800
Actual revenue in test regions$41,200
Incremental revenue lost by pausing$4,600
Spend removed$2,700
Incremental ROAS (iROAS)4,600 / 2,700 = 1.70
ROAS reported by the platform for the same period3.40
Incrementality factor (iROAS / reported ROAS)0.50
Gross profit per ad dollar at a 40% margin1.70 x 0.40 = $0.68

The platform says every dollar returns $3.40. The experiment says it returns $1.70 in revenue and $0.68 in gross profit, so at current spend the channel loses money. The incrementality factor of 0.5 can then be applied to the platform’s daily reports until the next test, which is far more useful than either number alone. Report the confidence interval with the point estimate; an iROAS of 1.70 with an interval from 0.4 to 3.0 is a reason to run a longer test, not to cut the budget.

Common ways geo tests go wrong

  • Spillover. People see ads in one region and ship to another (a gift, an office address). Larger regions reduce this.
  • Targeting leaks. Platform geo-targeting uses the viewer’s estimated location, which is not always accurate. Check delivery reports by region during the test.
  • One oversized region. A test region much larger than any control is hard to match. Recast’s study includes exactly this scenario because it is common.
  • Reading the result as causal for everything. A holdout measures the effect of the spend you removed, at that spend level, in that season. It does not tell you what doubling the budget would do. QuantumLayers’ piece on correlation versus causation is a good reminder of how narrow a causal claim usually is.

Where this leads: calibrating a marketing mix model

One test answers one question. A marketing mix model estimates every channel at once from historical spend and sales, but it needs real experiments to keep it honest. Google released its open-source MMM, Meridian, to all marketers in January 2025, and in September 2026 made Meridian GeoX, which runs geo experiments and feeds their results into the model, generally available. For a store with modest spend, two or three well-run geo tests a year on the largest channels will do more for budget decisions than a model built without them.

Sources