How to tell if AI creative is working

Set up a fair test

• Test one variable, the image. Keep the copy, offer, audience, placement and budget the same.
• Write the hypothesis first, for example "The AI lifestyle image will get a lower cost per purchase than our studio photo."
• Use the platform’s test tool. Meta’s A/B test compares up to 5 versions and splits the audience so nobody sees more than one; Meta warns that turning ad sets on and off by hand gives unreliable results.
• Give each version the same budget, as Meta recommends.
• Run at least 7 days. Meta allows tests of 1 to 30 days and suggests longer runs, such as 10 days, when customers take more than a week to buy.
• Use an audience you are not reaching with other campaigns at the same time.
• On TikTok, use Split Test. TikTok says it picks a winner at a 90% confidence rate, and creative is one of the variables it tests.

Which metric answers which question

Meta names the version with the lowest cost per result for the key metric you chose as the winner, so pick the metric that matches the goal before launch. A click won by an image that misrepresents the product can come back as a return, which is why the returns row belongs in every AI test. Why AI changes your product lists what to look for.

How many results you need

A test needs enough people to tell a real difference from luck. Statistician Evan Miller gives a rule of thumb for a test with a fixed size: n = 16 × p × (1 − p) ÷ δ², where p is your current conversion rate and δ (delta) is the smallest change you care about. His calculator reports the answer per variation.

Example: your current image converts 3% of clicks (p = 0.03), and you want to detect a rise to 4% (δ = 0.01). n = 16 × 0.03 × 0.97 ÷ 0.0001 = 4,656 clicks for each version, or 9,312 in total. At $1 a click, that is about $9,300 of spend. To detect half that change, from 3% to 3.5%, you need four times as many clicks.

If the numbers are out of reach, test a bigger difference, such as a new concept instead of a new crop, or judge on a more frequent event, such as clicks, and treat the result as a first signal. How many ad creatives you need explains the difference between a concept and a variation.

Brand lift and conversion lift

An A/B test says which ad did better. A lift test says whether the ads caused sales or awareness at all, by holding back a control group that sees no ads. On Meta, a Brand Lift test needs at least $120,000 in ad spend over the last 90 days, and a Conversion Lift test needs a high-quality conversion data source such as the Conversions API. Meta notes that lift tests can raise your cost per thousand impressions, because of the control group.

Read results without fooling yourself

• One win on one product is one data point. Repeat the test on a second product before you change how you make all your creative.
• Match the ideas. If the AI version also has a better scene or message, you cannot tell which change won.
• Break results down by placement. An image can win in Stories and lose in Feed.
• Watch returns for the full length of your return window after the test ends.
• Keep the losers on file. Knowing which scenes and messages lost saves you from making them again.
• Expect the winner to tire. A winning image still fatigues; see what creative fatigue is for the signs.

A test plan you can copy

Hypothesis: The [AI image or concept] will get a lower [cost per purchase] than [current image].
Variable tested: the image only. Copy, offer, audience, placement, budget: identical.
Platform and tool: [Meta A/B test / TikTok Split Test]
Key metric: [cost per purchase]. Also watch: [CTR, conversion rate, return rate]
Current rate (p): ____   Smallest change worth knowing (δ): ____
Sample size per version = 16 × p × (1 − p) ÷ δ² = ____ clicks
Budget needed = sample size × 2 × cost per click = $____
Dates: [start] to [end] (at least 7 days; at most 30 on Meta)
Stop rule: no decision before the end date.
After the test: check returns and reviews for "looks different" until [date].
Decision: [scale / retest / drop]. Owner: [name]

Questions people also ask

Should the AI version carry an AI label in the test?

Follow the platform’s labeling rules for both versions. If the AI version needs a label, the label is part of what you are testing. See do you need to label AI images in ads.

Can I trust the winner Meta shows?

Meta puts a trophy next to the version with the lowest cost per result for your key metric, and recommends extending the test when it ran too short to find a winner. Check that the test reached your planned sample size before acting on it.

What if the AI image wins on clicks but loses on sales?

The image promised something the product page or the product did not deliver, or it drew the wrong people. Check whether the AI image changed the product or the setting in a way the listing does not match, and watch returns.

Sources

Checked on September 24, 2026. Prices, specs and rules change; follow the links for the current versions.

1. 1.Meta Business Help Center: About A/B testing
2. 2.Meta Business Help Center: Best practices for A/B testing
3. 3.Meta Business Help Center: About Experiments (A/B, Brand Lift, Conversion Lift)
4. 4.Meta Business Help Center: Viewing and understanding A/B test results
5. 5.Meta Business Help Center: 3-second video plays rate per impressions
6. 6.TikTok Ads: Split testing
7. 7.Evan Miller: How not to run an A/B test
8. 8.Evan Miller: Sample size calculator