A/B testing your videos

In short: we hide your video widgets from a small, random slice of your visitors and compare what those visitors spend against everyone else. After a few weeks you get a straight answer to "are these videos actually making me money?"

When you launch a test, every visitor to your store is sorted into one of two groups:

Which group a visitor lands in is decided from the anonymous ID already stored in their browser, so the same person always sees the same version — no flicker, no "the videos were there yesterday" support emails. The split happens at our edge network before the page finishes loading, so there is no delay and nothing flashes on screen.

We then measure both groups on:

You decide how big the holdout is. This is the most important choice you'll make, and it's the one most people get backwards.

A smaller holdout is not the safer choice — it's the slowest one. The smaller group is what decides how long the test takes, because both groups need enough visitors before the difference between them means anything. A test that finishes in 68 days at a 50/50 split can take over 300 days at a 10% holdout. That's not a test; that's a year of waiting.

When you move the slider, we show you the estimated duration right there. If it warns that the test is impractical, move the slider closer to 50%.

Duration is driven by one thing: how quickly the smaller group fills up with visitors. So it depends on your traffic, not on how long the test has been switched on.

Rough guide, assuming a 2% store conversion rate and looking for a 15% improvement:

Two things to read off that table:

These are estimates for detecting a modest 15% difference. If videos are doing much more than that for your store — or much less — you'll usually see it sooner, because a big difference stands out from the noise faster than a small one. Your store's own numbers also matter: a higher conversion rate shortens the test, a very spiky order value (a few huge orders among many small ones) lengthens it.

To calculate that estimate we need your store's visitor count and conversion rate, so the first time you open the form we'll ask permission to read your Shopify Analytics. It's read-only, we only use it for this estimate, and you can decline — the test still runs perfectly, you just won't see a duration prediction up front.

Next to the split you'll find a second dial — how big a difference the test should be built to detect. It is the other half of the duration equation, and it moves the numbers hard, because the required traffic grows with the square of the precision you ask for:

(At a 2% store conversion rate.)

Pick "big changes only" if your traffic is modest and the question you actually have is "are these videos pulling their weight at all?" — you'll get an answer in weeks instead of months, at the cost of not being able to detect a small-but-real improvement. Pick "small changes" only if you have the traffic to spend on precision.

You can name a specific page — a product, a collection, your home page — as the page you care about.

This does not limit where the test runs. The holdout group has videos hidden everywhere, on every page, for the whole test. What the page setting does is give you a second, cleaner view of the results: "all visitors" versus "only the visitors who actually reached this page."

That second view is usually the one to trust. Plenty of visitors never land anywhere a widget appears, and including them dilutes the real effect.

Every test moves through the same stages, shown as a badge on the results page.

Two things are worth knowing about why we make you wait.

A single good day is not a result. If we called a winner the first time the numbers crossed the line, we'd be wrong about one test in three — we simulated it. So a reading has to hold before we call it, which is why you may see "Leaning" for a while on a test that already looks decided.

We use the same minimums the rest of the industry does — at least a full week (to cover a whole weekly cycle) and at least 300 orders in each group before any verdict. If your test is stuck at "Leaning", the results page tells you which of those is still missing.

How long the early phase lasts is purely a matter of traffic. A store doing a million visitors a month is through it in a couple of days; at 50,000 visitors a month it's the first month or two. Use the table above to know which one you are — acting on an early swing is the single most common way to draw the wrong conclusion.

A "no difference" result is a real, useful answer, not a failed test. It means the effect is smaller than the test was built to see.

A single enormous order can swing a whole group's average and make a test look decided when it isn't. For the statistics only, we cap unusually large orders (above the 97.5th percentile of that day's orders, applied identically to both groups). The revenue figures you see are never capped — that's your real money. This is standard practice and it makes tests conclude about 14% sooner.

When you end a test, you choose whether to roll out a winner. Rolling out simply means the holdout stops: everyone sees the videos again.

Ended tests keep their results, so you can come back to the numbers whenever you want. We also process one final day of data after you stop, so the last day isn't cut off halfway.

Two things are locked once a test is live, and it's worth knowing why:

You can still rename a live test at any time.

One test at a time. Two overlapping holdouts split each other's groups, and neither produces a usable number, so we only allow one running at a time.

"My widgets disappeared!" — If you're in the holdout group, that's the test working correctly. Open your store in a private/incognito window to get sorted again, or check the test's results page, where both group sizes are shown.

"The numbers look identical." — With a small holdout in the first weeks, they often do. Check the estimated duration on the test; if it's very long, end the test and relaunch with a larger holdout.

"Revenue here doesn't match my Shopify reports." — These numbers are built from visitor sessions attributed to the test window only, and converted to a single currency. They will not tie out to Shopify's totals, and aren't meant to — what matters is the comparison between the two groups, which is measured identically on both sides.

"Can I see which group I'm in?" — Not in the dashboard. Assignment is anonymous by design; we never store a per-person record of it.