A/B testing for Shopify

Prove your Shopify videos make you money

Run an A/B test on your Shopify store: half your visitors see it with videos, half see it without. Stores with 100,000+ monthly visitors get an answer in about five weeks, and bigger stores sooner.

Talk to sales
Illustration of a Storista A/B test results screen for a Shopify store, using sample numbers: Control and Variant groups compared on visitors, revenue, conversion rate and revenue per visitor, with the Variant marked Significant
Illustration of a result in the Storista Shopify app, with sample numbers, not customer data. Control is the group without videos; Variant is the group with videos.

Why run an A/B test

Watchers buy more in your Shopify store. Is it because of the video?

Storista Analytics compares shoppers who watched your videos with shoppers who didn't. It's a useful number with one catch: people decide for themselves whether to watch, and the ones who stop for a video were often going to buy anyway.

An A/B test takes that decision away from them. We sort visitors at random, so the two groups differ in one way only: whether your videos were on the page. If the group with videos spends more, that gap is what your videos earn.

When the budget comes up for review

You'll have revenue per visitor with videos and without, from your own Shopify store and your own shoppers. That holds up better than a case study about somebody else's.

When you're deciding what to film next

Name the product page or collection you care about and see the effect among visitors who actually got there. If videos aren't moving revenue on that page, the test will say so, and that's worth knowing before you pay for another shoot.

How it works

You launch it once and it covers your whole store

There are no variants to design and no pages to copy. One group gets your store with videos, the other gets it without.

Visitors are split at random

You choose the split, and we sort each visitor into the group with videos or the group without.

One group sees your store without videos

For them, video widgets are hidden on every page until the test ends, so they see your Shopify store as it looked before you installed Storista. Everyone else sees no change.

Nobody sees the page change

We sort visitors at our edge network before the page finishes loading. Nothing flashes or loads late, and a shopper who comes back tomorrow gets the same version they saw today.

Both groups are measured the same way

We track revenue per visitor and conversion rate for both groups over the same days, using your real revenue. Revenue per visitor is the headline number. When the result is in, end the test and everyone sees your videos again.

Choosing your split

Uneven splits feel safe and drag your test out

How long a test takes depends on the smaller group, because both groups need enough visitors before the gap between them means anything. A test that wraps up in 68 days at 50/50 can need more than 300 days at 90/10. Nobody wants to wait most of a year for an answer.

When you set the split, Storista uses your Shopify Analytics, if you allow it, to estimate how long the test will take, and warns you if it won't finish in a reasonable time.

Monthly store visitors50 / 50 split90 / 10 split
1,000,000~4 days~2.5 weeks
500,000~1 week~5 weeks
100,000~5 weeks~5 months
50,000~9 weeks~11 months
10,000~11 monthsNot realistic

Rough figures, assuming a 2% store conversion rate and a 15% improvement to detect. Large effects show up sooner, and a higher conversion rate shortens the test. With modest traffic, you can set the test to look only for big changes and get an answer in weeks rather than months.

Reading the result

A status badge in dashboard tells you when it's safe to act

Collecting data

The first day or two. Let it run.

Gathering results

Numbers jump around here. Don't act on them yet.

Leaning

A direction is showing. OK for low-stakes calls.

Significant

The result held up. Act on it.

No difference

Nothing meaningful showed up, which is an answer too.

How we calculate results

Shopify video A/B testing questions

Will shoppers notice the test?

No. We sort visitors at our edge network before the page finishes loading, so there's no flash and no delay. A shopper keeps the same version on every visit.

Is my Shopify store big enough to run a test?

If you have roughly 50,000 or more visitors a month, yes. A 50/50 test takes about nine weeks at 50,000 visitors and about five weeks at 100,000. Below that, set the test to look only for big changes, or wait until traffic grows: at 10,000 visitors a month even a 50/50 test takes close to a year.

Does hiding videos from half my visitors cost me sales?

It can while the test runs: if your videos lift revenue, the group without them misses that lift. That's why we recommend 50/50. It reaches an answer fastest, about five weeks at 100,000 monthly visitors, so you aren't paying for it for long. When the test ends, everyone sees your videos again.

Why don't the results match my Shopify reports?

Results come from visitor sessions inside the test window, converted to one currency, so the totals won't line up with Shopify's. They aren't meant to. Compare the two groups with each other, since both are measured the same way, and use your usual reports for total revenue.

Can I change or pause a test once it's running?

No. Changing the split or target page would rewrite the days already recorded, and pausing would show videos to the group that isn't supposed to see them. If you need a different setup, end the test and start a new one; you keep the data collected so far. You can rename a live test at any time.

Can I run more than one test at once?

No, one at a time. Two overlapping tests would split each other's groups and neither would give you a usable number, so run them back to back.

Can I see which group a shopper is in?

No. Assignment is based on the anonymous ID already stored in the shopper's browser, and we never keep a record of which group any individual was in.

Still stuck?

See what your videos are worth to your Shopify store

Talk to sales