A/B testing

Run two versions of a widget side by side, split your traffic between them, and keep the one that sells.

An A/B test shows each shopper one of two versions of a widget: variant A or variant B. Foxr counts views and orders for each, and tells you when one is ahead by more than chance.

Where to find it: click Widgets in the app menu, open a widget, and click the A/B test tab.

How to run your first A/B test

Plan on about a week. Change one thing, so you know what made the difference.

Pick one thing to test, for example Stories against Carousel, or a new heading.

Open the widget and click the A/B test tab, then Create variant B.

With Variant B selected, change that one thing on its tab.

Back on the A/B test tab, turn on Run this test. Leave Traffic to variant B at 50%.

Click Variant A in the top bar, then click Save. The test starts.

Wait. Give each version at least 500 views, and let the test run at least a full week.

Open the widget's analytics and read the verdict. See Read the results.

When there's a winner, click End test on the Widgets page and keep it. See End a test and keep a winner.

The next sections explain each part.

What you can test

Almost anything about the widget:

The layout: switch between Stories, Carousel, Grid, and Pop up. Product page widgets can only test against Product page.
Sizes, shapes, rings, colours, and product chips.
The player: fullscreen, mini player, or playing inside the card.
The heading, clip order, and minimum clips.
The clips themselves: a different selection or a different order.

Two things are shared by both versions: the Widget name and the Placement. That keeps the test fair, since both versions run in the same spot.

Some widget settings can be overridden by your shop-wide Settings, so testing them may change nothing:

The add-to-cart button colours always come from Settings → Appearance.
A widget's share button, Start with sound off, and Show captions in the player only make a difference while the matching Settings option is off. Show share button is on in Settings by default.

On the A/B test tab, click Create variant B. Variant B starts as an exact copy of your widget, and the editor switches to editing it. Variant A and Variant B buttons appear in the top bar.

With Variant B selected, change what you want to test on any tab. To change its clips, use the Media tab. The preview shows the variant you're editing.

Back on the A/B test tab, check What variant B changes. It lists every difference, for example Layout: Stories → Carousel.

Turn on Run this test, and set Traffic to variant B. The default, 50%, splits traffic evenly.

Click Variant A in the top bar, then click Save.

Switch to Variant A before you save

Saving while Variant B is selected resets the widget's Placement to Global / any page. Variant B's own settings are saved correctly. Always click Variant A before Save. If it has already happened, click Variant A, set Placement again on the Media tab, and save. A Product page widget goes back to Product pages by itself when you save from Variant A.

The test starts as soon as you save. The Widgets page shows A/B test · 2 variants and how long it has been running.

Test one change at a time

If variant B changes more than three things, the tab warns you. A win then tells you which version sells better, not which change did it. To learn something you can reuse, change one thing per test.

If variant B is still identical to variant A, there's nothing to measure. Saving keeps variant B, but saves the test as paused, with Run this test switched off. Change something in variant B, turn Run this test back on, and save again.

On the A/B test tab you can also:

Click Open variant B to switch to editing it.
Click Copy variant A into B to throw away variant B's changes and start again from an exact copy of variant A. The test stays.

How shoppers are split

Each shopper is assigned A or B at random, using your Traffic to variant B percentage. The field goes up in steps of 5.
They keep seeing the same version on later visits, because the choice is saved in their browser.
Shoppers with browser storage blocked may get a new draw on each page. Private windows keep their version until the window is closed.
Changing the percentage only affects shoppers who haven't seen the test yet. Setting it to 0% or 100% sends every new shopper to one version, but returning shoppers keep the version they had.
Your own visits are assigned a version and counted too. A private window gives you a fresh draw.

Pause a test

Open the widget and click the A/B test tab.

Turn off Run this test.

If Variant B is selected, click Variant A, then click Save.

Everyone sees variant A while variant B waits. The numbers already collected are kept, and the Widgets page shows A/B test paused. Traffic during the pause counts toward the widget's totals but toward neither version, and the Analytics A/B section is hidden until you turn the test back on. The test's running time isn't reset. To carry on, turn Run this test back on and save. Shoppers get the same version as before.

You can also set Traffic to variant B to 0% or 100%. That sends all new shoppers to one version while the test stays on.

Read the results

Results appear in two places:

Where	Shows	Date range
Widgets page	Click the arrow next to the A/B test badge to see Variant A and Variant B rows with the columns you've turned on under Performance metrics: Views, Orders, CR, and Revenue. A version gets a Winning badge once it's significantly ahead.	All time
Analytics for the widget	The A/B test section shows what variant B changes, the verdict, orders per view for each version, and a table with views, plays, add-to-carts, orders, revenue, and orders per view.	The range you pick, 30 days by default

The editor's A/B test tab doesn't show results.

To read the full results:

Click Widgets in the app menu.

On the widget's row, click the chart button (View analytics).

Open the Date range dropdown and pick All time if the test has run for more than 30 days.

Scroll to the A/B test section. Read the verdict at the top first.

Compare the Orders per view bars and the table. If it says Still collecting., come back when the Traffic collected bar is full.

The verdict

Message	Meaning
Still collecting.	At least one version has fewer than 500 views. A Traffic collected bar shows how close you are.
Variant A wins or Variant B wins	One version gets more orders per view, and a gap that size is unlikely to be chance. The message gives the size of the gap and how likely it is to be chance.
No clear winner yet.	Both versions have enough views, but they convert too similarly to call.

A win reads like Variant B wins: 23.4% more orders per view, and a difference this size would happen by chance about 1.2% of the time. The percentage always compares variant B with variant A. So when Variant A wins it says fewer, meaning variant B got that much fewer orders per view.

Pick All time for long tests

The A/B section on the Analytics page only counts the date range you've picked. If your test has run for more than 30 days, choose All time to see all of it. That's also why the verdict on the Analytics page can differ from the Winning badge on the Widgets page, which always uses all time.

How the winner is decided

The test compares orders per view: the orders a version earned, divided by the times it was shown. Revenue and add-to-carts are shown, but they don't decide the winner.
Each version needs at least 500 views before a winner can be called.
A winner is called only when the difference is statistically significant at 95% confidence. That means a gap that size would happen by chance less than 5% of the time.
A view is one page load on which the widget was shown, not one shopper. A shopper who visits five pages counts five views. For a Pop up, page loads where the bubble never appeared count too.
Orders are credited to the version the shopper saw when they added the product. See How sales are credited.

There's no automatic end date or auto-winner. A test runs until you end it. Let it run at least a full week, so weekdays and weekends are both included. To estimate how long you need, divide 500 by the views one version gets each day. A widget with 200 views a day at 50% gives each version 100 a day, so about 5 days to reach 500.

End a test and keep a winner

On the Widgets page, click End test on the widget's row. It's there while the test is running or paused.

Read the recommendation at the top of End this A/B test?. If one version is significantly ahead, it's marked · winning and selected for you. Otherwise Variant A is selected.

Under Which version do you want to keep?, choose Variant A or Variant B. Each shows its all-time views, orders, conversion rate, and revenue. If you pick the version that isn't ahead, a warning reminds you, but you can still do it.

Click Keep variant A or Keep variant B.

The version you keep becomes the widget, and the other is discarded. This can't be undone. Traffic stops being split straight away, and the widget's own totals keep all their history.

After ending, the Widgets page shows Past A/B test with each version's final numbers. The A/B section on the Analytics page only shows while a test is running, so note the results first if you want them.

To keep variant A from inside the editor instead, click Delete variant B on the A/B test tab, then save.

Good to know

Deleting a clip that variant B relies on can end the test: if variant B is left with no clips, it's removed and variant A is kept.
A new test on the same widget adds to the same Variant A and Variant B numbers as earlier tests. The Winning badge and the End test recommendation always use the combined numbers. To look at the new test alone, pick a 7, 30, or 90-day range on Analytics that starts after the old test ended.
Changing variant A's layout to or from Product page also changes variant B's layout to match.
Deleting the widget deletes its test numbers too.

If your results don't look right

What you see	Why, and what to do
No A/B test section on the widget's Analytics page	The test isn't running. Check that Run this test is on and that variant B differs from variant A, then save. The section is also hidden after you end a test.
Still collecting. for a long time	One version has fewer than 500 views. Check Traffic to variant B: at 0% or 100%, only one version gets new shoppers. Otherwise, wait for more traffic or widen the date range.
Both versions show 0 orders	Orders are credited through the attribution pixel, only for shoppers who allow tracking. See Analytics show zero or no revenue.
The Analytics verdict and the Winning badge disagree	The badge uses all time. Set the Analytics range to All time.