How do I measure effectiveness of my A/B tests?

There are two ways you can see detailed analytics on your A/B tests to determine their effectiveness on your users:

  • Via the analytics button in the A/B Testing section;
  • Or for more detailed reporting, you can look at your A/B testing results as an event in your Funnel and Retention reports.

Analytics in A/B Testing

Click the Analytics button within A/B Testing to see basic analytics on your test, including the conversion rate to a particular event from your experiment and its variants:

alt text

The Analytics section will also show you the number of users participating in each variant as a baseline for looking at conversions.

View A/B test results within Mixpanel reports

When you add an experiment in A/B Testing, Mixpanel automatically tracks an event, property, and super property associated with that experiment so that you can use Retention and Funnels for a more in-depth analysis of your test. For example, you can build a Funnel with multiple steps after the Experiment Started event or use Retention to see how the A/B test affects the long-term behavior of your users:

  • Experiment Started event, which fires only at the point the experiment is delivered to the user’s device.
  • Experiment Id property, which fires along with the Experiment Started event and lets you know which experiment the user saw.
  • Experiments property, which stores a list of all experiments and variants a user has been exposed to as a super property.

Use the Experiment Started event and corresponding Experiment Id property in your reports if you want to understand and analyze an active experiment.

Use the Experiments property and segment by the specific experiment if you want to understand the behavior of users exposed to an experiment previously. Note that if you filter a report by an Experiments property you will be looking at people exposed to that experiment at some point in time, but not necessarily recently, which is why as a best practice, you should always use Experiment Started and Experiment Id to analyze an active experiment.

Other notes on viewing A/B test results within Mixpanel reports

  • The Experiments property is intentionally not available for use in Segmentation reports. The Experiments property indicates that someone has been exposed to an experiment; it does not indicate that user is actively in an experiment. Allowing it to show in Segmentation would be misleading, as it would seem the selected Experiment is associated with events even when it is not.
  • The Experiments property is also not visible on user profiles, though you can filter for it in Explore.
  • You can use the Experiment Started event to target notifications, but you cannot target notifications based on the Experiments property.