How can I effectively A/B test the very first screen or sign-up flow of my app?

To test something like a welcome screen or sign up flow, it is best to understand how fetching an A/B test works and how it affects your ability to test the introduction screen of your application.

How A/B tests work

  • When you first open an app and initialize Mixpanel, Mixpanel’s SDK will immediately grab the A/B test available for the current user. This process requires hitting Mixpanel’s /decide API.

  • Until the api returns a response with the waiting A/B test, the device will not be able to deploy an A/B test.

In this sense, if you target the A/B test towards everyone, the only holdup in terms of grabbing the A/B test right away is the inherent latency between Mixpanel initializing and requesting an A/B test for the current user from the API.

So how can I test my first screen or sign-up flow?

Should you wish to A/B test your initial view, you will want to take this latency into account when qualifying for and then applying the A/B test. Typically, this would mean waiting for the update to be received by your user’s device before showing a screen and then applying the test using a callback. Without a callback in place, there is potential for the end user to see one view and then switch to another view.

The expected usage case is to ensure your app automatically checks for an A/B tests by enabling checkForVariantsOnActive, and then uses the method joinExperimentsWithCallback to run code after the test has been received and applied. Read more about using the joinExperimentsWithCallback method.

Other best practice and recommendations

  • It is best practice to target first screen or sign-up flow A/B tests toward everyone instead of targeting toward specific groups of users. This is due to the fact targeted A/B tests rely on people properties to determine eligibility. If you target the test towards everyone you don't have to worry about a profile existing prior to checking for experiments, and you won't need to create people profiles for every user who opens the app.

  • If you wish to apply targeted experiments on the first view, we recommend adding a loading view to buy time to create profiles for targeting and fetch the experiment. Detailed guidance is outlined below.

How do I make targeted tests on the first view that go to everyone?

Android

  1. Create a user profile for new users as soon as the app is open. You can do this by setting a people property (such as "open date") as soon as the app opens, and executing the flush method to send this data to Mixpanel right away.

  2. Unfortunately, there is no callback available to confirm the profile has been created, so you'd want to build in a buffer (~500ms should do it) before explicitly checking /decide for eligible experiments by calling identify again.

  3. If the check to /decide returns experiments, the addOnMixpanelUpdatesReceivedListener callback should execute. This callback should trigger joinExperimentIfAvailable.

  4. Once you've joined the experiment, you can move from a loading screen to the first view of your app that depends on the Tweak.

iOS

  1. Create a user profile as soon as the app is open. You can do this by setting a people property (such as "open date") as soon as the app opens, and executing the flush() method to send this data to Mixpanel right away.

  2. Unfortunately, there's no callback available to confirm the profile has been created, so you'd want to build in a buffer (~500ms should do it) before explicitly checking /decide for eligible experiments by calling joinExperimentsWithCallback.

  3. Once you've joined the experiment, you can move from a loading screen to the first view of your app that depends on the Tweak.