Crusader of Friendship

Recently, I found myself needing to build an A/B test for a section of our onboarding interface. For those of you unfamiliar with A/B testing (which included myself up until this point 😅), you can find more details on the A/B testing wiki page. The gist is to serve two different variations of some component: the existing, control version, and the new experimental version. With these two versions, we want to assign visitors one of two groups when they land on the page containing our A/B test, and serve them the corresponding version of the component.

Easy enough right? At least, this is what I thought going into this project. But I definitely ran into some challenges while coming up with a solution, and so I wanted to share my findings with y’all in case you come across something similar.

The final result is an A/B test component that accepts the following props:

• name – A unique A/B test name
• control – A control component
• experiment – An experiment component
• onComplete – An optional callback fired after the A/B Test component finishes serving a treatment.
• start – An optional start timestamp
• end – An optional end timestamp

You can find a slightly revised implementation on repl.it. I’ve removed some of the project specific code, and added in some placeholders for things like fetching or storing values to a database. Below, I’ll go over each part of the code and talk a bit about my approach along the way.

Storing Groups

The tricky part of this project is actually ensuring that returning visitors always see the same version of a component. It wouldn’t be much of an A/B test if visitors are able to see both control and experiment treatments. In order to do that, we’ll need to persist the result of our A/B test assignment somehow:

const getCachedGroup = name => window.localStorage.getItem( name )

const setCachedGroup =
  ( name, value ) => window.localStorage.setItem( name, value )

// Placeholder async request to fetch from db.
const fetchGroup = name => fetch( 'example.com?abtest=name' )

// Placeholder async request to persist to db.
const persistGroup = ( name, value ) => {
    fetch( 'example.com, { 
        method: 'POST',
        body: JSON.stringify( { name: value } ),
    } )
}

To do this, I decided to rely on two layers of storage. First, the browsers local storage, and second, a fetch to the site’s backend that gets or sets a key, value pair to a database. In my case, I am working with WordPress, and because the test is meant to target site admins, I am able to take advantage of the sites options table.

Get and Set

With these helper functions, we can now create the primary logic for getting and setting group assignment. Here is what I came up for this:

const getAndSetGroup = async name => {
 	try {
    	const storedGroup = await fetchGroup( name )
    	const group = storedGroup || getRandomGroup()

    	if ( ! storedGroup ) {
      		persistGroup( name, group )
      		recordABTestStart( name, group )
		}
		
		setCachedGroup( name, group );
    	return group;
	} catch( err ) {
    	setCachedGroup( name, 'control' );
    	return 'control';
	}
}

This function will only be called when a group is not already present in localstorage for a given test. It’s not uncommon for visitors to change browsers, or clear browser data, so it’s important we don’t assume that because a group is not stored in localstorage it won’t be present when fetching from the database. And so, the first thing we have to do is attempt to fetch from the backend. If a group does exist for the test, we go with that one, otherwise, we assign a new random group and proceed to store this in cache with name as the key and group as the value, returning our selected group once we finish up.

One important thing to note here is we only persist the selected group to the database if fetching doesn’t return a meaningful value. In this case, the visitor is viewing the test for the first time, and it’s via this code path that we need to store the assignment for the first time. Also note that, in theory, each visitor should only enter this code path once, and so this is also the point where we should record a visit for our A/B test tracking.

One final point of interest is the try/catch block used here. If for some reason during the fetching, assigning, and storing process something goes wrong, we need to make sure our visitor doesn’t notice somethings gone wrong, and so we default to the control group. If something did go wrong, this visitor will not be counted in the test as they would not have been recorded. Further, because control is saved to localstorage, future visits will bypass getting, setting, and recording altogether.

Odds and Ends

In getAndSetGroup, you’ll notice a few functions we have not talked about yet. These are just additional helper functions to help keep the code clean. Here are the remaining helpers that have not been shared yet:

const getRandomGroup = () => Math.random() < 0.5 ? 'control' : 'experiment'

// Placeholder tracking for A/B Test.
const recordABTestStart = 
  ( name, group ) => console.log( `Test ${ name } start.`, `Assigned to ${ group }.` )

const isActive = ( start, end ) => Date.now() >= start && Date.now() <= end

Hopefully these are fairly straightforward, but just in case, here is a quick summary of what they do:

getRandomGroup – Assigns a random group, with a 50% chance of selecting each. Note that it is a good idea to extract the .5 into a size parameter, but for this example, I’ve left this hard-coded as is.

recordABTestStart – This just logs the test name and assigned group. This is meant to be a placeholder for whatever method you use to track your A/B test.

isActive – Given a start and end timestamp, this tells us if a given test is currently active.

The ABTest Component

Finally, we’re ready to put it all together and create our actual component. Here is the code for this:

import React, { useCallback, useEffect, useState } from 'react'

const ABTest = ( {
    name,
	control,
	experiment,
	onComplete,
	start = 0,
	end = Infinity,
} ) => {
    const [ group, setGroup ] = useState( 'control' );
    const [ isFetching, setIsFetching ] = useState( true );
    const active = isActive( start, end );
    const handleComplete = useCallback( () => {
        setIsFetching( false );

        if ( onComplete ) {
            onComplete()
        }
    }, [ onComplete ] )

    useEffect( () => {
        if ( ! active ) {
            handleComplete();
            return;
        }

        const cachedGroup = getCachedGroup( name );
        if ( cachedGroup ) {
            setGroup( cachedGroup );
            handleComplete();
        } else {
            ( async () => {
                const group = await getAndSetGroup( name );
                setGroup( group );
                handleComplete();
            } )();
        }
    }, [ active, handleComplete, name ] );

    if ( isFetching ) {
        return null;
    }

    return group === 'control' ? control : experiment;
}

There’s a lot going on here, but to start, you’ll notice the expected props match what we defined earlier in the post, and that we’ve set defaults for start and end so that if they are omitted, isActive will always return true. We’ve also defined some constants at the start of the component, using React hooks for group and isFetching state and a handleComplete function to be called whenever the ABTest component is done doing its thing.

Let’s skip the call to useEffect for now and have a look at the final few lines of the component. You’ll notice that we first check if isFetching is true before returning either control or experiment. This is because the component will render once before performing any of the useEffect logic, and so we need to be sure neither of the two test components is returned prematurely. Once the component re-renders, after useEffect is executed, group will be correctly set, and isFetching will be false via our handleComplete function. Only then can we safely return one of the two treatments.

Going back to useEffect, you’ll notice the first thing we do is check if the experiment is active. If it isn’t, we need to bail early, and trigger our complete function. Since the default value of group state is control, the visitor will be shown the unchanged version of the component, so no harm no foul. Next we check for a cached group in localstorage and if it exists, we set our group state accordingly, then call our complete function. At this stage, the visitor will see whatever treatment was stored in localstorage. It’s worth noting that visitors going down this code path are likely seeing the test again after having already visited one or more times.

Finally, if the test is active and there is no cached group, then we need to call our getAndSetGroup method defined earlier. Because this function is asynchronous, it can take a few seconds to complete, so we wrap this in an async IIFE, and place our logic for setting group state and calling our complete function inside.

Loading…

This is pretty much the implementation for our ABTest component. But there is still one point of discussion. Loading. You may have noticed that although we are dealing with asynchronous functions, we have not dealt with any loading state inside of our ABTest component. This is actually what the optional onComplete prop is for. We offload that responsibility to whatever parent is rendering ABTest. You may have noticed we check to see if onComplete is defined, and if it is, we call it inside of our handleComplete function. This means that the parent component can pass ABTest a callback so that it can trigger a loading complete state at once everything is all set.

In Conclusion

In the end, we can utilize our ABTest component like so:

...

<ABTest
    name="my_abtest_name"
    control={ <ControlComponent /> }
    experiment={ <ExperimentComponent /> }
    onComplete={ this.handleABTestComplete }
/>

You can test this with the repl.it project linked earlier from the following page. Open your browsers dev tools, locate your local storage, and find the myABTest key. Go ahead and change this value to control or experiment then reload the page.

Phew. That was a lot to cover! I hope it was all clear. If not, feel free to reach out to me via email or Twitter and ask any questions you might have. I should note that this implementation is a bit catered to WordPress in that we don’t have to worry about a mechanism for associating a given visitor with the stored test name, group assignment pair. This is something I haven’t thought through for non-WordPress environments. But overall, I hope this has been even a bit helpful for any of your React A/B testing needs.

Thanks for reading! ❤️