Understand survivor bias to get the most out of your session replay tools

What’s in this article:

• Pros and cons of using GA4 vs. your session replay tool
• Practical examples of how to use insights from session replay to improve your website digital experience and boost conversion rates
• How to quickly filter session replays by users who dropped out of your funnel
  

If you’re on LinkedIn today it won’t take you long to find a post about how 2023 is going to be really tough economically, customers have less money to spend and budgets are being slashed. While nobody disputes this, it’s important to remember that as with your home budget, there’s a limit to how much you can cut back before it becomes impossible to survive. Ultimately the solution will always come back to growing your revenue. If you have a website and use it to sell stuff the solution is lifting your conversion rate.

The tricky part of that equation is understanding why some website users don’t make it all the way through your funnel. These funnel drop-offs represent real-world missed revenue opportunities. If their digital experience had been different then they probably would have converted and helped grow that pot. Most organisations use some form of session or website replay to provide an over-the-shoulder view of their user experience. This allows them to walk in the shoes of those individuals, empathise with their pain and figure out how best to solve digital experience friction.

In 2022, Travello identified issues with their payment processing. Using a combination of Session Replays, Audience Segmentation and Funnel Analysis, the Travello team were able to successfully answer why only some users dropped out of the funnel at that critical last step and not others. This allowed them to unlock $750,000 of monthly missed revenue opportunities in a short period of time.

Here are a few ways to use session replay;

• Find and identify points of friction in the digital buyer journey
• Quickly discover and reproduce customer support issues/bugs
• Optimise your new user onboarding workflow after initial signup (useful for organisations in banking, financial services and higher education)
• While rage clicks are “all the rage” (excuse the pun) mouse hovers can provide insights into which specific keywords or phrases catch users' attention

What these use cases all have in common is they require visual context. Getting an over-the-shoulder view provides a clear understanding in a way that quantitive data alone cannot provide.

GA4 vs. Session Replay

Consider the following example:

One of the pages has an average engagement time of 1 minute and 3 seconds. Using this quantitative data is this page engaging users and converting them faster through the funnel or confusing users and causing friction forcing them to bounce?

It’s impossible, the main reason being these data points are averaged across all sessions (both users who converted through your funnel and those who did not). Sure, if you have a trackable conversion point such as a thank-you/ page then you can use audience segmentation to compare the vector but it’s still an averaged data point.

As you can see in the above session replay, it’s much easier to empathise with an individual user to quickly identify (visually) all the little nuances which fly under the radar of GA4.

Examples of how your insights can go deeper by changing perspective include;

• Identifying where users hover their mouse prior to clicking (or not clicking)
• See why users might have registered a click - if they attempt to click a button did it work for them? If not, what was their next action?
• It’s great if they scroll 84% down the page but how fast did they scroll relative to the amount of text? Based on what you’ve watched, do you think they read everything, just the key parts or were they unlikely to have engaged with much?
  

This is all very useful if you happen to chance upon a relevant session replay, but what about if you need to figure out how much of a problem it is for all other users? That’s a lot of wasted time sifting through endless session replays trying to find a needle in the haystack.

With Insightech you can easily find similar users who experienced the same problem with a click of a button.

We can overlay our scrollmap data over the top of this individual session replay to see that 56% of pageviews reach the “read case study” CTA.

With one click can quite easily filter our session replays to look only at those individual user journeys who made it to that particular point on that page (the pixel depth has automatically calculated as 1228 pixels down from the top of the page).

From here we have a clear list of filtered session replays based on that segmentation criteria. This opens up a new world of possibilities for digital experience insights. For example, when doing funnel analysis we might want to look at users who converted to understand why were successful. This is demonstrated below.

Study the users who didn't make it through your funnel

So where does survivor bias become important? Let’s first be clear about what survivor bias is and how it relates to conversion rate optimisation.

“Survivorship bias (or survivor bias) is a cognitive fallacy in which, when looking at a given group, you focus only on examples of successful individuals (the “survivors”) in the selection process rather than the group as a whole (including the “non-survivors”).” (Masterclass)

In WW2 the US military was losing a large number of bombers from sorties over Germany. Mathematician Abraham Wald was tasked to investigate why and report back. This would be solved by adding armour platting but due to limited available resources at the time, they would have to be very selective about which parts would be reinforced to maximise the chance of the plane returning home.

“Initially, their plan had been to examine the planes returning from combat, see where they were hit the worst – the wings, around the tail gunner and down the centre of the body – and then reinforce those areas. But Wald realised they had fallen prey to survivorship bias, because their analysis was missing a valuable part of the picture: the planes that were hit but that hadn’t made it back. As a result, the military were planning to armour precisely the wrong parts of the planes. The bullet holes they were looking at actually indicated the areas a plane could be hit and keep flying – exactly the areas that didn't need reinforcing.” (BBC)

For conversion rate optimisation this story perfectly aligns with the challenge at hand. Session replays are powerful because they provide individual-level viewpoints of digital experience but you must focus this on the right areas of your funnel to really see the full benefit.

If our goal is lifting conversion rates then don’t look at the users who successfully made it through your funnel, study the ones who didn’t survive and do this for each step in your funnel.

What can you learn from the 137 replays in Step 2? Why didn’t they progress to the final step? There is likely more than one cause of friction with some users experiencing different friction based on their unique experience (eg. a key button might be hard to click on for Google Pixel 6 mobile users). If you found this to be the case for not only one user but 27 others then solving it would improve the Step 2 -> Step 3 conversion rate by 20%! This process of digging into the why behind some website audiences convert better than others is infinitely scalable with the right technology, whether your funnels are a simple 3-step process like the example above or more complex journeys involving multiple registration forms, pages and other touchpoints.

Want to schedule a live demonstration?

If this article has sparked your own curiosity to explore how you can apply these techniques toward lifting your own website conversion rates then our team will be happy to help you. Let’s book some time below.