Roopa Carpenter, Blast Analytics & Marketing - Show Me the Money: How Page Speed Impacts Your Bottom Line

November 6, 2017

Show Me the Money: How Page Speed Impacts Your Bottom Line

Slide 1:

Thank you so much. It’s a pleasure to be here today to talk to you about how we can show you and your team the money. And specifically looking at how we can test pages speed and how it impacts KPIs that matter to you.

Slide 2:

First, what we’re going to do is talk about what we know generally about page speed, but also what we don’t know because that’s a critical aspect here. Then finally, how do we be in the know? How do we understand the impact?

Slide 3:

First, what do we know?

Slide 4:

I think we can all agree that we live in a world today where things must happen instantly. If you think about all the advancements in technology we have, all of the devices we have available at our fingertips, people expect things to happen and to happen now. What does that mean for our website? What does loading instantly mean? Well for right now, it means loading in three seconds or less. I say right now because if you think back just five years ago, the expectation was loading in five seconds or less. Go back even further, it was around eight seconds.

The bottom line is that things are picking up speed. They expect it to load quicker and quicker, and that’s how I see it going into the future as well. So, what does it mean then if our website is not meeting this expectation? It’s not loading instantly? what happens?

Slide 5:

Let’s think about the user experience. one thing to understand here is that user experience is not just about elements on a page, buttons, and color. There’s a human element. The human element is emotions. When you have slower page speeds, when you’re not meeting expectations, you’re going to create a lot of negative emotions. So, think about confusion. I’ve just gone through and filled out this really long form or put in my credit card information. I click that button and I’m just waiting for the confirmation page. It’s taking its sweet time and all of a sudden, self-doubt appears. Did I enter the information correctly? Did I miss something? That’s my credit card info. Am I going to have to do this all over again?

Or think about frustration. I think we all know someone in our life who does this where they’re repeatedly hitting that enter button as if somehow that’s magically going to get the page to load faster. The thing to understand here is the pain is real for your visitors.

That’s why I included this screenshot here. This is actual customer feedback we got from an audit we did for one of our client’s websites. The thing I want to point out here is that the question that we asked made zero references to page speed. It was just asking generally about the experience. But you can see the pain was so really that they felt the need to mention it. There’s a few responses I couldn’t even include on here because the language was so explicit. But think about that type of language being tied to your company, your brand. Think about the chances of getting those visitors as a repeat customer or them spreading the good word. Chances are, they’re not going to do that.

Slide 6:

Let’s dive in a little bit more in terms of impact to the business. There’s been a number of studies that have been done over the years about what’s the impact of page speed to your business performance. This is one here by Gomez and Akamai. What they found is that a one second delay in page speed can result in a seven percent loss in conversions. They even provide this example for an ecommerce site making $100,000 per day. A one second delay could potentially cost you 2.5 million dollars in sales every year. That’s a pretty striking statistic right there. It makes you wonder is all businesses are aware of this, why aren’t they running to take action to improve page speed?

Slide 7:

Well let’s look at what else we know. There are a number of online page speed tools. These are just a couple here, Pingdom and WebPageTest. They’re really good because they allow you to evaluate the current state of your site. You can get information like what’s the load time of my site or a specific page. You can test this from different locations, devices, and browsers. They’ll even grade your website. I laugh because the grade itself doesn’t matter, it’s merely there to be a guide to show you what’s working and what isn’t working. This is a good first step, understanding whether or not you’re meeting expectations to begin with.

Slide 8:

As I talk about page speed tools, the one thing I want to bring attention to because I can almost guarantee some of you are thinking, “Well, we use GA. I can easily reference that site speed report.” I want to strongly caution against using this report. The reason why is that it’s based on sample data and specifically by default, it’s only looking at one percent of your traffic. If we’re trying to get impact and understand impact and we want accurate information, sample data using only one percent of your traffic is not going to cut it.

If you’re interested in learning a little bit more about the issues with the site speed report, I’ve included a link to a blog post article in the resources box. Please check it out. It will tell you there are a couple workarounds you can do to increase simple size for this report, but you’re never going to get it to 100 percent. For that reason, I would definitely recommend using one of those other page speed tools.

Slide 9:

Another thing that these tools provide, which are really like our specific recommendations on how to improve page speed for your site. These are typical recommendations you’re going to see, including compressing images, leverage browser caching, minify code, and leverage a CDN. Now let’s step back here. We know generally we’re at risk of a seven percent loss on conversion, 2.5 million in sales for ecommerce, we have specific steps we can take to improve page speed.

Slide 10:

The question I want to ask you and your team is that, when you are thinking about the content changes, a massive redesign, or adding new assets, do you bring up the topic of page speed and its potential impact?

Slide 11:

What we find with our clients, the answer is often, “No, not really.”

Slide 12:

Why is that? Knowing what we know, why aren’t people taking action?

Slide 13:

It has to do with what we don’t know.

Slide 14:

So, let’s talk about that. What we do know, on one side, we have general insights like seven percent lost conversions, we have on the other side, specific recommended actions. What are we missing here?

Slide 15:

We’re missing the filling. It’s a specific set of data that will help complete the picture. Or complete the Oreo, in this case. But why do we need that filling?

Slide 16:

It has to do with those general insights. They’re just not enough. Knowing that generally people have seven percent lost conversions maybe 2.5 million in sales. If you were to try to present that information to your team, to the key decision-makers, I can pretty much promise you that they’re going to push back. Why? It’s because you’re not giving them the filing. They want to know the real impact to their site. Or if you can imagine them screaming, “Show me the money!” that’s what they need. Frankly, I don’t blame them. They’re completely reasonable in want to know what’s the real impact before deciding on whether or not to take action.

Slide 17:

The reason for that is because, really, how page load speed impacts the bottom line for you depends on the type of business you have, your site, and your visitors. So, let’s walk through a few examples here. Imagine I’m a visitor on an ecommerce site. I’m looking to buy a 20-dollar pair of socks. As I navigate through your site, I’m coming across slow page speeds left and right. I just want to get to the product detail page, I just want to see the different colors and styles they have available. Frankly, I’m just getting really frustrated by the slow pages. I’m not that invested. It’s only a 20-dollar pair of socks. I can find a comparable product on a different site. Frankly, I can stop by the store on the way home from work. So, if I’m not as invested in the socks, the slower page speed is likely to have a bigger impact on the KPIs for your company.

Now let’s think of a different ecommerce site. Let’s say I am looking for a gas range or a gas stove. I found this beautiful red one. Again, I’m coming across a slow page speed and it may be a little frustrating, but you know what, I want that red stove. It will complete my dream kitchen. And I know, because I’ve done so much research, that I can’t find this in many other places. So, you know what, I’m in it. I’m invested. Maybe in this situation, even though it’s a slower page speed, maybe it’s not going to have as big of a negative impact as it does for the 20 dollar socks.

But now let’s step away from ecommerce. Let’s say we have a different type or organization. Let’s say it’s one that’s seeking donations for disaster relief or maybe it’s a political group seeking donations to help your candidate for the next election. Here now, we may have more of an emotional connection. I believe if I make my donation, I’m going to have an impact. I’m going to help someone who needs my assistance because their house was destroyed. I’m going to help my candidate get elected in the next election. I think often times as websites, they realize that there is an emotional connection, so the messaging itself will be tugging at our heartstrings. Even in this type of situation, even if it’s a lower page speed, because they’re so emotionally invested, they might be willing to stick it out, so it may not have as big an impact.

This is why the decision-makers want to see what it means to them. They want to see the filling because we know page speed and the impact it has on the site will not be the same for everyone.

Slide 18:

Now that we know what we need—we need real impact—how do we be in the know?

Slide 19:

How do we get this information?

Slide 20:

Again, let’s take a step back. What do we have? We have those current page speed tools like Pingdom. They’re really good for evaluating your site currently. They’ll even provide that grade. What’s most useful is that they provide those specific recommendations on what to improve. You know once you convince decisions-makers, you know exactly what steps to take. But they don’t connect the dots. They don’t show the real impact. It’s not like it’s going to say this is you [age load speed and this is the revenue transactions or completions tied to it.

Slide 21:

What’s the solution? The solution to show you the money, is going to be with AB testing.

Slide 22:

The benefits of AB testing include the fact that you’re going to be able to measure real impact without blind implementation. You’re not going to go implement all the recommendations, cross your fingers, knock on wood, and hope that maybe you’re going to see a positive impact. How do you determine that? With AB testing, you’re going to have a controlled environment. You’re going to make those changes and a variation, you’re going to measure that against a baseline. That’s going to allow you to get quantifiable results. That’s the filling. You take that hard data, turn it around, and present that to the decision makers. That’s how you’re going to make a strong case for taking action.

Slide 23:

In terms of testing page speed, it’s not simple just doing and AB test. There’s one other part to it, and it’s basically utilizing this page speed script. This is something my teammates and I created. It came out of many of a need of our own clients. The clients are wanting to understand page speed and how it impacts their KPIs, whether it’s tied to an upcoming redesigned or just generally they want to know. This page speed script if available in the resources box. You’re actually going to see two different links. It’s going to depend on which testing platform you’re going to use. We have one for Optimizely X and we also have one for Optimize 360. So please pick the correct link because the instructions are ever so slightly different.

Once you do that, you’re going to go ahead and implement the page speed script and that’s going to allow your team to calculate the page load speed for your targeted pages. Then you’re going to create custom attributes or custom dimensions depending on your testing platform. That’s going to allow you to receive the page load data. Then finally, you’re going to launch your AB test. These are the general instructions. We’re going to spend the rest of the presentation diving head-on in here.

Slide 24:

We found that the best way to implement the page speed script is to do it through a tag management tool. For the purposes of this presentation, we’re going to talk about Google Tag Manager, but you can do it through Tealium. When you download the script, you’re going to get instructions and it will provide instructions on how to do it through Tealium as well. The first thing you want to do is create a new customer HTML tag. Name it something relevant like “Optimizely Page Load Script.” Then it’s simply a matter of copying the script and pasting it to the new tag. The important part here is how you set up the trigger. Make sure it fires on DOM-ready.

You’re also going to want to pick either one page to target or a set of pages. We find that it works best instead of targeting the entire site. You can definitely leverage your analytics background and identify the key value pages, maybe the cart page, maybe a set of product detail pages. Once you set up the trigger, just go through your usual QA process, preview the change, ensure the tag fires on those expected pages, and then publish the tag.

Slide 25:

The next step is to look at your testing platform. In this case for Optimizely, if you’re going to do this within Optimize 360, you’re going to be looking to create a custom dimension in GA. When you create the custom attribute, what you’re doing is allowing your test to receive the page load data from the script. So, within Optimizely X, you want to go to Audiences, select Attributes, then select Create New Attribute. Give this a relevant name, something like “Page Load Speed,” something you’ll be able to identify quickly.

The most important part of this step here is the third one, the API name that you include there must be typed as you see here: “pageLoadTimeRange”, capitalization the way you see it. That’s because it’s referencing the API name in the script. So, if there’s any discrepancies between the two, they’re not going to be able to communicate with each other. Then after you do that, just hit Save Attribute.

What that will do now is you will have these page speed buckets available for all your future tests on those targeted pages. You see a screenshot in the bottom. That’s Optimizely’s result dashboard, and you can actually go and segment by the different page speeds from one second all the way up to eight seconds and view performance for that particular bucket.

Slide 26:

And then finally, it’s just about creating your AB test. You want to have a test variation that’s going to wither load faster or slower than your control treatment. If you’re trying to decide which one to do, leverage those other page speed tools like Pingdom. Go evaluate what’s the current load time for your site. If you already have a quick site, let’s say it’s loading in one second or less, then you might want to try out a variation that loads slower. Now you might say, “Why in the world would we ever want to test something slower? It’s not like we would implement it.”

And that’s a fair point, but the thing is, you need to understand the purpose of this test. What we’re trying to do is understand real impact on our KPIs. While your site is loading really quickly now, that’s great, but websites are always evolving, right? You’re having content changes, maybe a massive redesign, addition of new assets like videos and images. Every time you do one of those changes, you’re at risk of slowing down your page speed, so if you have this baseline test, you’re going to understand whether or not you really need to pay attention to page speed if it’s going to impact your KPIs.

If you want to take this route and slow down a page, one of the easiest things to target on those pages are the images. Chances are, if your site is loading very quickly, you’re probably already using compressed images, so then what you can do is go within the visual editor of your testing platform and switch out the compressed version with the larger version of that image asset. If your site currently does not load very fast—let’s say it’s three second, four second, five seconds—then you might want to test a variation that’s loading fasters. Again here, images are a great thing to target. In this case you’ll probably want to use compressed images. There are tools online like ImageOptim, you can simply go upload your image file and it will give you the compressed version and then just go swap it out within the visual editor.

However, if you want to try out some of those other recommendations like leverage browser caching, minify code, this is where we recommend you create a second test page, implement those recommendations, and then utilize your testing platform to server redirect test.

Slide 27:

One important aspect of an AB test is to ensure you are tracking your primary KPIs. This is the way we are going to be able to connect the dots. The testing platforms make it relatively easy, especially if you’re tracking revenue. So, on Optimize 360, if you’re already tracking revenue in GA, you’re automatically going to be able to pull that information into the testing dashboard. For Optimizely, they do provide you with revenue tracking that you would need to implement on your site. The benefits of doing this is that once you run the test and you’re analyzing results, you can just easily look at the results dashboard, not only so see if there’s a performance lift or a performance decrease, but you’re going to get statistical significance calculated for you. That will make it a lot easier for you and you’re able to turn this report around a lot quicker.

However, we understand that in some cases, you cannot add that revenue tracking or you prefer to pull this information into GA and analyze revenue data there. That’s perfectly fine. I’ll say the only thing that might complicate things is when you’re trying to calculate statistical significance. The reason why is that there are a number of standard online statistical significance calculators. Those are great for calculating transactions, completion, button-clicks, it’s because the calculation that’s used to determine that requirement several functions being made. It’s fine for those metrics.

The problem with revenue is that it violates all those assumptions. You actually need a customized calculator. I’ve provided a link in the resources box for the Revenue Per Visitor Calculator. This is the one you want to use because it uses the appropriate calculation. It doesn’t need those assumptions. Nothing wrong if you want to look at the data in GA, you’re just going to want to use this calculator when you’re doing the test results.

After you have your goals being tracked properly, it’s simply a matter of going through your QA process, make sure things are working as you expected, and go ahead and launch the test.

Slide 28:

I will caution you, please do not try this shortcut—just speaking from experience here. We have had the scenario where is was said, “We don’t need to run an AB test. That’s going to take too long to build out that test page. We’ll just run a benchmark.” A benchmark is where you’re basically running a control experience. The reasoning behind this was, “We know if our results dashboard, we can segment the performance by the page speed buckets. What we’ll do is just look at conversion rate performance for the three-second bucket and then measure the drop off for conversion rate performance of the four-second bucket.”

That analysis will only work if your traffic is equally distributed across all the different page speed buckets as you see in the screenshot.

Slide 29:

Folks, this is the reality of it—trying to measure conversion rate performance for three-second bucket when you have close to 40,000 visitors and comparing that to maybe 11,000 or 12,000 visitors for your four-second bucket, you will essentially be comparing apples to oranges. If you do care about maintaining the rigor of your test results and your test analysis, you’re going to stay clear of this.

Slide 30:

If I’ve convinced you enough to stick with the AB test here, and you’re ready to go, maybe you have some questions about how long we need to run this test. I would say, generally when you do AB tests, you’re going to want to run it for two business cycles, which for most people that’s going to be two weeks. The reason why we say that is that we have seen—and we’ve seen it with our own clients—sometimes the behavior of visitors on the weekend will be much different than their behavior during the week. Even from the beginning of the week to the end of the week. So, when you run it for two weeks, you’re able to encapsulate all those different behaviors and kind of cancel that outside factor. You can run it for longer than two weeks, that’s perfectly fine, but I would say at minimum run it for two weeks.

The other aspect that you need is sufficient conversion volume. I know for sure, Optimizely will do it, it is fully possible to reach statistical significance with like 25 conversions. The problem is that you’re not going to have consistency. By that I mean if you just get a few conversions here or there, that’s going to completely skew your performance. You want to make sure there’s enough conversion volume for your original, for your variation, so you can feel confident in the results. While there is no magic number, I would say shoot for a minimum of 350 conversion for the control, 350 conversions for your variation. If you’ve done that two weeks, plus sufficient conversion volume, the test results you look at can be much more confident in determining whether there’s an impact.

Let’s say you’ve done that, you’ve run it for long enough, you have enough volume, you’re ready to analyze results. What do we do? The first step: go to your test results dashboard. This is going to show us whether or not the variation provided a positive lift, negatively impacted performance, maybe had no impact at all. Again, you’re going to have that statistical significance information there, so you’re going to see whether or not there was an impact. That’s the first step.

The second thing you can do—not required, but I think it’s pretty cool—is that since you can segment performance by the different page speed buckets, you can get a very nice visual breakdown of traffic across the different page speeds, for your original, for your variation. It’s just one other way to prove that your variation shifted the page speed. If you made it faster, you’ll see the traffic distribution shift to the left. If you made it slower, it will go to the right. Another thing this is useful for is spotting anomalies.

We expect that wherever the average page speed ends up, that’s where most of our traffic’s going to be. So, for the one on the bottom left, that screenshot, it’s roughly around a three and a half second page speed. The three to four seconds is going to have most of the visitors. Then as you go to the outliers, less and less. But what if for one reason, the eight second page speed bucket had like over 10,000 visitors. That’s going to stand out here when you do this visual break down. That could be due to maybe visitors coming from a certain device or certain browser. At the very least, this will provide some motivation to dig in further and to figure out what’s going on.

Slide 31:

But really, we want the filling, right? That’s the whole point of running this test. What you’re going to want to do is use this page speed test results template. When you download the script and instructions, you’re going to get this as well. It’s simply a matter of filling out the yellow fields. What you do is in your test results dashboard, go select the different page speed buckets—let’s say I select the one second buckets and I say, “Okay, these are the number of visitors that fell into this bucket. If I’m tracking conversions like these completions or transactions, include that information. Or if I’m tracking revenue, put that.” And you just do that for each page speed bucket. Do that for the original and the variation. It doesn’t take long at all.

The magic will happen when you look at the summary table. This is it. This is where the dots are connected. We’re going to have our average page speed calculators for us and we’re going to have it tied to our primary KPIs. So that first table is specifically for revenue, and the second table is tracking conversions in addition, or in place of, revenue you would want to look at that second table. Here it is. For example, in that first table, what we’re seeing is that the original had an average page speed of around three and a half seconds.

For the variation, we shaved off about two seconds worth, so it was loading in around one and a half seconds. Because of this, we’re able to see a positive impact on our revenue per visitor metric. In fact, it’s a 23 percent improvement. We know that this is statistically significant because we did that first step; we looked in the dashboard and we saw it passed our threshold. We even see a positive impact for transactions as well in that second table.

This is it. This is the filling. This is information you take—maybe with a little bit of background information about the tests that you did—pair that with the specific recommendations that you got from the page speed tools, and that’s how you make your case to the decision makers and get them to take action. It’s not just about that one-time thing, about improving page speed just one time. We want to leave an impression, so when you do have routine content changes, when you’re thinking about a redesign or adding new assets, they’re going to remember this test and say, “We need to stop and think about page speed. We know it can impact our KPIs. Let’s think about this a little bit more.”

Another thing to understand here is that this page speed script is not just one and done. This is the great first step to understanding impact to KPIs, but you’re going to have this available now for all your future test on those target pages. What it’s going to do is allow you to have yet another layer of analysis when you’re doing your test report.

Let me just give one final example. Let’s say you ran this baseline test on the product detail pages and you know there is an actual impact on KPIs. Months later, you decide you want to do a massive redesign of those pages. You run an AB test, you’re all excited because the page looks nice and modern, you’re positive it’s going to have a great impact on your revenue, and you run the test and all of a sudden, you see a negative impact. Then you’re left wondering what lead to that negative impact.

Now that you have the page speed script implemented, go into the test results, segment that data by the different page speed buckets. You could utilize the segmentation, the visual breakdown we talked about on the last slide so you can see whether or not there is an actual shift in page load speed. Use this test results template and see if the variation does in fact impact your page load speed. You might find it did. It may have increased page speed by one second or two seconds. You know what? That’s probably a big factor in why the variation did not perform well. Then you can go ahead and throw this into your test results, your test reposting. You’re going to impress your team that way. Again, it’s just to think about the many different ways that you’re going to be able to use the page speed script and this test results template.

Well that’s it for me everyone. I appreciate your time. I hope all of you go ahead and check out the script and AB test. I’d love to get your feedback on that.

Slide 32:

I’ll be available for any questions. Thank you so much.

Previous Video
Sun Sneed, ObservePoint - 7 Strategies for Mobile App Analytics Testing in 2018
Sun Sneed, ObservePoint - 7 Strategies for Mobile App Analytics Testing in 2018

Mobile app testing is still a bit like the wild west, reign in your strategy with these seven best practices.

Next Video
Peter Symuleski, NBC Universal - 10 Organizational Tips to Get the Most Out of Your Data
Peter Symuleski, NBC Universal - 10 Organizational Tips to Get the Most Out of Your Data

Tips focusing on creating a working environment that optimizes output across implementation and analysis.