My name is Krista Seiden. I’m the Analytics Advocate at Google. I’m really excited today to close off the ObservePoint Analytics Summit with my talk on best practices for testing, adapting, and personalizing.
So as mentioned, I’m the Analytics Advocate for Google. Prior to this role however, I actually spent many, many years as a practitioner of analytics and optimization at several companies; Google, The Apollo Group, and Adobe. And also a chapter co-chair for the San Francisco chapter of the DAA. As mentioned, you can find me on Twitter @kristaseiden, and I also blog at kristaseiden.com. If you do follow me on Twitter, you’ll actually know that besides tweeting on analytics and optimization, I quite enjoy tweeting about running.
A few weeks ago, I was in Sydney and I was running from Bondi to Kuji and I snapped this picture and tweeted that out.
My very favorite—the run out from my house to the Golden Gate Bridge in San Francisco.
Just a little fun fact about me, if you follow me on Twitter, you’re going to see some pictures as well. If we take a look at the agenda, I’m going to talk to you about two things. First, I was to talk to you about building a culture of optimization in your business. Then, I’m going to transition to using that to provide personalized experiences for your users.
Let’s go ahead and jump right in with building a culture of optimization. I want to start by level-setting everyone and giving a definition of what optimization is.
“Optimization is the ongoing, data driven process of continually discovering and quantifying the most effective experience, both online and offline, for your customers.”
Let me offer the question of; what is a test? A true test controls for everything except for test variables. You’re able to isolate exactly what it is that you’re changing. However, what I actually hear more often, especially in some of the marketing teams that I’ve worked with over the years, is something that is called “time series testing.” And this is when, let’s say, you push a new website live and two weeks later you say, “Oh look, it’s performing 20 percent better than my previous website. I tested it, and it’s doing well.” I hate to break it to you, but that’s not actually a test. You’re not controlling for variables such as seasonality or any of the other marketing campaigns you have out there, pushing more traffic to your website. So you’re not able to actually say that that new website is, in fact, better than the old one because you didn’t test them in the same time period.
Then when it comes to what to test, of course I would love to test everything. The analytics persona and the optimization person in me doesn’t want to launch anything unless I know for sure that it’s going to be better.
But, you have to be realistic, and we have limits. Many times, those limits are actually around our traffic. We’d all love to run hundreds of tests at a time, but we only have so much traffic, and so we have to decide which test is going to be more important at what time, in what market.
Let me give you an example: when I used to run the Google for Work testing program, we used to run tests on an English US website on about 5 percent of website traffic per variation. But if we wanted to run that same test in Brazil, Russia, India—really anywhere else in the world—we would need a lot more percentage of traffic because the overall traffic to that website would be lower. We would have to run those on 25 or 30 percent of our website traffic per variation. And not just that, but we’d have to run them for significantly longer. In the US, we would usually run tests for about two weeks. Internationally we tend to have to run tests anywhere from four to six weeks to be able to get significant results.
Then there’s the question of what is testing versus analysis? Which is appropriate really depend on the question that you’re asking. If you want to know which layout works better, that’s probably best answered with a test. But if you want to know what content people look at more, that’s better answered with analysis. But analytics can and should be used to justify, and propose, high-value tests.
One thing I like to tell my organizations is that we should collect ideas from everyone. Bring everyone along and let them all participate. I actually have a story about this. Back in my Google for Work testing days, we actually sourced most of our testing ideas from marketing and sales. About six months into our testing program, our engineering team came to us and say, “Hey, we would like to test the headline on your website. Right now, it says ‘Discover a Better Way of Working.’” A little background on that; the Google Apps for Business website was selling Google apps—Gmail, Docs, Drive—to people who were wanting to use that for their businesses. Marketing really loved this slogan of “Discover a Better Way of Working” because we thought it was a holistic message out to the market about this suite of tools that we were selling.
Engineering said, “We actually know that 80 percent of people who come to your website and actually sign up for a trial of Google Apps for Business only ever use Gmail. So let’s change the headline of the website from ‘Discover a Better Way of Working’ to ‘Get Email for Your Domain.’” Now if you’re a marketer like me, like many, you probably cringe a little bit when you hear that. It’s not very marketing friendly, “Get Email for Your Domain—” many people don’t even know what a domain is. Anyway, we decided to go ahead and test it because marketing was pretty sure that this would never win.
We ran the test, and about two weeks later—it was actually, I think, somewhere on the other side of the world because I was getting ready to go to bed—I started getting frantic pings from my VP of marketing saying, “Don’t you dare launch that test for that winner.” And I was like, “Uh-oh, what happened?” I quickly checked the results and this test had validated. It turns out that the “Get Email for Your Domain” headline actually blew the old one out of the water and was about 20 percent better in terms of getting people to sign up for a trial of Google Apps for Business, which was our main CTA. At the same time, I’m getting frantic pings from our VP of engineering saying, “You better launch that new headline. Otherwise, that testing platform that we built you—we’re going to pull all of our support for that.”
You can imagine me being in the middle between marketing and engineering and trying to make everybody happy and not wanting to lose support for my testing program. I was a little concerned, but after a good deal of back and forth over the next couple of hours, we came out with a plan. And that plan was that we would go ahead and launch that new headline because, first and foremost, we did not want to lose support for our testing program. At the same time, we would go ahead and launch a couple of follow-up tests that would try to get back to a little bit more of a marketing friendly message with that headline, while still targeting those users for what we know what they’re coming for, which in this case is email.
A couple of morals to that story, the one I’m telling you. One, as you see on this screen, I like to bring everyone along and get ideas from the whole organization, and this is a really good example of how a very powerful idea can come from areas outside of your normal realm that you’re testing within. I actually have this spreadsheet—you can see that short link there, I call it the test road map. It’s got a lot of tabs that I’ll talk about more through the rest of this talk—it’s a great place to have people input their ideas of what exactly they want to test and why, to collect those throughout the organization.
The other moral to that story is, when you do go ahead and launch a test, make sure that everyone you are working with has signed off on that idea. You should have, going into that test, made sure that everyone was okay with launching that new headline if it won. That’s not something that we did ahead of time, and unfortunately we had to scramble in the moment. The takeaway is to make sure we talk about those kinds of things ahead of time.
I also like to remind my organizations, when I’m asking them about ideas, to think about the low-hanging fruit. One of the very first tests that I ran in the Google Apps for Business group was a simple CTA test. We have one main call to action on the site, it is “Start Free Trial”—or it used to be “Start Free Trial”—it’s a big green button that sends people into a 30-day trial process for Google Apps for Business. I had some feedback that maybe the words “free” and “trial” had a little bit of a negative connotation from guerilla marketing tactics and what not. So we decided to test out a few new ones. As you can see, “Get Started” actually overwhelmingly won with 21 percent increase over “Start Free Trial” for people starting a free trial. It was a really big win and we went ahead and launched that across all of our sites.
Not only did we test this early on, but we actually ended up testing the same CTA against many others every time we made a big, significant change on our website, to ensure that it was still the right CTA for us, still helping us to capture as much of that user interest in trial as we could.
I also like to make it fun. When I’m asking everybody in my organization for ideas, I usually send out an email about once a quarter, near the end of the quarter to plan for the next. I list out the tests that we’ve run, the really big wins, big losses, some ideas, I link out to that testing road map, and then at the end I usually include a fun meme—my favorite being this one of a half-naked Ryan Gosling saying, “Hey girl, your call to action is definitely converting me.” To note, this email went out to about 250 people in my organization, including VPs on all sides and they loved it. Making it fun really encourages people to be more interested in what you’re doing.
We’ve already mentioned this one, make prioritization collaborative, with our earlier example of “Get Email for Your Domain.” Make sure that when you’ve collected these ideas and you’re prioritizing them for the quarter, that you have somebody from each area, a key stakeholder—to help you understand and to go through the pros and cons of each test design, the potential impact and relevance to the business trade-offs of running one test versus another—and prioritize those tests to actually run in the quarter. Make sure that you have sign-off for all those variations during this process.
Next, once we’ve decided on the road map for the quarter, I like to evangelize that process back to the business. In that same testing road map, I have another couple of tabs. One’s a gantt chart, and it shows visually when each test that’s been prioritized will run run in the quarter. There’s actually another tab that actually has all of the details about the individual tests that we’re planning to running in the coming quarter. It’s a great resource for anybody with questions about the testing program to go to at any time.
After I’ve collected ideas from everyone in the organization and gotten their input, I also like to make sure that they understand what a good test design is. There are a lot of kinds of tests, and I’m going to go over two: AB testing and multivariate testing. AB testing compares two or more different versions of the same website. In this case, we’re looking at a two column versus a three column layout.
A multivariate test is testing variations of multiple elements in one test. In this case, you can see we have a different color header bar versus numbers or letters down the sidebar.
This is a bad test design. Let me give you just a second to think about why that might be… if you were paying attention earlier, we talked about a good test being one that controlled for all of the variables. In this case, we’re actually looking at a different color headline bar, a different color button, and whether or not that button is actually on the page or not. This is a bad test design because if any one of these variation won, we wouldn’t know if it was because of the button, color, headline color, or lack of a button—we wouldn’t understand why it is that it won.
This might be a better test design. Or this one. Or this one.
Or one massive A/B/n multivariate test design that is accounting for each of these potential designs that we want to look at here.
Once you’ve designed a test, I like to document it. Every test that I run, I have a similar template to this one that I fill in all of this information for. It’s got a name, a launch date, how much traffic the test is running on, a summary of what it is that we’re testing, a hypothesis, and most importantly, the success metrics that we have agreed on as an organization that will make one version versus another a winner. I also like to include screen shots in this test document, so that people can come in here and, very quickly, visually understand what it is that we’re testing. I have the short link there on the screen to this test design document. I think it’s a really great resource or template that you can take and make your own to use for your testing programs as well.
One other thing I’d like to highlight is qualitative tests or surveys. I like to run surveys—specifically I run Google consumer surveys, but any survey program will work—with major website changes. For example, back in my Google for Work days, after about two years of optimizing, we had a pretty optimized website and we had used all of the learnings along the way to actually build a brand new website that we then wanted to test against the old one. We had a lot of criteria that would make it a winner or not, but when we launched this test, we also launched a GCS survey on each variation of the test to collect information about customer satisfaction. How satisfied were they with the website? What was their main reason for visiting? Were they able to find everything that they were looking for?
When our test results came back from that test, we actually saw that the new website was pretty flat in comparison to the old website when it came to our main KPI of trial sign-ups. However, when we looked at the qualitative survey data, we found that people were overwhelming more satisfied with the new website. They found it easier to navigate. They found it easier to find the information they were looking for. Overall, higher customer satisfaction. That actually helped us to be able to push that new variation out to market because we had that qualitative survey data as a differentiating factor. So it can be really, really powerful when you want to understand the overall impact of a new website redesign, not just the quantitative impact.
You’ve ran some tests, and now you’ve found the big winner. Now what do you do? I think the first step, most importantly, is to triple check your results. Are they statistically significant? Did you control for external variables? Remember that first CTA test I mentioned? We ran the CTA test, we found a really big winner with “Start Here”—23 percent better than the baseline. That was awesome.
I actually put together a whole long email with the test results and all this information about the test and mentioned in there that we were going to go ahead and launch these results to all of our websites in the next week. I sent this out to my whole marketing organization, again about 250 people including VPs from all the different areas of the business. I don’t know about at your companies, but at Google, when you send an email like this, you generally start to get a lot of reply all’s, and they’re congratulatory.
I’m sitting there watching these emails come in and it’s like, “Oh, great job!” “So impactful for the business!” “This is awesome!” And then, I get an email from my very favorite department, engineering, from a department manager in engineering who says, “Hmm. This is interesting. I actually see a lot of spammy trial data coming from that version of ‘Start Here’. Specifically coming from the Philippines. Did you control for spam data and data from outside the US?” The answer was, “No…I hadn’t. I hadn’t even thought of that.”
When we took all of that data out and re-ran the numbers, we actually found—if you were paying attention earlier—that we still had a really big winner with “Get Started”—21 percent better than our baseline—it just wasn’t the winner that I had already told everyone in my organization about. We actually reran the test as well, just to ensure. Then I had to send out another email to my organization saying, “Hey, we still have a really big winner from this test, it’s just not the first one that I told you about.” So learn from my mistakes and make sure that you triple check your results before you go ahead and send those out to your whole organization.
After you’ve triple checked those results, go ahead and put together a summary deck and send all that information to your organization and really evangelize it. Get it in front of all the relevant stakeholders and schedule those review meetings with them. Get everyone really excited about what you’re doing. When you’re done with all of that, throw a link back to that results deck into the testing road map so that it’s one source of truth from start to finish for everything about these tests.
To summarize this first part, on building a culture of optimization. First you want to teach your organizations how to test; what is a test, testing versus analysis, and good test design. You want to bring everyone along; marketers, sales, devs, engs, execs. Good information or good ideas can really come from any area of the organization. Then you also want to keep the momentum and the cadence of these tests. And triple check your results before you send them out to the organization.
Now I’d like to switch gears a little bit and go from just testing to personalization. I’m going to do this by talking about personalization in four ways. We’re going to start very wide at a geo location level and then go all the way down to the on-site action level.
First we’re going to talk about understanding the customer behavior. Then how you use that understanding to provide personal experiences to those customers.
So understating the customer behavior…
In an offline world, when somebody walks into a store, you’re able to pick up on their body language and understand a little bit about what they’re thinking or how they might be reacting to your store or how satisfied they are with what they’re looking for. But online, understanding customer in the same way is more difficult. We don’t have that body language.
We do have analytics, and analytics helps us to bring that customer data together to better understand it by slicing and dicing and segmenting and creating different audiences. We can look at people by their geo location or by the actions that they’re taking on our website or by their demographics or their interests. And were able to start to understand different groups of customers. Even so though, a lot of the time we’re still serving the same we experience to all the different customers even though we know they’re not the same. So actually at Google, we recently released a new product called Optimize 360. When we were doing a lot of research for this product, we asked people why this was; why are they still serving the same web experience to everyone when they know they’re not the same?
What we heard, overwhelmingly, was that it’s really difficult to get data from an analytics tool to a testing tool—and vice versa, from testing tool back to an analytics tool—to really take advantage of the information that we know about them and personalize that experience. That’s one of the things we really set out to build and provide when we were building Optimize 360, was a way to really deliver more personalized experiences to your customers.
Let’s look at that. I mentioned that I’m going to talk about this in four ways. Before I get there, a stat—because what’s an analytics conference without a stat? According to a Gartner study last year, they believe that 89 percent of companies are expected to compete mostly on the basis of customer experience this year. What I take away from that is, if you are not understanding and personalizing those experiences for customers, you are not going to win. Now let’s dive into those four levels of personalization. Many of these are tool agnostic, and then we’re going to talk a little bit at the end about how you can actually use Optimize 360 to do some of this.
The first level of personalization here is understanding the marketplace: personalization at a local level.
I mentioned to you before that I ran the Google Apps for Business testing program. Along the way, this is what our website looked like. This is pretty early on. You can see we still had that headline of “Discover a Better Way of Working.” This was the English, US website. I don’t know about a lot of your companies, but in marketing at Google, we generally have one template that we then translate and localize for every other region. Again, this is the US, English website.
And this is the Japanese website. Pretty similar, right? Actually, exactly the same. We got a lot of feedback from our Japanese team that Japanese customers do not use the web. They do not interact with our website, and they do not buy our products online in the same way that we do in the United States. If you’ve ever seen a Japanese website, you’re probably aware that they tend to be very busy—lots of different boxes and pieces of information, tons of widgets—very, very busy, crowded websites. For me, that might seem overwhelming, but to a Japanese customer, this it what they’re used to, it’s what they like, it’s how they like to buy products online. Our website, being very simple and just a translated version of our US website, wasn’t serving those customers in a way that they needed to be served.
I’m going to back up here to this English, US website and you can see at the top, we have two calls to action: “Contact Sales” and “Get Started”. “Get Started” as I mentioned earlier takes you into that trial sign-up flow. “Contact Sales” actually pops up a box that has a form to fill out to then contact an enterprise sales rep.
So the feedback that we got from Japan was that literally no one there is going to contact sales online like this. This is not how they do business. When we looked at the data we saw that in fact no one was clicking on this button on the Japanese website. Our team recommended that we replace this button with something different.
This is actually what we did. You can see that it’s roughly translated to “flow of available,” but what we called this was a “guided flow,” and it was a pop-up box that actually detailed each step of that 30-day trial process that a customer would have to go through. They’d have to set up their billing and their MX records, and a bunch of different things to be able to set-up Google Apps for their business all within that 30-day trial period. Typically, in Japan, a lot of this information is presented to a client in a nice packet, they have all this printed information. This guided flow online was a way to try to bring that experience online. When we ran this test, we actually found that there was a 693 percent, almost 700 percent, increase in clicks on that “Contact Sales”, now guided flow button. Not only that, but that actually lead to about an eight percent increase in customer actually starting the trial— process.
So very significant impact on the bottom line by trying to personalize and provide a more Japanese-like experience. Just because we didn’t learn anything about how we need to have localized content in this situation, we actually ran this test in the US and the UK as well to see if it would work there and it failed miserably because US customers, UK customers, the rest of the world, does not buy products in the same way that Japan does. It was a good learning for us to remember that this whole endeavor was to understand at a local level, how people buy, and to personalize that experience for them.
We actually took that a few steps further. What you see right now on the left hand side is a mock of that Japanese website trying to be a little bit more Japanese-esque. You see there are a lot of boxes, a lot more information below the fold on that page. We also mocked up a new canvas image, that was an image of the Tokyo skyline to try to be a little bit more local and more personal. Fast-forward to about a year later, we actually launched a completely different template, specific just to the Japanese market, that was very focused on being Japanese-esque. This site ended up performing much much better for this organization because we were able to step outside that normal template and realize that speaking to the customer at their level, at the location level, was really impactful.
The second type of personalization I want to talk about is understanding the referral source, so this is personalization at the traffic level.
A little later on, we had a new website for Google Apps for Business and we actually rebranded to Google Apps for Work. You can see that this is one of our product pages for that website. Each product—Gmail, Drive, Docs—have their own product pages, and we would send a lot of ad traffic to these individual product pages. We had a theory that if we actually used the same headline from ad to product page, we could increase the click through into trial of those customers. So we tested it.
Here is the baseline product page that just had Gmail as the headline here. Then for users coming from an ad that said, “Get Gmail for Work,” we actually customized this landing page to have the same headline of, “Get Gmail for Work.” When we compared it to users who didn’t come from an ad, or who came from an ad and went to the just “Gmail” one, there was a significant increase in users starting trial here. That was a very simple personalization that we did just by bringing that ad content to the website, so people felt like that message was following them.
The next type of personalization I want to talk about is understanding the customer with personalization at the action level.
Earlier this year, I had the opportunity to speak at the Wharton Customer Analytics Initiative conference. This was a big honor. It’s a great conference. If you ever have a chance to go or attend, I definitely recommend it. This is their event website, and I had gone to their website early on to register as a speaker and submitted my form there and had my registration sent to me in an email and all that. A few days before the conference, I come back to this website because I want to check out the agenda and I see this message, it says: “The event is sold out.”
That’s great, but they know me. They know that I’ve been to this website before. They know that I’ve submitted a form, I’ve signed up for this website, so they could use that information to actually personalize this experience when I come back. Perhaps something like, “We look forward to seeing you there, Krista.” Or, if they don’t want to personalize it with my name, they still know that I’ve submitted that form, that I’ve submitted my registration, and they could just say, “We’re looking forward to seeing you there.”
The last step of personalization I want to talk about is personalization in real-time.
This is the Serene Sierras, it’s a fictional outdoor gear website that sells a lot of outdoor gear.
Specifically—we know that users are coming to this website and they’re very interested in this Adirondack chair. We know that because we have analytics on the page and we see that users are coming to this chair often, but they’re not actually clicking that big green button to add to cart.
When they come back to the Serene Sierras homepage, they still see that same message: “Sit back and relax. Life’s busy.” I’m a big fan of subtle messaging, so actually what I’m going to do here is I’m going to add a sentence to where it says, “Life’s busy.”
We’re going to load this website up into the Optimize 360 editor, and I’m going to click in there and just type another sentence: “Life’s busy. Take a seat.” Then I’m going to go ahead and set up my objectives for this experiment.
The objective of this page should be pretty clear, it’s to click that big, green Add to Cart button. When I go over into my objective builder, I can see that I’ve selected my primary objective to be added to cart.
It says, “Goal 1 Completions.” This is really important because this is actually the integration of Optimize 360 with Google Analytics and pulling over the goals that you already have set up in analytics to be those experiment objectives for you in Optimize 360. I can add actually up to 10 objectives here. I’ve added a couple more: that they’ve actually placed the order and that they signed up for a newsletter.
Next is targeting.
So that first section where it says, “URL equals,” this is just the homepage of the Serene Sierras website. It’s the page that I want to test on. The next part though, GA audience lists, this is where the magic happens. I mentioned to you that I know these users are coming to the Serene Sierras website and then going to this Adirondack chair page several time, we’ll say three times, but they have not yet added to cart. So what I did, was I actually built a segment in Google Analytics that had those details, that they’d visited that page three time, and that they had not yet added to cart. I save that as an audience, and audience between Google Analytics and Optimize 360 are shared, so now I’ve selected that audience here in Optimize 360 and I’ve called it, “Chair Lovers,” so somebody who has come to this chair page at least three times is a chair lover.
Now as soon as somebody qualifies for this audience, meaning they’ve seen that chair page for the third time, but have not yet added to cart, they’re going to fall into this audience and be eligible for this test.
When they go back to the homepage, now they’re going to see this new message that, “Life’s busy. Take a seat.” I think that’s pretty cool. I’d actually like to take that one step further though.
So this is Lightinthebox. It’s a major online retailer. I’ve come to their website and I’ve been looking at digital cameras. I’ve done a lot of research, looked at several, I might have even added one to my cart, but I didn’t check out. I didn’t buy that digital camera.
In Analytics 360, there is actually a report called, “Custom Funnels.” It’s a really cool report. Try to learn more about it later on because it’s really powerful, but here I’m just going to show you a really quick example. I’ve can look at this website data and I can see a lot of people come to this website, and a lot of people even view this digital camera page. Some even add to cart, but virtually no one checks out. I can see this through the fallout. The coolest part of this report though, is actually that red arrow towards the bottom of each section. If I click on that arrow, it pops up a box that says, “Hey, I see that 2,346 people were at the add to cart page, but did not check out. They fell off. Would you like to create a new segment and remarket to these people?”
Well as any good marketer would, of course I would! I want to bring them back and I want them to buy that digital camera, but to bring them back I would probably need to give them some kind of an offer. I’m going to incentivize them, I’m going to make that ad headline say something like, “Get free shipping on your next order.” That’s an audience that is working with AdWords to target those people to come back through remarketing, but because it’s an audience, I can also share that with Optimize. I know something about these users, I know that they’ve been to my website before, they’ve researched digital cameras, they’ve even added one to their cart, but they haven’t checked out. Now they’re coming back through the remarketing ad, so they’re interested. Because I know all this information about them, I should use it to actually serve them a little bit more of a personalized experience when they do come back.
So I’m actually going to carry that ad message through from the ad to the website and I’m going to have a little box up there on the website when they come back that says, “Welcome back. Get free shipping on your next order.” I think that’s a really cool way to bring all of that back together from onsite analysis to remarketing to really providing that more personalized experience and helping to drive sales.
In summary of this part, you can see that not all of your customers are the same and I’ve actually now given you four different ways that you can consider personalizing that experience to deliver a different experience to each of those different customers.
Thank you very much. I hope you’ve enjoyed today at the ObservePoint Analytics Summit