You know how to audit your data, but how do you measure your org’s ability to act on the data?
By Jennifer Kunz of 33 Sticks
Alright, great. Thank you Brian, and welcome everyone. I hope you’re enjoying the Analytics Summit today, getting some good information out of it.
I’m assuming that anyone on this call is going to be familiar with ObservePoint and what a great tool it is to help measure and maintain the health and quality of your data. But today I want to talk about something that involves that but goes a lot further, which is, how do you know if you’re getting enough value out of your data? How do you measure and maintain the value of your analytics practice and of your organization's data usage?
One of the big concepts that’s going to up with this over and over again is, is your analytics practice proactive or reactive? Are you kind of subject to the whims of your company of of the industry, trying to just stay afloat as things move along? Or are you helping drive your analytics practice’s fate and the direction that your data is going in that your organization can really get value out of the data?
So, I’m going to use a couple of symbols to represent being proactive versus reactive. Here we have a symbol for reactive. We’ve got Harry who really is almost more of a plot device. He has a lot of things happen to him, and he’s just trying to do whatever he can to stay afloat and move on. Versus Hermione who is much more proactive, who is a problem solver and responsive, taking in all sorts of information all the time so that she can respond to whatever is thrown her way. If you disagree with the way I’m defining these character traits, you can send me something via muggle mail.
So, as far as being proactive goes, one of the big reasons this has come up, is there’s been a lot of changes in our industry as far as compliance goes. Particularly with GDPR, the general data protection regulation, the EU privacy laws that went into effect May 25. A lot of companies found that they weren’t able to respond to this quite in time or are still trying to figure out who’s going to answer questions about data retention and IP addresses. GDPR requires that clients and users of your site or apps be able to submit a request to you to say, “i want to know what data you have on me. Maybe what PII you have on me, or I want you to delete whatever data you have on me.” So, companies need to have in place something for clients to communicate those requests. They need to have a process of what to do with those requests, and they need to know who’s going to be owning that going forward.
Unfortunately, especially in the US, a lot of companies were caught off guard and weren’t about to be fully compliant in time. So, if we’re looking at the US which is the yellow bars here. This is from a survey done by spiceworks which is a worldwide community of IT professionals, and it found that 25% of effective US companies, that’s represented in yellow, were compliant on or before the May 25 deadline. That’s the two columns on the left.
But almost half of those surveyed didn’t even know if they were compliant. That’s that big yellow column on the right, and in between, we’ve got 20-27% of folks who know that it’s going to be months before they’re compliant possibly up to a full year before they’re compliant with GDPR.
When asked “why?” there weren’t more companies that were compliant and ready to comply with those regulations, the biggest reason for the US, at least, again looking at those yellow bars, was just that it wasn’t a priority for the organization. Then this is followed by 35% people who said they just didn’t have time or resources. Now this can’t be too surprising because we hear something similar elsewhere in the industry.
This is taken from a great white paper from ObservePoint. The 2018 Digital Analytics and Data Governance Report. You can follow the link there if you want or find it on the ObservePoint site. The biggest roadblocks keeping folks from GDPR compliance are the same things that we see keeping folks from having the data management and governance that they want which is, number one, not enough resources. And number two, not enough time.
Usually, the things that we here as far as what’s keeping us from getting value we want out of the data or being proactive is data quality. You can’t get a lot of value out of data if you don’t trust it or it’s an incomplete data set. Technology and tool limitations. If I could only stitch visits across devices or I could have more data sets in one place, or maybe it’s resources and headcount problem. And that’s legitimate. It’s really hard to find experienced resources in our industry. But I venture to say that those things are only part of the problem.
Even if you had perfect, beautiful, crystal clear data, you still might not be able to get full value out of it. If you don’t have analysts, if you don’t have opportunities to communicate with your org and make suggestions and all of that, then a perfect data set is not going to do you any good.
But also, thinking about something like GDPR, it doesn’t matter how beautiful your data set is, that’s not going to prep you for something like GDPR or changes in industry policies and best practices. So, even if you have a perfect data set, you may not be getting a lot of value out of it.
And of course, that goes the other way too. If you have a perfect organization that’s fully primed to get value out of the data, but your data stinks, you’re not going to get that far either.
We jokingly call these, unicorns. Everybody’s looking for unicorns, and they don’t actually exist. If you happen to have someone who fits this description, then they are a rare and precious thing, and you need to hold onto them. But even if all of these people existed, I wonder how often we’re maybe looking for the wrong skill sets.
Frequently what we need instead is somebody who can adapt to curious and motivated and learns quickly. Asks the right questions, and pulls whatever resources they have available to them. Someone who can communicate effectively from developer up to c-suite analyst marketers. And someone who really values and can follow processes and force some consistency. Of course, it’s a huge prop if they have analytics experience, but somebody on the right here is going to be able to problem solve and be proactive. Whereas somebody on the left, that’s a reactive job description. That’s saying, “What have they done in the past?” so we can just check off boxes. Whereas on the right, this is a proactive job description of, “This is someone who’s going to be able to help define the way that your practice is going moving onward and evolve and roll with the punches.
Let’s face it. If you had had an open job description a year ago, you wouldn’t have thought to put GDPR compliance in there. That wasn’t really a thing people were talking about. Nobody had it on their resumes yet. So, if you had instead just said, we need somebody who is going to be able to roll with the punches, who adapts and learns, and is going to be forward thinking enough to spot this GDPR thing coming, then you’d be in a much better position when a change like GDPR rolls around. Skills can be learned, but that proactive mindset cannot.
So, all of these things combined, are making it so that there’s a data value gap between what we are investing into our analytics tools (money, time, effort, tools and resources). Hopefully, you feel like your company is getting at least that much back out of your investment. To be honest, I think that sometimes if we move out of the abstract, and we’re actually looking at the bottom line: Is it changing? How much our company is making? We might feel a little less confident in, “Yes, we are indeed getting a good return on our investment into these analytics tools.”
But even if we feel like we are, there’s always so much more that we could be doing. There’s a huge gap in the amount of value people are getting out of their tool and what they could be getting out of it if they had the right mindset and organizational structure in place. Ya know, I’m not bringing this up because I feel like this problem is caused by that resource gap or that data quality gap, but rather because I think this might be the cause of those problems. The data value gap is a reason why we have resource problems. At bare minimum, it’s a symptom of the same problem which is analytics maturity.
You may have seen a graphic like this at some point in the past that shows the relationship between the value you get out of your data and the maturity, the things that you’re using your data for. At the very bottom here we have things like dashboards, this is a low value activity. That weekly/monthly standard reporting. Looking at year over year stuff. Really just saying, “This is what happened.” Usually this is more to check off boxes. Say you did your analytics thing. Make sure that nothing broke, and everybody can continue as normal.
One step up on the value chain from that is having analysts actually be able to dive into the data and do some ad-hoc, seeking out of insights and all of that. Where they can ask questions and hopefully turn around and give some of that insights back, say, into project management or marketing so that things can be changed and evolved a little bit going forward.
One step up from that would be self service business intelligence where you have not just a few people in the data, but the whole org feels comfortable, can access the data, knows how to get answers to the questions that they have and apply those answers to the side or your digital properties to get more value out of them.
Above that, we have optimization and personalization where you are using the data to get value, to do the work for you. To actively improve things. This is where we generally really see that bottom line. ROI on analytics go up.
And finally, is predictive analytics. Where your data is not only telling you a story about the past, it’s helping you to know which questions to ask, which areas to focus on, and really helping you think into the future.
So, if you notice this, going from low value to high value, there’s a trend again of low value things are reactive. They’re looking backwards. This is Harry who barely does his homework, to be fair he’s got a lot going on usually, but he’s just trying to stay afloat. Versus Hermione is learning things she doesn’t even have to learn for school just so she can be prepared for whatever comes her way. She can be really proactive and drive her own destiny.
So, if I don’t think that the real problem is data quality or that these are the primary problems of headcount and technology and tool limitations, now those are problems.
But I’m going to move those to the side and talk about what I feel are real gaps. Match processes, roles and ownership, and ecosystem. Now, of course, data quality, and resources, and technology, those all fit within there. But there’s so much more to it as well.
So, if we’re looking at ecosystem, there’s leadership and executive buy-in. How important it is to have leadership that’s able to communicate to the entire organization the importance of data and the priority that data and getting value out of it should have. And communication is another really big problem in the industry of analytics teams everywhere frequently working in silos rather than being involved in conversations with product management or with marketing helping make decisions rather than being dictated to.
I call all of these three things governance. I know that that’s a bit of a buzzword, but hang in there with me.
So, when we’re talking about governance, frequently people are talking about aligning your variable maps. Making sure that custom dimension 4 is used the same everywhere, or eVar 36 is turned on and named the same thing in all of your report suites. And that is important.
Or sometimes with governance, people are thinking about data health and monitoring. Maybe using a great tool like ObservePoint, but either way, they’re thinking about it in terms of data quality or maybe it’s about ticketing systems and working with developers and QA teams.
Once again, those are all things that we do need to do and they are important, but they kind of only fall under that processes side of our triangle here. We’re still not talking about ecosystems and roles and the other things that are going to enable us to be proactive with our data.
So, as an example to show why it’s important to really consider all three of these parts of governance, I’m going to walk through an example scenario.
Let’s say a company has decided to start tracking a new type of user ID, something that’s based on email address so that we can tie CRM data into our analytics tools. Maybe some of you guys are already doing this or it’s aspirational, but it’s a common enough goal.
So, let’s start at the beginning and talk about what things we’d like to see happen as for the process of setting this up. We might have somebody ask, “Is this new implementation effort going to provide good, valuable, data?” Hopefully this is something we ask for any new implementation effort. Is this really even going to be worth our time of implementing? Then, do we have somebody who can consider how this might affect our international properties or our mobile apps? Is it going to go across multiple report suites or properties? And for bonus points, maybe somebody can even think about how this might be used to tie visits and visitors across devices. Like if you’re in Google Analytics, you can use users. If you’re in Adobe Analytics, you might start thinking about how you can use custom attributes and pull things in. And hopefully, someday, we will get that cross device visitor stitching.
So, that’s not all. Then you have somebody who needs to think about compliance. Does this fall in line with our company's privacy policies? Who’s going to set it up appropriately within the Adobe Analytics admin interface? Get things labeled for GDPR. Somebody needs to decide on a format that respects user privacy but works with the CRM, something that the developers can actually put into the data layer.
Then we need someone to decide, “Well where is it going in the data layer and how do we communicate that to the developers? Is it a tech spec? Is it a jira ticket? How do we talk to product managers about getting that into an upcoming sprint and communicating about publishing timelines?” Making sure that leadership backs up the analytics on the priority of this and it doesn’t keep getting bumped down. Then we need somebody to map that data layer object into the appropriate analytics variables and whatever tag management system you’re using or deployment method you’re using.
So, you’re starting to see, there’s a lot involved here. You also need somebody to set aside a new variable in your variable map and document whatever the expected values are. And decide what attributes to bring from the CRM over and set up that CRM integration.
And it just keeps going. You need to validate it. Get a test account because you can’t do validation if you don’t have a test account. Make sure that changes in the data and the TMS don’t negatively impact the user experience. Make sure it’s showing up in your beacons and variables properly, maybe using ObservePoint. Make sure that the integration shows up in reports properly.
And finally, hopefully, someone at the end of all of this is going to set up some reports or dashboards, something that provides some insight and can help those evangelise that, “Hey we have this new integration that has some really valuable data in it. We put a lot of work into it, now let’s get some data out of it.
So, in the end we have a lot of questions that people, some of these things are kind of a given and some of then you have to be thinking proactively about so that the questions get asked and the answers are found and the company is on board with what we’re going.
So, let’s talk a little bit about those processes. This is an example. I’m frequently asked to help clients come up with the right processes and forces and enable everybody involved when it comes to implementation.
So, if you have any reporting requisitions, what kind of process do we go through from beginning to end? I could spend a whole half hour on this slide, but we’re not going to go too in depth here. More than anything, I just wanted to show this as an example. A, it is complicated, there are a lot of moving pieces. It’s very easy to let things slip through the cracks. So if you don't have documented processes or even just everybody on the same page about processes, it might be easy to forget a few of these items.
Particularly at the bottom here, again, it’s important that after you’ve done an implementation change that you include in your process something about looking at the reports, seeing what insights you can gather, and making sure that you’re able to propose some actions or changes based on those results.
So, as far as a health check on your company and your analytics practice and how you’re doing on processes. Think about the type of tasks that your team is doing regularly and the processes you’re following.
Something like a new reporting request like I just showed on the previous slide. That’s one of the bigger ones that comes to my mind. Maybe a new analysis request. What do you do if the marketing team comes to you and say, “Hey, we had a big marketing push last week, and it didn’t go the way we thought it would and we would love some analysts help to spot where thing could’ve gone differently or why we didn’t meet expectations.”
New analytics users, that’s often an easier one. It’s a small thing to set up. Hopefully you also have a process in place to regularly clean out old users of folks who have left the company or were with vendors or anything like that.
GDPR is a new thing. That’s something that a lot of companies are still figuring out a process for. Things like turning on new variables, handling a new code release, especially if analytics wasn’t a part of the code release. Frequently the analytics team may not even know that they just published some change to mobile web and it broke something in analytics because we were relying on something in the DOM in our tag management system. Making sure that there are processes in place so those situations don’t happen.
How do you handle new marketing pixels? How do you handle it if ObservePoint says that something has failed in your audit or you get an anomaly detected or you get some sort of alert from your analytics tool. And finally, hopefully there's a process in place for insight creation to make sure you’re not forgetting to go in and get value out of your data.
So the big question with each of these tasks is, if you left your company tomorrow or if your whole analytics team died in a freak quidditch accident, would these tasks continue to get done? Would people know even what it is that your team is doing, and would they know how to go on and continue doing it?
So, for a big of homework here, I challenge you to create or visit your analytics practice’s wiki or Sharepoint. Hopefully you have one. If not, you really should. Even if it’s just a google drive and a few google docs. But have a centrally located place to keep solution documentation like a SDR or your variable measurement plan. Any tech specks that you have. Any processes you’ve already documented. And then make sure that people know that it’s there and are using it. Don’t let people continue using an SDR off of their local drive, basically.
Then, choose three common processes that you currently don’t have documented and start documenting them or even start informally thinking about what those processes involve or what you’d like to see them involve.
For instance, since GDPR is so new out there, that might be a good starting point of documenting and making sure that everybody knows this is what we do and how we’re handling GDPR.
Next, let’s talk about roles and ownership which I really want to belabor the point that is different and distinct from merely having enough resources and headcount. We don’t need staff augmentation necessarily here, but it’s about ownership and making sure that everybody knows what they’re expected to do and they feel enabled and empowered to do that and make proactive decisions that might not be clearly specified in their job description.
These are the rolls that we’d often like to see an organization have covered. Just because there are nine bullet points here, that doesn’t mean you need nine people on your team. It could be four people splitting these rolls, it could be twenty five people all doubling up on things.
The point is, usually we need to see an analytics lead or manager. Someone who has a vision for where the data is going and what we should be focusing on. And who’s able to make sure that every team member has what they need to make that vision happen.
We need a project manager. Someone to keep focus and make sure that priority is enforced and we don’t get swept away by the winds of the rest of the company. We need analysts of course. Maybe a solution owner, somebody who is keeping track of what are the types of values that we get in our reports and which variables are we using. And making sure if somebody has a new variable request it doesn’t actually double up on something somebody else is already tracking. That sort of thing.
And then implementation architect who then takes that solution and figures out how it’s going to apply to the actual site. And how to implement it using data layers and tech management systems. Then you might have a data steward. This is someone who’s working directly with developers and QA teams to actually execute on that architected solution. You’d have somebody in the report and the tools, administering it making sure that things like components of segments, captivated metrics, workspaces, all of that are curated and everybody, when they’re talking about bounce rate, they’re talking about the same thing.
Optimization is hopefully something that everybody has. At least somebody’s thinking about. And then finally, a new role of regulation and compliance.
So, with each of these, how many do you feel like your org has covered? And more than anything, does each team member have clear ownership and support in each of these roles? Do they know that this is part of their job description? And does everybody else know too so you know who to go to if you have questions about, say, compliance?
So, homework, on this roles and ownership portion. I’d like you to list the tasks that your analytics team has had to do in the last month. Typically, we might look at the last month and say, “Well that was exceptional because of XYZ.” Well, every month is an exception, so let’s be realistic about it and figure out what is it that we’re actually being asked to do day to day and spending our time on?
Then, I’d like you to think about what tasks you wish you had done maybe in the last month or that you hope to be able to accomplish in the next month, things that you don’t want to let slip through the cracks anymore. For each of these, think about, is there clear ownership? Whose roll would this fall into and do those people feel empowered to do these tasks and do them in a proactive forward thinking way?
And for a little bit of bonus points, if you have any open positions because it does feel like everybody is looking for more tasks, consider how well the title and description of that position matches the task that you are actually doing and hoping to do in the month to come and make sure that it matches where your organization is in its maturity.
Last, I want to talk about ecosystems. So, we’ve already talked about the concept of your analytics teams. Sorry, this is going to be a little bit small, but you’ll see why here shortly. So let’s say we have five folks on our analytics team.
Next, let’s talk about their responsibilities, and we’re just talking about the task and processes that we use. This might be things like product management, configuring implementation, troubleshooting data problems, gather reporting requirements, working on compliance making sure you’re fitting in with GDPR, whatever it is you need to be focusing on. Curating those components in your analytics tools. Making sure that users know how to access the data and understand what they’re seeing. Making sure that everybody has access who needs it or those who don’t need it don’t have access. Evangelising. Making sure folks see that there is value in analytics and showing off some of the insights that have been gathered. And of course, working with data layers and QA and development in all of that.
Alright, so components. It just keeps getting broader from here. There is a lot involved in being a data driven org. So this involves third party tags, tag management systems, processing rules, administering your analytics interface, technical specifications, whether that’s jira tickets or word documents or excel sheets, whatever it is. Maintaining your variable map, working with your data management platform, your customer relationship management, marketing, social. Hopefully, you’re doing personalization or at least thinking about it. Hopefully you’re doing optimization or at least thinking about it. You’re working with desktop developers. You’re working with mobile apps and mobile web. There’s a lot going on in there.
Last, and perhaps most importantly, what I ultimately want to get at is for the teams outside of your analytics practice who are or should be involved and impacted with data. So that might be technical or business consultants. You might have data scientist. You’d be surprised at how often I have clients that have data scientists who are actually not involved with web analytics. They’re using different data sets. Marketers and analysts generally do a pretty good job of getting involved with digital analytics. Product management often does not. I think that that’s one of the biggest missed opportunities in our industry. That we should be talking to product management. They should be involved in conversations with us. We should know and be involved in conversations with them.
And finally, of course, there’s developers and the QA team. With them, we often need to work with their management to make sure that analytics is prioritized and that everybody understands the value of it. That comes down to leadership. It might a CIO, a CMO, whoever it is that an executive higher up who can look at this whole ecosystem and see how it impacts each other and help communicate back downwards the value of data and what a priority is and how all the different pieces need to play together. As you can see, it is pretty complicated.
As far as health check goes. Think about who is using the data, who maybe could or should be using the data. Who’s interacting with your analytics practice regularly and do those teams, the people that you’re working with, have support from leadership to make analytics a priority? All of this is so important. That leadership support. To make sure that your analytics team is driving the data rather than just being pulled along and trying to check off boxes to stay afloat as they go.
Last, as a bit of homework for the ecosystem, I’d love you to document, it doesn’t have to look like this, it doesn’t have to be circles and icons and all of that. But earlier homework assignments, you’ve already thought about roles and responsibilities. Also start to think about the components, all of the different pieces that are involved, which teams you’re working with, and what leadership support you have. Then after you’ve documented this, take this to your leadership and sit down and discuss. Make sure that they see, in a really concrete way, the way that data impacts the entire organization and how they need to support and communicate that vision downwards and give analytics a priority.
Because really, the whole data ecosystem branches so far beyond the analytics team.
So, that was a lot of information, but to summarize, GDPR, that’s kind of the thing of the moment. It’s really highlighted this overall gap that we have in our maturity in a lot of different companies where it’s preventing us from responding to things like GDPR, but also preventing us from getting all of the value that we should be getting out of our analytics data. It’s not just a matter of more resources or different tools. You have to be proactive and not let the industry or your company decide what’s going to happen and how much value is going to come out of your data.
It’s important to have the right roles and processes and ecosystems to move your organization up in that Analytics Maturity curve and get more value out of your data. And last, you need to work with leadership and really have them sponsor and prioritize changes. Whether that’s role changes or process changes, to get more communication in your ecosystem to become more data drive.
I hope this was helpful. You can always contact me on twitter and everything like that. And now, I think, I’m going to pass it back to Brian, see if maybe we have any questions we can answer.