Pat Hillery - You Know You Need to Document Your Analytics—Here’s the Shortcut

November 22, 2016

Slide 1:

I appreciate the opportunity to present at the ObservePoint summit, especially about a topic that I’ve spent a lot of time on and done a lot of work in. I’m going to jump right in.

Slide 2:

I’m going to walk through the ObservePoint data quality governance background, more specifically, why ObservePoint cares about digital documentation and governance. I’ll walk through some problems and some solutions, and then I’ll do a live demo of those solutions in the tool that we built. First, why does ObservePoint care about digital documentation? Third-party implementations start with documentation. Some of that documentation will be the foundation of governing that vendor moving forward throughout its life cycle. ObservePoint will QA third-party tag data once your implementation has been deployed into your web and mobile application. And with ObservePoint, the documentation can create tests to inform the expected results for each implementation. Really, we want to create a process where your documentation informs your development, your governance, and your testing in one integrated process.

Slide 3:

Here are some reasons why your documentation might be out of date. The most central reason that third-party documentation falls short—and we see that more often than not throughout our engagement— is really that there’s a lacking for automation. While developers and project managers have benefited from tools that do project management and agile process management, the analytics, and specifically, the third-party tag implementation process has remained unchanged for the 11 years or so that I’ve been in the industry. We still document and implement things the same way essentially today as we have in the past and we really haven’t benefited from having tools and automation.

There also isn’t a place where you can all centrally access and update the documentation. Often times the documentation lives either on a shared drive or on a computer. In either case, you need to save it and update it in several steps. There’s also not a lot of incentive to document. There’s really no short-term business impact. Nothing is going to happen negatively if you don’t document any of your solutions to your business in the next month or two months or probably even three or four months. And most organizations don’t have governance in place to mandate a certain level of documentation. So all these reasons combined create a situation where most of the time our documentation can be lacking.

Slide 4:

What is ObservePoint Labs? ObservePoint Labs is a fast, lightweight solution to inform enterprise product builds. Essentially what we want to do is we want to create solutions, and we want to get them out as fast as possible for feedback for improvement. Then we’ll take that feedback, and we’ll use and we’ll analyze how people are using a tool and use that feedback to build enterprise class products. That’s what the solution that I’ll be presenting today sits under. It sits under this ObservePoint Labs mantra.

Slide 5:

There’s a lot of different types of documentation that we’ve come across as we engage with clients and prospects. I’ve organized the types of documentation into two groups: one is at the global governance level, and the other is at the project level. Today, what I’ll be showing will be focused on the global governance level. It’s going to be around automating and documenting all of your third-party tags as well as automated, documenting a vendor at the variable detail level. Some of the other opportunities for documentation all surround more of a project basis once you have a standard implementation or a standard structure in place. It’s about that ongoing management and managing the communication with your business stakeholders as well with the development team if it resides outside of your team that’s managing that third-party.

Slide 6:

To that end, or to that mantra that we’ve outlined, we’ve decided that the first and easiest solution that we can build would be a free Google Sheet add-on. That’s where the solutions will be discussed and demoed today is through the Google Sheet public add-on marketplace. This solution is going to integrate with the Adobe Analytics API, the Google Analytics API, and the ObservePoint API in its first iteration.

Slide 7:

I’ll start walking through some of the problems and associated solutions that the tool can provide. I consulted for a company and there was a big disconnect between what was documented and what was enabled and what was implemented. There were variables that were enabled, but not documented. Variables that were documented, but were not enabled. And both, in some cases, were missing from their websites. At this point with this particular engagement, we had run out of variables that we could use for our solutions. And we were afraid to make changes to either vendor settings or to the documentation because we weren’t sure which one represented the truth and we weren’t sure if there was a project that had already been allocated to those variables that had been enabled in the vendor and had been documented. And we were really crippled, we couldn’t move forward and expand the solution to accommodate the growing needs for that company.

Slide 8:

Really, an ideal state, is we don’t have these three things operating in isolation. That really, there are connections in between documentation to the vendor settings through the analytics vendor API and through the documentation into what’s actually implemented on the site or app through the ObservePoint API.

Slide 9:

By doing this, instead of having three circles with minimal overlap, you know have three circles that have 100 percent overlap that represent the truth no matter which way you look at it. Whether you’re looking at what’s implemented on your website, your vendor, or your documentation, they’re all communicating the same set of truths.

Slide 10:

The way you can solve for this is by actually connecting your documentations to the analytics vendor’s API and actually pulling in all of the variable settings, whether or not they’re enabled, whether or not they’re in use, and therefore, immediately your documentation and your connection to the vendor are in sync.

Slide 11:

Next problem, how do I document where all third-party tags are implemented? We’re all very familiar with this really big list of third-party vendors that exist out there. And the list is only growing. There are more and more SaaS providers that are wanting to be deployed on your site, solve specific business needs. When we do audits, often times at ObservePoint, most people are not even sure what all the third-party tags are that they have implemented.

Slide 12:

And so to combat this and to try to have some governance around it, what we’ve done is we’ve created a template where you can specify each vendor that you have deployed, whether or not it has the status of approved, or trial, or banned, or out of contract. You can also document the scope, so where it should be implemented. And how you deployed it. You can also document the accounts, both for production and pre-production. In case you don’t want to go through, or you’re not able to go through manually and populate each of these rows with the right vendor, you can connect this document with your ObservePoint audit. And you can automatically pull back every vendor that ObservePoint has audited for a property. Thereby giving you a great starting place where you can have a really quick first crack at getting this governance in place.

Slide 13:

Once you’ve documented it, you can also check that status at any point in time. And that will integrate again with ObservePoint and give you a pass or fail to make sure that each of these vendors that you’ve marked as banned or removed had actually been banned or removed from your website. You can also create compliance rules for those vendors that should exist on your website. Those compliance rules can then give you a pass or fail again as to whether or not a vendor is implemented in the right scope on your website.

Slide 14:

Another problem that we run into is often times there are global report suites. There are a lot of different report suites that are all in a single solution and have the same variable allocation and the same variable usage. So how do we audit and automatically resolve the differences that we find across all of those different report suites?

Slide 15:

To solve for this solution, we created a piece of functionality where you can group multiple report suites into groups and then you can grab all of the variables from that solutions or from each of the report suites. It will highlight actual differences in the names, the expirations, the allocations, or all of the vendor settings across all of the report suits that you’ve added to your group.

Slide 16:

Next problem is: how do I automatically validate? I’ve got digital documentation in this case and it’s fairly accurate, but how do I connect it with my actual site and make sure that it’s been implemented correctly.

Slide 17:

So we’ve added additional columns to the template of our solution design reference. And these columns are designed for you to document not only the settings for that vendor, but more specifically, they’re designed to document the scope, where each variable should be populated. The format of that data, whether or not that data has a delimiter. It also allows you to specify a finite list of values and explicitly say, “Well this variable only has five potential unique values, so let’s create a rule where it must be one of these five values.” Once you’ve documented this, as mentioned before, we’ve integrated this piece of technology or document with the ObservePoint API, where you can create compliance rules for your online validation and make sure that this documentation stays intact and stays connected with what you actually had implemented on your website.

Slide 18:

So in case you need a good starting place and you don’t necessarily know or have a lot of context around how the variables or vendor have been implemented, we’ve also created a pullback: the top five values, and generated a regular expression button. What this will do, is it will pullback the top five values for each variable to give you a small snapshot of the data that’s been implemented and it will also check and make sure or see if there are only a few unique values that exist for that variable. If there are, will automatically create a regular expression and say, “Well this Evar should only have new or repeat. Or this value should only have a day of the week or a weekday versus weekend.” And etc. Again, regular expressions can be configured to sync with your ObservePoint tool.

Slide 19:

Another problem is: I use processing rules and I have a really hard time documenting those processing rules and they often influence the data that’s being collected on my website to the translation of what’s actually being reported. There’s kind of a disconnect and I need a lot of help so I can outline those processing rules.

Slide 20:

Another thing that we’ve done is we’ve automatically integrated importing processing rules. When you create the template for the first time, and you create a group of report suites, we’ll query, in this case specifically, the Adobe Analytics API and pullback all the processing rules that are associated with that group of report suites.

Slide 21:

With these types of features we’re again trying to create standardization in your organization. We’re also trying to create a place where you can both document, and you can create tests, and you can govern your implementations going forward. Again, we’re trying to standardize the vendor settings, what’s implemented on your website, and your documentation so there’s a single source of truth.

Slide 22:

Let’s jump into a demo and walk through some of this use cases as best we can.

Demo Screen 1:

So we start here with a black Google Analytics Sheet. In order to access the ObservePoint Labs add-on, you’ll simply click “Get add-ons” and do a search for ObservePoint. You’ll be able to integrate and add the tool from here. This is just a quick demo of what that would look like, that process. Once you’ve added it, underneath the add-on tab, you’ll be able to launch the ObservePoint add-on. The first time you run, you’ll have to accept. The first thing you’ll need to do is you’ll need to enter your ObservePoint and Adobe API credentials. Both of these are optional, there is some functionality available without both of these, but obviously, to use the full functionality, you’ll want to create and you’ll want to copy/paste the API credentials for both. Once you’ve done that, you click “Continue”. The first piece of functionality that this is going to do, is it’s going to create a configuration tab.

Demo Screen 2:

This config tab is going to govern all of the different places where we can document the scope, and how a piece of information is going to be implemented. It also, in this case, pulls back the Adobe Analytics report suites, both the report suite ID and the report suite name. In this column C, is where we can add a group if there are multiple report suites that should all exist in the same group, where there should be a common set of implementation variables and standards. For example, I’m going to create a group for the two ObservePoint Labs report suites. I can also, at this point, add any additional scope definitions, delimiters, and formats. We’ll come back to that in a second. Let me click inside of the digital SDR, in this case, the Adobe version of the digital SDR. The first thing that you’ll do is click to generate the template. Generating a template creates one tab for each variable so there’ll be one tab for Evars, one tab for Crops, one tab for Events, and one tab for your list bars.

Demo Screen 3:

And each of these tabs, when you just click “General Template”, is going to come back with a blank sheet. If you don’t have any API credentials, this functionality can give you a great starting place for documenting your solution. It’s also going to create a logs tab, which can keep track of any changes you make inside of this document to your analytics solutions. It usually takes about 30 seconds to a minute for us to generate each of those tabs.

Demo Screen 4:

So once you’ve done that, you can select which report suite or report suite group you want. In this case, I’m going to select the ObservePoint Labs value. That’s going to be the group of the two report suites. And when I click “Get Variables”, again, it’s going to go to each of those report suites and pull back all of the settings and then provide a difference between those two. That usually takes, again a minute or two, it can take several if you have 5 or 10 report suites in a group.

Demo Screen 5:

I’ll jump to the output, and what that looks like. It’s going to highlight, in red, the cells that contain settings that are different. In this case, Evar11 and Evar12 have differences. In this case, it’s going to have differences at the name, and it’ll actually show you both values that have been configured for this Evar11 name. So if I’d rather change this to Custom Evar11, I can simply select that from the dropdown. It’s also saying that the enabled and the expiration are different for each of these variables. What I can do then, is I can select which one I would like, whether or not I’d like them to be true or false, whether or not I want them to be enabled, and what the correct expiration is. I can then select Evars and then click “Update Settings to Adobe”, and then all of the report suites in the OP Labs group will have the settings that have been identified here in the spreadsheet.

Demo Screen 6:

If variables are completely missing or not enabled altogether, the entire row is highlighted. For example, in this case, Prop6 and Prop8 were enabled in one report suite, but were not enabled in the other report suite.

Demo Screen 7:

The same applies with the events that we’ve pulled back. Once you’ve created some standardization, and audited and made sure that all of your variables exist and have the same documentation and same settings across all of the report suites, now it’s time to move on to the definitions about where and how this variable should be populated.

Demo Screen 8:

These yellow columns represent, not so much the settings, but represent the how and where. Where have these variables been implements? Where should they be implemented? And then how, what are the approved values? Rather than populating these values manually, you can click “Generate Validation”, which will give you the friendly list of dropdowns that we’ve configured in our config tab. So if, for example, Evar1 has an intended scope of global, I can select global. And let’s just say Evar2 has a scope of homepage. This again identifies which pages these variables should be populated on. I can then also add custom descriptions around how that variable should be populated.

In this case, I will create a delimiter of a pipe and save the format of Evar1, in this case, should be alphabetic lowercase. For Evar2, we’re looking at the homepage, and let’s say there are only two approved values: Evar1 and Evar2, and let’s also document Evar3 and let’s just leave the other criteria blank. Let’s just say in this case, Evar3 is just too complex to have a specific format or delimiter, it really can just be any value possible. In that case, you would just leave these blank and we can still provide a first level of validation that the variable has been set. Once you’ve got this populated, you’ll then click “Sync OP Compliance Rules”, and that should take just a few seconds.

Demo Screen 9:

Let me switch over to the ObservePoint compliance rule set. In this case, we don’t have any compliance rules that have been identified. For those of you who aren’t familiar with ObservePoint compliance rules, compliance rules are automated rules that will validate whether or not a vendor or page is meeting a set of requirements. So that set of requirements can be: this Evar should be implemented on this page and this is how this Evar should be implemented, etc. Let’s refresh and see the rules that we’ve just created from our documentation. We have two rules: one called global and one called homepage. These match the scope definitions that we’ve provided from our documentation and let’s edit the global rule to see how it connects with the document.

Demo Screen 10:

In this case, we’re looking at Adobe Analytics. The first condition is Pages Matching. On which pages should this rule execute? In this case, we’ve selected global, we have a basic wildcard regular expression so that it’s saying every URL should match this regular expression.

Demo Screen 11:

Those definitions are created in and managed on the config tab. If we go back to the config tab on the right Google Sheet, we’ll see that the scope definition that we’ve created here for global is the same value that appears in the ObservePoint rule set.

Demo Screen 12:

Then we have Adobe Analytics. So we’re saying on every rule where Adobe Analytics is present, make sure that Adobe Analytics has Evar1 set to the regular expression of lowercase pipe, followed by lowercase. And then for Evar3, if you remember, we didn’t have any specific criteria around how it should be populated. So in this case, it’s just validating Evar3 is set. And now every time I run an audit inside ObservePoint, I can validate and make sure that these settings have been configured and these settings have been followed.

Demo Screen 13:

Let’s jump from the variable documentation into the tag-level documentation. In this case, we’re going to jump inside of the approved tags piece of functionality from the ObservePoint Labs main menu. That’s going to be under the governance section, kind of up at the top, so this approved section. Initially, when we created this for the first time, this document will be blank. Once we click the “Check Status & Get Tags”, it’s going to pre-populate this field with all of the vendors that have been found and that have been implemented on the selected property from ObservePoint.

Once I’ve done that, I can go through and create a status and scope definition. For example, for Adobe Analytics, I can say that this is banned and once I’ve selected that it’s banned, all the other information is irrelevant. And let’s say, for example, for Google Analytics, I’ve selected that this is approved. And let’s say the scope is global, so it should appear on every page. And let’s say I’ve used my tag managed system to deploy Google Analytics. And let’s say that I have a [sic] account number of this. Then, I have a preproduction account of, let’s say of this. And let’s also maybe say that I have banned DART Floodlight, and let’s add a new vendor, let’s say I have OneStat as well and let’s say that this one is also banned.

Once I have gone through and defined all of these criteria, I can then click “Check Status & Get Tags”. This will take a few seconds, what this is doing is it’s going through and checking and making sure that any banned values are actually banned. In this case, we know that Adobe Analytics and DART Floodlight were implemented on the site. So these two rules have failed, and it’ll provide us with the corresponding audit that will allow us to identify where that has failed. OneStat did not appear in our properties, so that vendor has actually passed. Again, this is providing you a centralized document that connects with the ObservePoint API, and will allow documentation to be validated immediately. This one central document can manage all of your vendor implementations.

Demo Screen 13:

In creating this approved compliance—so for any vendor that’s approved, it’s going to create compliance rules that you can then validate, much like we just watched on the example from the variable. As I mentioned before, all of this functionality exists with the Google Analytics vendor as well, in addition to Adobe Analytics.

Slide 23:

So again, you can pull in your custom dimensions, and your custom events. You can also document how you’re using the Ecommerce section of your Google Analytics implementation and connect that with ObservePoint to create business and compliance rules.

With that, I’m going to close this presentation. I appreciate your time and I look forward to any questions. Thanks.

Previous Video
Combining Multiple Data Sources to Tell the Whole Story
Combining Multiple Data Sources to Tell the Whole Story

Unpack how to leverage GDPR and the work required to help establish data as a priority in your organization.

Next Video
Chris Slovak - 5 Steps For Balancing Customer Experience with Data Governance
Chris Slovak - 5 Steps For Balancing Customer Experience with Data Governance

Learn to make sense of the current legal landscape, and ensure successful integration across your organizat...