Chris O'Neill, ObservePoint – Perfecting Release Validation: Catch Errors Before they Destroy Your Data

January 16, 2020 Chris O'Neill

Join Chris O'Neill, Solutions Architect at ObservePoint, as he discusses how you can:

  • Address data quality issues with less risk and cost,
  • Focus on implementing tag governance in early development environments
  • Instill a culture of proactive tag governance

Fill out the short form to view the presentation on-demand.

Chris: (00:02)

Hello. My name is Chris O'Neill. I'm a solutions architect here at ObservePoint. Today I'm going to speak about release cycle validation. We're going to talk about a maturity model inside of release cycle validation. How we want to go through that, how we want to think about that. I guess I want to start my presentation just with a very common story that most people know. And uh, the question is, or the story is, you know, how do you eat an elephant?

Chris: (00:40)

And if you're familiar with this, it's pretty well known, but the answer is one bite at a time. When we're thinking about implementing a quality assurance practice around your MarTech data, uh, it can be very overwhelming. We can have two schools of thought. One is I want to validate everything, uh, during all releases or at all times. Um, and the other one can be on the other end of the spectrum where you're not really sure, uh, even where to start or what to do. So we're going to talk about the release cycle validation and the maturity model inside of ObservePoint. I'm going to recommend some best practices, talk about some stories from clients and then we'll do a fair amount of talking around, uh, you know, where to go going forward. Once I've got my QA practice to a point where I'm getting value, I'm able to QA the data, our data quality is up. Um, people inside the organization are really starting to see this improvement and make more data-driven decisions. We'll talk a forward thinking some other things that we can do.

Chris: (01:57)

So first I want to just discuss real briefly the problem that we face and why we need to QA the data. Uh, there are two reasons. Uh, one is you implement the tracking technologies on your site today and just cite changes. And as your site is changing, uh, nobody notifies, the people in charge of implementation of these tracking technologies.

Chris: (02:25)

I'm not sure if it would be efficient for a dev team or all the different dev teams, depending on how many properties you manage, to notify you of all the changes and then for you to go back and look through your documentation and make sure that it doesn't affect any of your tracking. That could be a very tedious process. So this is a problem that we live with and it just may be by design. Uh, the other problem is more of an overall governance problem, um, and the analytics managers or running analytics teams or web analytics, you need to be aware of all, not only all the technologies that should be on your site and make sure that they're working properly, but the other side of that coin is you need to be aware of when people are putting other technologies on your site. Maybe multiple teams have access to the tag management system.

Chris: (03:20)

We see that a lot. That's pretty common to democratize that out to the organization. But you, you need a way of monitoring and alerting if somebody adds something or changes something. So those are the two problems we're trying to solve. And ObservePoint is an automated platform and it's designed to help you build out this QA practice, centrally focused around the release cycle validation and also these governance concerns. So ObservePoint takes a testing methodology which we leverage and is very similar to functional QA or any other sort of web QA process. And so we do testing just like they do. And this is the best way to bridge that communication gap between dev and the analytics team of understanding when an update to the site after a release or a sprint has affected, or could potentially affect, your tracking. In this test based approach or this testing methodology after every single release. So if you have a release that you know every other week or monthly or weekly, ObservePoint will run the suite of tests against the property that had the release and we'll send our browser to your site to either load all the page load events from every single URL, or we'll send it to go through conversion flows that a user would do or these main call to actions or flows that you're trying to do to track.

Chris: (05:09)

And so as we run these tests, ObservePoint's, crawling your sites, and we scrape all the network requests that fire from every interaction with your sites. So on one of these, uh, larger scale scans, we call them audits. This is where we will spider through the site, discover all the URLs, we will execute all the page load events. So after a release, we could run this scan on the entire property, make sure that all the proper pages are tagged appropriately, and they meet the minimum tagging standard for that property, or globally, or maybe specifically to sections of sites. As we pull back that data, we can then send you alerts, letting you know what didn't match our expectations. And those expectations are given by the person with domain knowledge. So if I'm an analyst and I'm in charge of property X, I have my minimum tagging standard that I'm going to send ObservePoint out to crawl my site and pull back all the tags that are loading on all the pages. So first you want to make sure that the tag presence is working and then we can work our way down. We can be more granular. We can look at the variable levels and we can look at the values that are being passed in as well and make sure that the parameters we set in our expectations around what those values should look like are being passed through to the network requests to the vendors. The other scan we do is what we call a journey and this is the flow based test. This is where we send our browser to go to your site, click on that main call to action and convert, whether that's a form fill or a checkout, whatever it may be. As we're going through each step, so every action we perform, so let's say we load the page of the landing page. We'll validate that that landing page was set up correctly. Then we'll click on the main call to action, for example. Maybe that takes us to a form and for each step when we click the button, we're going to make sure that all the proper network requests are firing. When we get to the form page, we're going to fill out the form and submit. Make sure on that conversion, that thank you, that submission page that all the proper tags, variables and values are firing correctly. So a good example of that would be possibly there are key events that are supposed to fire on that submission page. When we get to that submission page, we'll make sure that that event fires. And if not, we'll send you an alert.

Chris: (08:01)

So now that we have a baseline for understanding what the problem that ObservePoint's here to solve the methodology of how we solve it, let's talk about the maturity model and how we mature through this process or how we eat that elephant one bite at a time. Typically, uh, most people when they start out implementing tracking technologies, if they don't have a QA practice in place, you're going to be much more on the reactive side versus the strategic side. So that's going to force you to do a lot of ad hoc problem-solving. Maybe after a release has happened, you notice a dip in the data. Uh, when you run a report, maybe you have anomaly reporting set up inside of your different vendors and they can alert you a few days when the numbers don't match up or they create or trigger that anomaly detection. Another use case is maybe someone from another department who relies on that data to make decisions. They come to your desk and say, "Hey, I've noticed our conversions on our checkout process are down and they don't match my other data sources that are fulfilling that checkout." There's a number of different use cases or scenarios where ad hoc problem-solving is applied, but basically this is where most people start. They get problems in sporadically and they try and just track them down and understand them on an individual basis. This is extremely reactive, A) because you're not getting out ahead of the problem B) because you don't know about the problem until you've already experienced data loss, and that is expensive. All the data you're collecting, essentially you're paying a price for it so your tracking vendors. So if you lose one day, two days worth of data that has a cost associated with that data loss.

Chris: (10:12)

Next, in the maturity model, we notice a lot of our clients, or we notice a lot of analysts, will start to recognize patterns. They start to just naturally understand that when a release happens, whenever there's been a code change on the site that their tracking is at risk to break. And so as smart individuals, as motivated individuals, they will go out and they will do manual spot checking. So maybe they know that these certain pages tend to have problems and break after release. Maybe they have certain conversion flows that are of extreme importance. And so even one breakage of one day in the data can cause headaches for your team. So after every single release you'll ask to receive a notification when a property is ready to be tested or ready to be pushed into production. And you'll go out, physically load the page either looking at the network request or looking at the tag bugger, and you're just going to manually do this every single time there is a release. Now this is a step in the right direction, but there are two problems with this. One, you're performing a manual task that could be automated, so you are losing value with your own time, where you could be doing more high value activities with your time rather than manual testing. And two, probably more importantly, you are limited to the number of man hours that you can commit to doing this manual spot checking, which means you cannot be comprehensive in testing your website. If your properties have 10,000 pages, it's physically impossible for one person to check 10,000 pages even in a week. Or maybe you have 50 different conversion flows that are important. How much of your time are you going to dedicate to looking at all those conversion flows? Chances are by doing a manual task, you're going to not be sufficient in the coverage of your site or doing the proper amount of QA that you should be doing.

Chris: (12:30)

Once people do this manual task and they do manual spot checking long enough, some people get pretty savvy and they start to reach out to development resources internally, possibly you'll start to be [inaudible] build out. Um, internal workarounds. Uh, you could be using proxy tools like Charles, maybe there's other tools like selenium, uh, Fiddler or maybe you just, you know, get a dev person's time to go ahead and build something that can automate what you're doing manually. This is helpful, but it's not getting you 100% there. One, you may need guidance on some best practices as far as how to do your coverage too. Once you've built your own solution or you've leveraged a dev resource to build your own solution, somebody needs to manage that solution. They need to update that solution when it breaks, they need to update requirements. And it actually has a huge time cost, not just in the initial configuration or creation of the tool, but in updating the tool with your new standards.

Chris: (13:32)

Because as your tracking gets more complex and advanced, your testing has to go hand in hand with that. And also if you have a dev resource, if something is breaking in that tool, something needs to get fixed, somebody needs to spend the time to do that as well. And if you even start to think bigger picture, you now have other people on your team. Eventually you'd like to democratize testing out throughout the organization just as much as you do tracking or access to your tag management system. So you would, you would prefer to actually empower people when they want to track something. They know that if they track it, they need to test it and so you need them to be able to use your tool or use your solution or you need to train them to do the manual process that you created. As you can see this kind of spiders into a lot of different problems.

Chris: (14:26)

The next thing you want to think about in your release cycle validation, so let's assume that we're able to automate these QA efforts that you've been doing. ObservePoint is a good platform for that. It's very simple and easy to create these flow based tests to check for the data points that are important and to send alerts to who needs to receive alerts when something breaks. Also, these site-wide scans are very simple, they're point and click, but once we've matured to that point, that's only the beginning. Now we need to start thinking about tag governance. Are we scanning the entire site for all the appropriate tag presence? Are we scanning the site to make sure that you know, no unauthorized tags are showing up on the site. If multiple groups or teams have access to the tag management system, then multiple groups and teams can put tags on the site.

Chris: (15:21)

Maybe just a proof of concept with some vendor, but how do we know that they took that tag off the site? Unless you have an automated way that can scale to check all the network requests across your entire site. It's very difficult to get that visibility. In ObservePoint we can create these tag governance scans. We can a make sure that a minimum tagging standard is enforced across all the properties. We can make sure that specific profiles are meeting certain sections of the site, and by profile a good example would be if you have a product page, all product pages should meet a certain standard of tagging. Certain tags need to be firing, certain variables need to be firing and certain values or parameters around those values should be established. ObservePoint can do that very easily, but the flip side of that coin is we also need to make sure that all those unapproved tags are taken off the site or that the person in charge of tag governance is notified when those tags show up.

Chris: (16:29)

Another use case for these white list scans or approved tag scans would be the rogue or piggyback tag problem. A lot of the tags on your site can initiate other tags and that's part of their design, but it's really hard to control what other tags they're initiating. So ObservePoint, as we're monitoring and scanning your site, we will run tests against your approved tracking standard list and if any tracking pixel shows up on your site as we're crawling it that is outside of your approved vendors list, we'll quickly flag it and send you an alert letting you know the URL and the, the URL where the tracking pixel fired and what the tracking pixel is. So again, it could be a well known vendor, it could be an unknown vendor, it could be a little known vendor.

Chris: (17:29)

Now as you're starting to get a hold on all the governance and the automated testing as far as these conversion flows, making sure that the data quality is high and the tracking is not falling off. We want to think about process. And by process is we to make sure that we have these scans set up on a regular cadence. Does that cadence align with your release cycle? So for all the data quality type scans (so these conversion path monitoring scans, a minimum tagging standard scans), we want to make sure that every time there has been a release or a code change on the site, we run these suite of tests to make sure that the tagging is set up correctly. And ideally we'd like to do this in your lower environments first. So if you have a QA environment, we'd like to run these scans in those environments as soon as they're ready to be tested. And if we can catch problems in QA, not only does that save us the problem of possibly losing data in production, but it's also cheaper to fix.

Chris: (18:47)

Also, when we're thinking about process, we need to think about if a new tool is being tracked on the site, have we set up testing as soon as we're doing the tracking. And if you set up testing and tracking hand in hand, you won't lose value or you won't have those dead spots in your tracking so you can maximize the value of the tracking of that tool. Another use case is if you have a content management system that is pushing new content out to the site, we need to create a process where we can test those new pages to make sure that they meet our minimum tracking standards and we need to get ingrained in this process. So whether the CMS, uh, can automatically push out those URLs and an API call and we can get notified and build the scans or maybe someone can notify your team and then we can build out those scans. So there's a lot of options to do it programmatically or if you're just getting notified, uh, we can just load them up. ObservePoint in our audits. We also have a crawler functionality, so we can always be crawling and spidering through the site to try and identify new URLs. As long as they are linked, we can find them.

Chris: (20:12)

This brings us to naturally, you're going to think about landing pages. Landing pages are unlinked, but we need to validate that those pages meet the minimum tracking standard as well. Uh, a good use case or a, a simple example would be we need to make sure [inaudible] that the campaign ID is being passed through to the tracking and that it persists all the way through to the conversion or to the submission or the thank you page of that conversion flow. So we need to think about process and how we're going to test when new pages are pushed out, when there's new tools that we're tracking, things like that. Once we've designed that process, uh, the next logical step would be workflow integrations. So we want to apply more automation to these processes at the beginning. We want to make sure that of the process, we want to make sure all these tests are kicked off automatically.

Chris: (21:12)

So we'll organize these tests by property. So if I have three properties. I have property 1 and I have property 1QA, I have property 2, property 2QA, property 3, property 3QA. Those are six different properties and I'm going to have six different suites of tests for each of those. Now they're probably going to be very similar, especially if your site is templated. We're probably going to be able to write the test once and then just make minor tweaks on the starting URL or something like that and just point them to the different environments. But we need to have each suite of tests live and group together. And so the team in charge of doing the release for property one, they know that when property 1QA environment is ready to be tested, if they're using a Jenkins or some sort of CI continuous integration tool, they can automatically just send an API call to ObservePoint and kickoff those scans.

Chris: (22:14)

Um, today I wouldn't recommend holding up the build process if a failure is triggered, but at least let's get the scans automatically kicked off. So if property one QA environment is ready at three in the morning to be tested, instead of coming in at 8:00 AM the next day and kicking off the test, you can come in at 8:00 AM the next day and just look in your inbox and look at the alerts and you can see property 1QA. We ran 10 tests, nine tests passed and one test had a failure on this URL or this step. And then we can start to notify the appropriate personnel to fix the problem.

Chris: (22:56)

The next part of the workflow integrations that we want to think about is if there is a failure, if we do receive an alert around a data failure, we want to be able to possibly automatically kick out that messaging inside of a Slack channel or a Microsoft teams channel. Maybe we want to create a JIRA ticket automatically. From that out of the box ObservePoint, we'll send you an email alert, we'll let you know the details of the failure and we'll link it to the test and ObservePoint, but sometimes we can automate that and we could automatically create a JIRA ticket and with the web hooks functionality inside of ObservePoint, anytime a scan is completed, we can push any messaging with any of the data points that we crawled to any system that can ingest an API call. So JIRA tickets are very impactful for a lot of our teams. So if ObservePoint runs a test on a certain conversion flow and we got through to the submission page and event 44 didn't fire, we can automatically send you an email but also automatically create a JIRA ticket saying event 44 did not fire on test X on the submission page. And then the team in charge of that can get after it and start working on it immediately while the analyst now has something in their inbox that they can follow up with that team as they're already working on it and check the progress.

Chris: (24:30)

Also inside of these integrations around alerts or when these scans are completed, we can start to think about tiered alerts. Um, maybe some alerts are more important than others. Some data points are more critical to get take action on and some are not as mission critical. So if you want to think in terms of a functional QA, they use verbiage called smoke tests, right? So the most important functionality, so those conversion events, maybe we automatically create JIRA tickets from those. And maybe we have a whole other set of alerts that just get pushed to a Slack channel where there's five or six people on the team that need to see them and they can go ahead at their leisure and work through that list. But it is not mission critical.

Chris: (25:16)

So once we've set up all the automated testing around the conversion flows, we've set up the automated testing on the page load events throughout the entire site. We've created the alerts around the key data points that we care about or profiles that certain sections of the site need to meet. Once we've moved into tag governance and we have scans running to alert you and let you know if any tracking technology shows up anywhere. Once you've integrated these into all your processes, when new content gets pushed to your site, it's automatically being tested. New landing pages are going out in a campaign, those are being tested. And we have the tests being kicked off automatically from the appropriate system. We have the tests automatically creating Slack messages, JIRA tickets, whatever it may be. Now we're much more on the strategic side of things versus the reactive side of things. And once we're on the strategic side of things, then we can start to think about different kinds of tests we want to run. Do we want to integrate lighthouse metrics into our scans? Do we want to see page performance data from there and compare that to tag performance? Do we want to see other data points being pulled into, we want to run scans as an opted in end user versus an opted out user to test consent management. Do we want to set up cookies in ObservePoint to show up as a new user, new user or new visitor versus a return visitor. Um, and really the sky's the limit. ObservePoint is a super flexible platform. Um, we move you through a maturity model, but once you start to hit more of that strategic side of things and you have process locked down, you have workflow automation locked down, you have your testing lockdown. Now you can start to think about what kinds of tests do we want to change the profiles of the browser coming to the site? Do we want to come from different locations? Do we want to leverage proxies? Do we want to scrape the Dom? And do we want to compare elements in the Dom to variables that are being passed through the data layer or the tax? Uh, lots of use cases. Um, but again, QA is such a huge, huge project to implement. I mean, this is really a huge elephant to consume depending where you are in this maturity model, the easiest way is just to think about things big picture. But when we're ready to eat that element elephant, and we want to implement our QA practice, we need to do it one bite at a time. So start small, start with a few tests. And once you've got that process down and you've responded to those tests, you've seen failures, you understand the process, we've automated it, then let's scale. And let's get as big as we can until we hit a point, uh, where our appetite for risk has been satisfied or the team we have in place, we don't want to give them more alerts because they would then get bogged down.

Chris: (28:38)

Thanks for attending the session today. My name is Chris O'Neill. Uh, if anyone has any questions or thoughts, you can always reach out to me. Uh, my email is Thank you.


About the Author

Chris O'Neill

Chris O’Neill is a full-stack web developer who has been in the analytics space for the past 5 years. As Solutions Architect at ObservePoint, Chris helps large enterprises set up automated testing of their analytics scripts with ObservePoint so they can verify custom website tracking is always up and running.

LinkedIn More Content by Chris O'Neill
Previous Video
Adam Greco, Search Discovery – Reasons Analytics Implementations Fail (and How to Avoid Them)
Adam Greco, Search Discovery – Reasons Analytics Implementations Fail (and How to Avoid Them)

Adam will share what he has seen companies struggle with in web analytics and strategies to avoid these iss...

Next Video
Alex Langshur, Cardinal Path – Navigating a Post-Cookie Marketing World
Alex Langshur, Cardinal Path – Navigating a Post-Cookie Marketing World

Alex Langshur, Co-CEO at Cardinal Path, discusses how regulations inhibit companies' ability to use cookies...

Get a free 14-day trial with ObservePoint

Start Your Trial