The highest performance products combine quantitative and qualitative information on a regular if not daily basis to inform decisions and make constant changes. The high-performance teams developing these products succeed because they’ve developed disciplines that make product analytics part of their agile rituals.
Many teams say they’re taking a similar approach but on closer inspection they don’t give enough time or attention to their product analytics and are really acting on instinct rather than data.
This article aims to provide you with some guidelines on how you can incorporate product analytics into your agile rituals through the lens of Scrum.
The guidelines are best understood through these areas:
- How to bring product analytics into your Scrum rituals
- Product Analytics Roles and responsibilities in Scrum
- How to know if its working
These guidelines are focused on Scrum in order to make the guidelines more concrete. That being said, you can take these guidelines and transfer them to Kanban, use them with SAFe or match them to your own flavour of agile.
Product Analytics in Scrum rituals
Attention to data comes through incorporating analytics activities into your regular rituals. Fortunately Scrum has some well-established rituals that you can leverage to get more attention and focus on product analytics.
Making analytics part of your routine will make sure it gets attention but then, over time, will make sure that it is just a regular part of how you develop, what you develop and how your team thinks about their work.
Here is how to make product analytics part of your regular discipline broken down by Scrum ritual (in no particular order):
Sprint Backlog Refinement:
Sprint Backlog Refinement is where you prioritise, add detail and do (rough) estimates for the work you’re aware of. Sometimes it is about adding new work.
You can make product analytics part of this ritual by starting your refinement session with a review of your product analytics. Sometimes it will create immediately obvious work to do, sometimes it will create research tasks to dig into. Sometimes it will highlight glaring oversights in what you are tracking (“what do you mean we put that feature live and have no idea if anyone uses it?” isn’t uncommon to be heard).
By starting your backlog refinement with data, your team can now make informed decisions about prioritisation.
Sprint Planning Meeting:
Sprint Planning is where you decide on what you are going to do in the next sprint, workout who will do it and make concrete plans for getting it done.
Analytics have a limited role to play here if you’ve incorporated product analytics into your Sprint Backlog Refinement. The ways product analytics can play a part in sprint planning are:
- Giving you a last minute sanity check on what you’re going to build. (“Does the data say we still need this?”)
- By ensuring that each task you will work on in a sprint includes details on how analytics will be developed and used for the work. (“How will we track the success of this feature/task?”)
Daily Stand-up
The daily stand-up is a short conversation between team members to get on the same page, uncover blockers and help each other progress.
This is an opportunity to quickly review the key daily metrics for your team or product as well as any focus metrics (such as if you’re focusing on a specific user journey). Things can shift overnight. Additionally, the more regularly you’re talking about analytics the more data driven your team will become.
To make this easier, if often helps if someone on the team (product manager, product owner or analytics lead) has reviewed prior to attending and brings any important insights to the team.
If you don’t have daily analytics then you need to ask whether you need daily or whether daily is overkill for your situation. A consideration here is that daily analytics, with today’s technology, is most often an easy-to-have (usually same effort as weekly) and can only lead to better outcomes.
Sprint Review Meeting
The Sprint Review Meeting is where you review your team’s progress in the completed sprint.
Incorporating product analytics into the Sprint Review Meeting involves:
- Reviewing the initial analytics results of any work that was deployed during the sprint (if you are continuously deploying)
- Ensuring completed features include analytics capabilities so you can measure whether they are successful or not.
- (Optional) reviewing the performance of features/tasks completed in the previous sprint. This activity should be performed during Sprint Backlog Refinement but it might be necessary to include it in the Sprint Review Meeting simply to ensure it happens.
Sprint Retrospective Meeting
The Sprint Retrospective Meeting is where you spend time looking at how to improve.
You can ensure a standing agenda item in the retrospective is focused on reviewing your product analytics. To start the conversation you can ask questions like:
- How could we improve the tools and technology we use to obtain and analyse analytics?
- How could we improve the way we track and capture insights that come from analytics?
- How could we increase the speed at which we action analytics insights?
- What insights from analytics did we miss? How can we ensure we don’t miss them again?
- Are we happy with the way analytics is part of our sprint and rituals? How can we improve this? Do we need more or less focus on analytics?
Product Analytics Roles and Responsibilities in Scrum
In becoming a data-driven team you will need people on the team to take on some product analytics related roles.
The roles you need to consider are:
- Product Analytics Lead: who will be responsible for bringing analytics insights to the team and championing analytics. You do want everyone to share some responsibility but having a lead will help drive improvements.
- Data Engineer: you may need to consider adding an engineer or making one of the engineers on the team responsible for data and analytics because sometimes getting analytics in the form you need requires engineering effort.
- Database Administrator: in some instances product analytics requires database experts to help build queries and transformations with existing data to get data into the format or database you need to do analytics.
- Product Analyst: someone responsible for interpreting product analytics in the context of your commercial objectives, user needs and technology. This might be the Product Manager or Product Owner but it can also be a specialist.
- Data Scientist: someone with mathematics and machine learning understanding that is able to apply more advanced algorithms to your data to perform the research you need to understand your customers and users.
Pro tip: You don’t need a Data Scientist. So many teams that are early or part way through this process jump straight to “oh we need a data scientist to do analytics”. The need for a data scientist when it comes to product analytics is rare. You only need one if you’ve maxed out your options with the other roles, you’re very advanced in your use of analytics (6+ months of usage), or you’re a very special case with a very unique angle to your product.
How to know when it is succeeding
The journey to becoming more data-driven can be a long one and it requires taking people on the journey with you.
Here are some observations that you will start to make as your team becomes more data-driven. These will give you some goal posts to note on your path to product analytics success:
- When you get more advanced with this you will start to add more and more research-style tasks to your backlog to further investigate points of interest and possible solutions to impact your data.
- You’ll start to find what you were thinking you would do next sprint completely changes based on the data you are seeing. This can be a very good thing.
- The squeaky wheel stops getting the grease: prior to including data regularly in rituals teams tend to pay attention to the loudest customer. While this can be a good idea at times, it may not reflect what your broader, satisfied customer base is doing. When you start letting data from your broader customer base override a squeaky wheel then you know product analytics has become a core part of your team’s process.
- Every team member will start contributing ideas of what could be measured and what could be improved based on their own analysis of the data.
- Some team members become obsessed with data – “hey did we manage to improve the conversion rate on that form I was working on last week?”
- Expect push back on your decisions from the team if you haven’t justified it with data.
Happy analytics-ing.