When I joined the product team at Nexonia, our platform needed to scale. We were onboarding more customers than our infrastructure could handle and we had over 500 bugs. Prioritization became increasingly difficult as high-profile customers were also requesting new features. It didn’t help that we had 4 developers on our web platform across four products and one mobile developer for our iOS and Android apps. In order to be ruthless in prioritization, we created a bug backlog score to quantify which bugs were worth fixing every 2 weeks.
I wanted to share this tool with everyone because I understand how difficult it can be to make accurate, timely and tough decisions. With this approach, not only were we able to invest more time in other Product Management activities, but we were able to bring down the average age of bugs from 250 days to less than 30 days.
Here are the 5 steps you need to get started:
- Discover internal stakeholders
- Build the Bug Backlog Score Sheet
- Define the impact
- Assign severity score
- Schedule reoccurring meetings and score bugs
1. Discover Internal Stakeholders
The first step to ensuring the success of this process is to discover and rally your stakeholders that work with customers everyday. They will be your ‘Bug Council’ for coming up with Service Level Agreements (SLAs) and escalating issues on behalf of the customer. The council should have at least:
- Product Management — An individual from the product team that can help facilitate the meeting and/or adjust the JIRA tickets
- Customer Support — An individual that can provide insights on how wide-spread the issues are since they are the escalation point for customers
- Implementation Specialist — An individual that can understand how bugs may impact the customer onboarding experience
- Technical Lead — An individual with enough product and technical experience to provide any technical notes and/or workarounds to issues.
2. Build the Bug Backlog Score Sheet
Now that you have your council, it is time to define, assign severity scores, and iterate on your Bug Backlog Score sheet, which will help you quantify how severe bugs are and prioritize them.
Here is a Google Sheet Template to get you started!
3. Define the impact
There are four main impacts that you need to score bugs against: Functional Impact, Customer Impact , Number of Customers Reporting, and Number of Escalations. For each pillar, define the attributes because you will need to assign a severity score to each attribute.
- Functional Impact — What functions of the product are being impacted? Is a feature not performing as intended? Do they not have access to certain settings or is it just a nagging performance issue?
Active Data/Payment/Business Process Impact; Cannot Access & Edit Feature
App Maintenance
Cannot access Admin Settings
Reporting data output incorrect
Serious performance issues
Feature Not Performing 100%
Display bug, misleading information
Nagging performance issues
Display bug, aesthetic only
2. Customer Impact — How does the bug impact your customers? Does it block them from going live or conducting their business?
New or existing customer at risk
Billing/Payroll Impacted
Business processes halted
Cannot Go-Live
Admin processes stopped
User processes stopped
Not yet reported, widespread
Admin processes inconvenienced
User processes inconvenienced
Not yet reported, not widespread
Display bugs
100% Satisfactory workaround
3. # of Customers Reporting — How many customers have reported the issue? Was the bug discovered by an internal employee? How widespread is the issue?
1 point for each customer reporting
Found Internally, potentially widespread & severe
Found internally, potentially widespread & non-severe
Found internally, potentially widespread & preventative maintenance
Found internally, narrow application & preventative maintenance
Found internally, not potentially widespread, not severe
4. # of Customers Escalating — How many times have customers called or emailed about the issue?
4. Assign severity score
Now that you have list of defined impacts a bug can have on your product, it is time to assign each impact a value with 1 — not important to 10 — mission critical.
Example
For functional impacts, we know that our customers value data integrity as mission critical, so we score that as a 10, while UX/UI may score a 1 because our customers put less importance about the feel and look of the product.
Data Integrity: 10
Cannot access critical function: 9
Configuration Not Retained: 8
Reporting data output incorrect: 7
Display bug, misleading information: 4
Display bug, non-misleading information: 1
When you’re done assigning scores your Bug Backlog Score should look similar to the following Functional Description column:
5. Schedule recurring meetings and score bugs
Now, the fun part — go through each bug and start scoring them based on your assigned severity score. You will start to notice that you may need to reassign severity scores as you go through a couple of bugs, since the scores may not resonate with the team.
It will take several meetings before you’re able to go through all the bugs, so set up recurring meetings to complete the scoring. Once you’ve gone through most of the bugs, it’ll be a breeze to prioritize the most impactful bug fixes.
Tip: If the bugs are the same score, use the First In First Out method. Older bugs should be a higher priority. This will help you reduce the average of a bug, which is an important metric to track because it can represent the average duration of a poor customer experience!
Results
From my experience, we were able to reduce the amount of bugs in the Nexonia mobile apps to zero! Also, our Nexonia web platform no longer had any P1 critical bugs and we were able to focus on priorities that helped increase revenue to the business. In addition, we were able to improve our prioritization speed and accuracy and we had unanimous alignment on what were critical issues across Customer Success and Support teams. You may not see immediate results, but stick to the process and have buy-in from the top, so that you can ruthlessly prioritize the highest customer value every two weeks.
Thanks to Michelle Chan
