A fast, powerful way to find your product’s biggest issues

Long, long ago, in the year 2013, I stumbled across an article by Jeff Sauro on the 80/20 rule. After reading his description of conducting Pareto Principle–based user research, I had an epiphany: I realized that I could modify what he was doing to obtain data that my organization had really been struggling to uncover.

What is the Pareto Principle?

In Richard Koch’s book The 80/20 Principle: The Secret of Achieving More with Less, he details how, in 1897, a brilliant researcher named Vilfredo Pareto discovered that the majority of the wealth in England and other countries was predictably controlled by a small minority of the population. Pareto’s research is also known as “the 80/20 rule” and “law of vital few,” among other names.

In the early 1900s, Joseph M. Juran discovered Pareto’s research and realized that the concept also applied to tons of other situations in life: A tiny percentage of criminals caused most crimes, a small percentage of dangerous processes caused a majority of accidents, etc.

He also realized that the concept could be applied to improve consumer and industrial goods. He created a consulting service to work with companies to identify top areas they could improve upon to make the most impact with product enhancements.

Linking the Pareto Principle to user research

By applying the Pareto Principle to user research, you can identify the biggest culprits in your product’s usability issues and feature gaps, and then spend your valuable time and money where they will have the most impact.

Imagine you could uncover a tiny code change that would vastly improve your product UX, for instance — it’s possible you can.

In my case, the principle was startlingly accurate. Our research showed that 18% of our core product areas were causing 83% of our clients’ frustrations.

Would a statistician or professional researcher cringe and shed some tears if they saw the method and data I’m about to show you? Absolutely. We aren’t even calculating standard deviations here. But does the average stakeholder care about that? Nope.

This method is for those who don’t have a background in research or statistics, or for experienced professionals who just need some quick and dirty data. It’s a powerful, fast, and cheap way to quickly evaluate how you can pack the most UX punch when you’re planning improvements to your product or service.


The Research Process

Step 1: Recruit research subjects

Do you have a list of your users? Email them and ask if they’d be interested in joining a “special community of customers who will have the opportunity to impact future changes to the product.” You don’t necessarily have to pay people; just being able to leave their fingerprint on the product is often more than enough motivation to get them involved in the research process.

In our case study, we didn’t pay our subjects a dime. They were all excited to be part of the community and gave us candid, brutally honest feedback.

Step 2: Create your survey

Making a survey is inexpensive — possibly even free — with Google Forms or a tool like Survey Monkey.

You’re going to ask exactly two questions in your survey:

  1. If you could change one aspect of our product, what would you change? (Provide a list of all core content areas and allow only one selection. Do not include an “other” option.)
  2. How would you change it, and why would you make that change? (Make this question open-ended.)

Step 3: Launch your survey

To launch your survey, you can use a plain old email list, or you can get a free subscription to a service like MailChimp.

I prefer MailChimp, and here’s why:

  • The intuitive dashboard.
  • It’s great for tracking open rates, click rates, and other fun stats.
  • You can easily create lists and groups.
  • It provides simple campaign templates.
  • Unsubscribe/spam rules are handled for you.

Step 4: Analyze your data

After you launch your survey campaign, you’ll be flooded with responses — and data! Don’t get overwhelmed; analyzing the data isn’t that intense.

You’ll start by calculating the total responses per core product area.

Part 1: Calculate Totals

  • Export your survey data to a spreadsheet.
  • Sort it by core product area (the ones in your multiple choice question).
  • Calculate totals for how many responses came from each core product area.
  • Order them from most to least.

In this case study, my results looked like this (out of 40 functional areas):

Number of responses

Headlines: 26

Editor: 21

Files and Folders : 21

Forms and Surveys: 21

Groups: 17

Calendar: 12

Reports: 8

Remaining core functional area aggregate responses: 26

Part 2: Calculate Product Area Percentages

Now, let’s do some quick math.

Let’s say I had 152 respondents total. Take the total responses for each key area and divide it by the total number of respondents to get the percentage of respondents who identified each key area.

In this case study, my results looked like this:

Headlines: 26/152 = 17%

Editor: 21/152 = 14%

Files and Folders: 21/152 = 14%

Forms and Surveys: 21/152 = 14%

Groups: 17/152 = 11%

Calendar: 12/152 = 8%

Reports: 8/152 = 5%

Remaining 33 areas total: 26/152 = 17%

Using this information I was able to gather some very useful data:

83% of the 152 responses fell into 7 key functional areas, while 17% fell into other functional areas.

The 7 key areas identified make up 18% of the 40 key functional areas. (7/40 = 18% rounded)

Therefore, 18% of our key functional areas were causing 83% of our clients’ frustrations. So, my results wound up being really close to 80/20! The areas the research identified were shocking; we were expecting completely different results. We made changes in our product based on these results, and then ran the same study the following year. The areas we’d adjusted were knocked out of the list of issues noted by users, and we wound up with another set of data with new areas identified that aligned with the 80/20 rule. We fixed those issues and replicated the study a third year in a row, and once again wound up with similar results. Our team (myself included) was pretty astounded by the consistency of the results year after year.

Step 5: Create a report

Now the fun part: Weaving the data into a simple, skim-able report for stakeholders.

In my instance, I started the report with a two-sentence explanation of the Pareto Principle, then summed up that this study revealed 83% of reported issues stemming from 18% of our core product areas.

Next I gave them the high-level stats for the seven areas of concern:

Headlines: 17%

Editor: 14%

Files and Folders: 14%

Forms and Surveys: 14%

Groups: 11%

Calendar: 8%

Reports: 5%

Finally, I grouped the detailed user feedback by functional area. For example, under a heading of “Headlines” I provided a bulleted list of all of the detailed feedback customers gave in response to the open-ended feedback question.

It made a neat package that my stakeholders loved. At a glance they could see the big picture, but if they wanted to dive deeper into individual pieces of feedback, they could.

These research findings made a big impact on decisions that guided our annual product roadmap planning. We were able to identify usability issues, areas that needed UX love, and even product gaps based on the research findings.

Step 6: Verify results

Stakeholder buy-in (based on your lovely report) should allow you to move on to the last step: following up with your customer base to conduct additional user research in person and through phone interviews, to confirm the results you uncovered and verify that you’re solving the right problems.

Why should your organization take advantage of this research style?

Conducting Pareto Principle–based user research is a trifecta. You obtain a clear view of really powerful data, your clients feel that they are being heard, and the method is simple, cheap, and effective, suitable for professional researchers and novices alike.

If you give a Pareto Principle–based user research study a shot, I’d love to hear about your experience and your results! Hit me up on Twitter and let me know how things go.

Source: https://modus.medium.com/pareto-principle-based-user-research-f90082b2be49