1. Using a scorecard helps organizations balance their strategic objectives across four perspectives: 1. What you measure is what you get. Quantifying user experience. As such, it is impacted by completion rates which are context-dependent (see #1 above) and task times which fluctuate based on the complexity of the task. Identifying clear goals will help choose the right metrics to help you measure progress. Figures 1, 2, and 3 show example scorecards (with names and details redacted or changed) that can be shown electronically or printed. Generally errors are a useful way of evaluating user performance. We create separate scorecards for each task that allow teams to dig into more specific task measures or understand whatâs causing problems (Figure 3). While youâll want to tailor each scorecard to each organization, here are some common elements we provide as part of our UX benchmark studies and ways we visualize them (and some advice for creating your own). Scorecards can help you measure: Rating Scale Best Practices: 8 Topics Examined. As UX designers, we need to challenge the sole reliance on data-backed hunches. You need to consider the audience and organization. Generally errors are a useful way of evaluating user performance. What metrics does Balanced Scorecard include" The most important one - the "key" metrics. 1. This scorecard template is focused on the financial performance of the business. Table 1: Raw System Usability Scale (SUS) scores, associated percentile ranks, completion rates and letter grades. Itâs not always possible to include both on one scorecard so consider using different ones that are linked by common metrics. It allows teams to track changes over time and compare to competitors and industry benchmarks. While helping Google product teams define UX metrics, we noticed that our suggestions tended to fall into five categories: 1. SUS scores range from 0 to 100. Non-UX execs will want the bottom line: red, yellow, and green, and maybe grades. ; Normalized indicators are presented in a hierarchical structure where they contribute to the performance of their containers. Like in sports, a good score depends on the metric and context. The value of usability scorecards and metrics" on Thursday, November 15 at 3:30 p.m. EST. The user error rate (UER) is the number of times a user makes a wrong entry. Weâve found providing visual error bars help balance showing the precision without overwhelming the audience. Tracking would help increase user experience maturity. Other benchmarks can be average scores for common measures (e.g. Calculate results by this form and find a common value: (Result 1 + Result 2 + …+… The scorecard shows overall SUPR-Q scores (top) and task-based scores that are aggregated (SUM) and stand-alone (completion, time, ease). Metrics & UX metrics. After all, a bad experience is unlikely to lead to a satisfied customer. They represent a product’s user experience, which is hard to quantify. 10.Â The average number of errors per task is 0.7: Across 719 tasks of mostly consumer and business software, we found that by counting the number of slips and mistakes about two out of every three users (2/3) had an error. The scenario-based UX metrics scorecard in practice 18 Scenario-based UX metrics scorecarding summary 20 UX metrics as part of a customer-centric strategy 22 About the Author 23 User Experience Metrics: Connecting the language of UI design with the language of business. While that’s bad for a usable experience, it means a small sample size of five users will uncover most usability issues that occur this frequently. 2.Â Consumer Software Average Net Promoter Score (NPS) is 21%: The Net Promoter Score has become the default metric for many companies for measuring word-of-mouth (positive and negative). This suggests that users are less loyal to websites and, therefore, less likely to recommend them. Standardization is good, but not if it gets in the way of communicating and helping prioritize and understand improvements to the experience. Give a questionnaire to people who know your product (at least 10 users outside your team). Contact Us, Consumer Software Average Net Promoter Score (NPS) is 21%. Average Single Usability Metric (SUM) score is 65%: Usability problems in business software impact about 37% of users: The average number of errors per task is 0.7: Associating completion rates with SUS scores, Standardized Universal Percentile Rank Questionnaire, User Experience Salaries & Calculator (2018). Denver, Colorado 80206 3300 E 1st Ave. Suite 370 6.Â Average Task Difficulty using the Single Ease Question (SEQ) is 4.8: The SEQ is a single question that has users rate how difficult they found a task on a 7-point scale where 1 = very difficult and 7 = very easy. 1 + 303-578-2801 - MST UX pros will want to dig into the metrics and will be more familiar with core metrics like completion, time, etc. We use colors, grades, and distances to visually qualify the data and make it more digestible. Collecting consistent and standardized metrics allows organizations to better understand the current user experience of websites, software, and apps. Financial—The Financial Perspectiveexamines the contribution of an organization’s strategy to the bottom line. It could be that the bulk of users on any one website are new and are therefore less inclined to recommend things they are unfamiliar with. They show us behaviors, attitudes, emotions — even confusion. Task-level metrics: The core task-level metrics address the ISO 9241 pt 11Â aspects of usability: effectiveness (completion rates), efficiency (task time), and satisfaction (task-level ease using the SEQ). Rating Scale Best Practices: 8 Topics Examined. UX scorecards are an excellent way to visually display UX metrics. Figure 2: Example UX scorecard (non-competitive) comparing experiences across device types. Time On Task Knowing how long it takes for your users to complete a task will give you valuable insight into the effectiveness of your UX design. 4.Â Average System Usability Scale (SUS) Score is 68: SUS is the most popular questionnaire for measuring the perception of usability. The task metrics in Figures 1, 2, and 3 have small horizontal lines showing the precision of the estimate. Errors can tell you … They can be used to more visibly track (and communicate) how design changes have quantifiably improved the user experience. Associating completion rates with SUS scores is another way of making them more meaningful to stakeholders who are less familiar with the questionnaire. A competitive benchmark study provides ideal comparison for all metrics. Study-level metrics: Include broader measures of the overall user experience. Increasingly those metrics quantify the user experience (which is a good thing). Happiness: measures of user attitudes, often collected via survey. Remember — we need fair answers. But UX metrics can complement metrics that companies track using analytics—such as engagement time or bounce rate—by focusing on the key aspects of a user experience. Across the 500 datasets we examined the average score was a 68. Since in the real-world people are more likely to talk about their frustrations, rather than how satisfied they are, a good approach can … The HEART framework is a set of user-centered metrics. Only 10% of all tasks we’ve observed are error-free or, in other words, to err is human. Figure 3: Example task-level score card that dives deeper into the task-level experience and metrics between three competitors on two platforms. If users cannot complete what they came to do in a website or software, then not much else matters. Examples mi… Latin and Greco-Latin Experimental Designs for UX Research, Improving the Prediction of the Number of Usability Problems, Quantifying The User Experience: Practical Statistics For User Research, Excel & R Companion to the 2nd Edition of Quantifying the User Experience. It measures perceptions of usability, credibility and trust, loyalty, and appearance. Latin and Greco-Latin Experimental Designs for UX Research, Improving the Prediction of the Number of Usability Problems, Quantifying The User Experience: Practical Statistics For User Research, Excel & R Companion to the 2nd Edition of Quantifying the User Experience. UX benchmark studies are an ideal way to systematically collect UX metrics. Balanced Scorecard (BSC) is a well-articulated approach to understanding how to describe strategy and metrics. User experience metrics aren’t just about conversions and retentions. The term "scorecard" has been a little hijacked by the " Balanced Scorecard" approach of analysing your business; however a scorecard only needs to contain data that is useful to you in your circumstances.Scorecards are particularly useful when used on an overview KPI dashboard because … Use multiple ways to visualize metric performance (colors, grades, and distances) and include external benchmarks, competitor data, and levels of precision when possible. Figure 3 also shows task-level metrics for two dimensions: platform (desktop and mobile) and competitor (base product and two competitors). This is one of the advantages of using standardized measuresÂ as many often have free or proprietary comparisons. 2. Latency- The amount of time it takes data to travel from one location to another. User experience scorecards are a vital way to communicate usability metrics in a business sense. They should be tailored to an organizationâs goals and feature a mix of broad (study-level/product-level) and specific (task-level) metrics. They can, however, be difficult to interpret and include in scorecards. 68 for SUS, 50% for SUPR-Q) or even other comparable products. For example: satisfaction, perceived ease of use, and net-promoter score. While a “good” completion rate always depends on context, we’ve found that in over 1,100 tasks the average task completion rate is a 78%. UX scorecards: Quantifying and communicating the user experience. Showing this precision can be especially important when tracking changes over time. 3. Its 10 items have been administered thousands of times. With multiple visualization of metrics, external benchmarks, or competitors, it becomes much easier to identify where you want to go. These usually include SUPR-Q, SUS, UMUX-Lite, product satisfaction, and/or NPS. Are Sliders More Sensitive than Numeric Rating Scales? This process provides an opportunity to build consensus about where you're headed. User Error Rate. You can ask users how satisfied they are with particular features, with their experience today and of course overall. 1 + 303-578-2801 - MST However, most of the datasets I have used are only 3-metric SUM scores. And finally, the goal of Balanced Scorecard is to measure, yes, the performance of your business, focusing on some specific aspects. Senior executives understand that their organizations measurement system strongly affects the behavior of managers and employees. Confidence intervalsÂ are an excellent way to describe the precision of your UX metrics. But they can be a good way for tracking and promoting your design change efforts. 2. Website Average Net Promoter Score is -14%: Average System Usability Scale (SUS) Score is 68, High task completion is associated with SUS scores above 80, verage Task Difficulty using the Single Ease Question (SEQ) is 4.8. Balanced Scorecard is a system that aligns specific business activities to an organization’s vision and strategy. But some useful frameworks can help measure user experience. It represents the strategic objectives of an organization in terms of increasing revenue and reducing cost. One of the first questions with any metric is “what’s a good score?”. The traditional financial performance measures worked well for the industrial era, but they ar… Photo by David Paul Ohmer - Creative Commons Attribution License http://www.flickr.com/photos/50965924@N00 The Role of Metrics in UX Strategy Figure 1 shows these metrics aggregated into a Single Usability Metric (SUM)Â and in disaggregated form at the bottom of the scorecard for three competitor websites. Collecting consistent and standardized metrics allows organizations to better understand the current user experience of websites, software, and apps (Sauro, … You may not realize that different members of your team have different ideas about the goals of your project. They can be both subjective and objective, qualitative and quantitative, analytics-based and survey-based. 5.Â High task completion is associated with SUS scores above 80: While task completion is the fundamental metric, just because you have high or perfect task completion doesn’t mean you have perfect usability. Figure 4: Example âoverviewâ card that can be linked or reference on scorecards for more detail on study metrics and task details. Engagement: level of user involvement, typically measured via behavioral proxies such as frequency, intensity, or depth of interaction over some time period. Uptime- The percentage of time the website or application is accessible to users. First, indicators are normalized (according to their properties like measurement scale and performance formula). Despite the context-sensitive nature, I’ve seen that across 100 tasks of websites and consumer software that the average SUM score is 65%. UX research must be at the core of the business and with it the qualitative way of acquiring feedback. Its 10 items have been administered thousands of times. Contact Us, UX metrics to collect in benchmark studies, User Experience Salaries & Calculator (2018). Quantifying the user experience is the first step to making measured improvements. Average System Usability Scale (SUS) Score is 68: SUS is the most popular questionnaire for measuring the perception of usability. UX scorecards are of course not a substitute for digging into the reasons behind the metrics and trying to improve the experience. Customer satisfaction is probably the best barometer of the quality of the user experience provided by a product or service. A score of 50% means half the websites score higher and half score lower than your site’s SUS score. The old paradigm of analytics is geared more towards measuring progress against business goals. The supplier metrics were evaluated both on impact to the supply chain process as well as the measurability by both suppliers and drilling contractors.