One of the LinkedIn groups that I’m subscribed to, Mobile UX, asked a question about how to benchmark UX.  This is something that we do at VMware, and that I did during my time at Microsoft too.  The methods at both VMware and Microsoft were reasonably similar.

My team has two different methods that we use to benchmark UX: scorecards and baseline studies.

Scorecards are during the product cycle, and rate the user experience of the top use cases. Scorecard studies can be repeated at any time during the development cycle, and allow the team to see progress being made (or not!). We use different methodologies for scorecard studies, depending where we are in the development process and what the resources available are. Scorecards are generally qualitative in nature. The goal is to provide an at-a-glance view of our current UX, with additional details about how to address any UX issues that might exist or what is currently being done about it (say, a link to a bug that has been opened to track a particular issue).

Baseline studies are conducted on the completed product that ships. Baseline studies are quantitative. Participants have a task list which they are asked to complete without thinking out loud. Our important metrics here are time-on-task and success rates. The task list for the baseline usability study is based on the most important use cases for the product, and is intended to be repeated on each subsequent product release.  A baseline study can help you determine whether you met your UX goals for this release.  It can also identify places where you unintentionally had a positive or negative impact on your UX.  Trend data over multiple releases helps others understand the ROI of UX.

I’ve also used baseline studies to help teams understand the usage of their application better.  When an application or team has never had involvement from UX before and isn’t sure how to best use this new UX resource, a baseline usability study of the top use cases often makes it clear where UX should focus their time and attention.  If you learn in your baseline study that your users have trouble completing one of the most important workflows in your application, you know what you’ve got to fix.

I like benchmarking UX to be able to show where we came from, how we got there, and what we learned along the way.  I especially like being able to show where we are today, since it usually illuminates at least part of where we need to go tomorrow.