One of the LinkedIn groups that I’m subscribed to, Mobile UX, asked a question about how to benchmark UX.  This is something that we do at VMware, and that I did during my time at Microsoft too.  The methods at both VMware and Microsoft were reasonably similar.

My team has two different methods that we use to benchmark UX: scorecards and baseline studies.

Scorecards are during the product cycle, and rate the user experience of the top use cases. Scorecard studies can be repeated at any time during the development cycle, and allow the team to see progress being made (or not!). We use different methodologies for scorecard studies, depending where we are in the development process and what the resources available are. Scorecards are generally qualitative in nature. The goal is to provide an at-a-glance view of our current UX, with additional details about how to address any UX issues that might exist or what is currently being done about it (say, a link to a bug that has been opened to track a particular issue).

Baseline studies are conducted on the completed product that ships. Baseline studies are quantitative. Participants have a task list which they are asked to complete without thinking out loud. Our important metrics here are time-on-task and success rates. The task list for the baseline usability study is based on the most important use cases for the product, and is intended to be repeated on each subsequent product release.  A baseline study can help you determine whether you met your UX goals for this release.  It can also identify places where you unintentionally had a positive or negative impact on your UX.  Trend data over multiple releases helps others understand the ROI of UX.

I’ve also used baseline studies to help teams understand the usage of their application better.  When an application or team has never had involvement from UX before and isn’t sure how to best use this new UX resource, a baseline usability study of the top use cases often makes it clear where UX should focus their time and attention.  If you learn in your baseline study that your users have trouble completing one of the most important workflows in your application, you know what you’ve got to fix.

I like benchmarking UX to be able to show where we came from, how we got there, and what we learned along the way.  I especially like being able to show where we are today, since it usually illuminates at least part of where we need to go tomorrow.

Wired posted an interesting opinion piece about user research, although its title is about startups and innovation.  The central point of the article is that startups often skip user research on the theory that they can fail fast and pivot to the next thing, and that this works but has a massive opportunity cost associated with it¹.  This paragraph quite nicely sums up many of the arguments against doing user research:

When the research focuses on what people actually do (watch cat videos) rather than what they wish they did (produce cinema-quality home movies) it actually expands possibilities. But a common concern and excuse for not doing research is that it will limit creative possibilities to only those articulated by the target users, leaving designers devising a faster horse (lame) rather than a flying car (rad).

Research isn’t transcription.  There’s a lot of other things that research is and isn’t, and the Wired piece is a great starting point to learning more about it.

Of all of the methods of doing user research, contextual inquiry and ethnography are two of my favorites.  They help you understand your the context that your user is operating in.  Working for VMware, I often think about this in terms of what is happening in a system administrator’s datacenter, what their workspace looks like, what kind of mobile devices they use and why, and so on.

These methods are important for consumer products as well.  Take, for example, Procter and Gamble’s newest men’s razor for the emerging Indian market.  They knew from talking to Indians living in America about some characteristics of Indian men that they would have to take into account in designing their razor.  They even tested a new razor on Indian students at MIT, and that razor failed.  It took a trip to India to understand the context of men shaving there for them to fully understand the user need.  Indian students at MIT had running water to rinse their razor in.  Men in India were using a cup of water to shave, doing it in the dark without a mirror, and the process would take around a half-hour.  They also learned that not cutting themselves was a much larger concern for men in India.

The whole article is a really interesting read about what happens when you think you know your user, or when you think you’ve got a pretty good stand-in for your user.  P&G didn’t know their user, and they didn’t know their context.  Once they understood their user, their concerns, and their context, they were able to develop a razor that captured 9% of the market in India, bringing P&G’s overall market share to 49.1%.

(Thanks to @JakeHercules for the pointer to this article!)