All posts by nadyne

user experience

how to benchmark UX

One of the LinkedIn groups that I’m subscribed to, Mobile UX, asked a question about how to benchmark UX.  This is something that we do at VMware, and that I did during my time at Microsoft too.  The methods at both VMware and Microsoft were reasonably similar.

My team has two different methods that we use to benchmark UX: scorecards and baseline studies.

Scorecards are during the product cycle, and rate the user experience of the top use cases. Scorecard studies can be repeated at any time during the development cycle, and allow the team to see progress being made (or not!). We use different methodologies for scorecard studies, depending where we are in the development process and what the resources available are. Scorecards are generally qualitative in nature. The goal is to provide an at-a-glance view of our current UX, with additional details about how to address any UX issues that might exist or what is currently being done about it (say, a link to a bug that has been opened to track a particular issue).

Baseline studies are conducted on the completed product that ships. Baseline studies are quantitative. Participants have a task list which they are asked to complete without thinking out loud. Our important metrics here are time-on-task and success rates. The task list for the baseline usability study is based on the most important use cases for the product, and is intended to be repeated on each subsequent product release.  A baseline study can help you determine whether you met your UX goals for this release.  It can also identify places where you unintentionally had a positive or negative impact on your UX.  Trend data over multiple releases helps others understand the ROI of UX.

I’ve also used baseline studies to help teams understand the usage of their application better.  When an application or team has never had involvement from UX before and isn’t sure how to best use this new UX resource, a baseline usability study of the top use cases often makes it clear where UX should focus their time and attention.  If you learn in your baseline study that your users have trouble completing one of the most important workflows in your application, you know what you’ve got to fix.

I like benchmarking UX to be able to show where we came from, how we got there, and what we learned along the way.  I especially like being able to show where we are today, since it usually illuminates at least part of where we need to go tomorrow.

iPad, iPhone

How to get Pages, Keynote, Numbers, iPhoto, and iMovie for free on your new iPad

4 Comments

I succumbed, and bought a new retina iPad Mini this weekend.  My old iPad, the first release of the iPad 2, was showing its age on iOS7.  Apps were running slower, and the spiffiness of the newest iPads had me thinking about an upgrade.  Based on specs, I couldn’t make the decision between the iPad Air and the iPad Mini; in-store playing with the two iPads side-by-side, I knew that I wanted the iPad Mini within seconds.

I knew that new iOS 7 devices (anything purchased after September 1, 2013) were supposed to get iPhoto, iMovie, Keynote, Pages, and Numbers for free.  After I had transferred everything from my iPad 2 to my new iPad Mini, I opened up the App Store to download these apps.  But they all showed their full price, and didn’t show that they were free for new devices.  My husband also purchased an iPad Mini at the same time (upgrading from the short-lived iPad 3), and the apps all showed up free for him.  If you’re like me and the apps didn’t show up for free, here’s what worked for me to get it:

  1. Quit all apps: double-tap your Home button and swipe every running app up to fully quit out of it).
  2. Clear out Safari’s caches: Open up the Settings app, tap on Safari, then “clear history” and “clear cookies and data”.
  3. Try downloading the apps again: Open up the App Store and search for one of them.  It should show up as “free” now.

The support page says that you’re supposed to be presented with a dialog offering to download all of them for you when you launch the App Store, but that wasn’t the case for me.  I had to individually search them all out.

On my first attempt to do this, it mostly worked.  I was able to search for and install Keynote, Numbers, iPhoto, and iMovie.  However, Pages behaved strangely: it showed up for free, and when I tapped on it to install, I would get the round progress indicator and then the “free” button again.  I tried a couple of times without success.  I finally waited for the other apps to finish installing, verified that they had all worked correctly, and then repeated the steps outlined above.  This time, when I searched for Pages, there wasn’t a “free” or “install” button, just the round button that had previously indicated progress.  When I tapped on it, I got a message saying that I had already purchased the app and that it would download it again to my iPad.  It did, it works, and I haven’t yet seen a charge for it.  As far as I can tell, it has worked.  I can’t explain why everything except Pages behaved well, but at least I was able to finally install it.

I couldn’t find this documented anywhere else, so I’m sharing it in case someone else has the same problem.

user experience, VMware

requirements for being a UX researcher

A major requirement for being a UX researcher is flexibility.  User research has many moving pieces.  Any of them can fail, and a researcher has to be able to handle it with aplomb and with minimal disruption.

If a user study involves a prototype or working code, there is always a chance for error.  There can be bugs in the prototype or unexpected technology issues.  The user could also come up with a different solution than the one intended to be studied by the researcher, which might not work at all.  The user researcher has to try to minimize the chances of such issues in advance, as well as handle them discreetly during the study in such a way that the research participant doesn’t feel interrupted.

Research that involves participants at scheduled times comes with its own set of challenges.  Participants can be late (or early!), or need to reschedule, or not show up.  There is also the chance that a carefully-selected participant might not be a good participant, in which case the researcher has to gently handle the situation to ensure that the participant doesn’t have a bad experience and also that anyone who is observing the research doesn’t draw invalid conclusions.

There are several characteristics that I look for in researchers to indicate how good they are.  Researchers tend to be scrupulously on-time (often early), because they never want to keep a participant waiting.  They tend to keep their calendar updated so that they can accurately schedule participants.  They build in time for unforeseen issues.  They are excellent communicators, and are also very good at following up.  They take excellent notes, and they are able to get to the heart of the matter quickly.  They can get up-to-speed on something new very quickly, and ask excellent questions along the way.

This is all a very long way of saying that my team is currently hiring for a researcher.  Ping me if you’re interested.

user experience

user research fallacies

Wired posted an interesting opinion piece about user research, although its title is about startups and innovation.  The central point of the article is that startups often skip user research on the theory that they can fail fast and pivot to the next thing, and that this works but has a massive opportunity cost associated with it1.  This paragraph quite nicely sums up many of the arguments against doing user research:

When the research focuses on what people actually do (watch cat videos) rather than what they wish they did (produce cinema-quality home movies) it actually expands possibilities. But a common concern and excuse for not doing research is that it will limit creative possibilities to only those articulated by the target users, leaving designers devising a faster horse (lame) rather than a flying car (rad).

Research isn’t transcription.  There’s a lot of other things that research is and isn’t, and the Wired piece is a great starting point to learning more about it.

  1. I’d argue that this theory isn’t limited to startups. ↩
user experience

context matters

Of all of the methods of doing user research, contextual inquiry and ethnography are two of my favorites.  They help you understand your the context that your user is operating in.  Working for VMware, I often think about this in terms of what is happening in a system administrator’s datacenter, what their workspace looks like, what kind of mobile devices they use and why, and so on.

These methods are important for consumer products as well.  Take, for example, Procter and Gamble’s newest men’s razor for the emerging Indian market.  They knew from talking to Indians living in America about some characteristics of Indian men that they would have to take into account in designing their razor.  They even tested a new razor on Indian students at MIT, and that razor failed.  It took a trip to India to understand the context of men shaving there for them to fully understand the user need.  Indian students at MIT had running water to rinse their razor in.  Men in India were using a cup of water to shave, doing it in the dark without a mirror, and the process would take around a half-hour.  They also learned that not cutting themselves was a much larger concern for men in India.

The whole article is a really interesting read about what happens when you think you know your user, or when you think you’ve got a pretty good stand-in for your user.  P&G didn’t know their user, and they didn’t know their context.  Once they understood their user, their concerns, and their context, they were able to develop a razor that captured 9% of the market in India, bringing P&G’s overall market share to 49.1%.

(Thanks to @JakeHercules for the pointer to this article!)