A truism of my job is that user research begets more research.  There’s two major reasons this is true: you always have more research questions than you can answer in any given amount of time, and you always learn something when you’re conducting research that you want to learn more about.

The first problem is the most apparent when you set out to do research.  You begin by identifying what questions you want to answer.  Your list of questions grows as you identify themes and trends.  As you share your research questions with others, they have more questions to add to it.

Once you feel like you’ve got a good list of questions, you then have to prioritize them.  Some questions are more important than others.  When I’m prioritizing my list of research questions, I try to determine what action will be taken if I answer a given question.  If I think that it would simply be nice to know a given piece of information, then it’s immediately struck from my list.  I craft my list such that every research question has something actionable that comes out of it.

Once the prioritization is done, then you have to decide what methodology you will use to answer those questions.  The methodology that you will use is dependent on many factors that are external to your research questions, such as the time that you have available, your budget, access to appropriate users, and support from your management.  Even in the best of circumstances when you have an infinite budget, lots of access to the right people, and lots of time, the methodology that you choose is unlikely to answer all of the research questions that you have identified.  There are a lot of factors in play, and you’ll have to make a decision about the best way to proceed.  You will have to leave some of your research questions unanswered, oftentimes with the hope of being able to revisit them at a later point to answer them.

As you’re conducting your research, you will learn new things.  This is, after all, why you’re conducting your research in the first place.  Inevitably, your new information will create new research questions.  Sometimes this will happen early in your research, and you’ll have an opportunity to tweak your methodology in an attempt to try to answer this new question.  In this case, you have to choose whether your new research question is one that you can answer using your current methodology, and whether it’s more important to answer this question than one of the ones that you’ve already identified.  You might have to remove an existing research question to make way for the new one.

In many cases, new research questions arise as you’re analyzing your data.  Perhaps you can’t answer one of your existing research questions because you need to know something else.  Perhaps you learn something entirely new that you don’t understand, so you know that you want to conduct additional research about it.

Perhaps a different way to look at this truism is to say that you have to accept that you’ll never answer all of the questions that you can identify.  I always have a running list of research questions.  That list always gets longer.  When I left my previous employer, I shared my running list of research questions with my colleagues in the hopes that perhaps it would be useful to them.  One of the hardest things about leaving my previous position was the day that I deleted that running list of questions.  I’d invested so much effort into my research there, and there was still so much left that I could learn.  I put off deleting that file for days.  Conversely, one of the most awesome days that I’ve had since I joined VMware was the day when I created my new running list of research questions that I’d like to answer some day.

Research begets more research.  While I sometimes find it frustrating that I’ll never answer all of the questions that I have, it’s really one of the things that I love about my job: there’s always something new to learn.  The day when I run out of research questions is probably the day that I’m taken off of life support.

A blog post that ends with “Microsoft UI has officially entered the realm of self-parody” is going to get quite a lot of mileage.  I lost count of how many times I saw it go by on my twitter stream.  Laurie Voss posted a response to Steven Sinofsky’s MSDN blog post about the improvements to Windows Explorer that are coming in Windows 8.

Voss takes a look at the data that Sinofsky posted about the usage of various commands in Windows Explorer, and is less than impressed at how they’ve applied this data to the new design of Windows Explorer.  He has two major complaints.  First, he complains that even by Microsoft’s own data, many of the commands that are elevated in the new Windows Explorer design are ones that aren’t commonly used.  Second, he complains that Microsoft’s data says that the menu bar within the Windows Explorer is very infrequently used, so what’s the point of doing it at all?

Both of these complaints are a very common misuse of usage data.  Usage data only tells you what happened in the past.  It doesn’t tell you why it happened, nor does it tell you what will happen in the future.  Furthermore, usage data can often not be broken down very far, so we don’t know what types of users and usages it represents.  I’ve written about this before in my blog post about the usage fallacy:

Usage data is directional. It doesn’t tell you what action to take, it tells you that there might be an action to take.

This certainly applies here.  For example, Voss is upset that few users use the menu bar, instead using contextual menus.  Do we know why they don’t use the menu bar?  Are there commands that they’re more likely to use in the menu bar?  Is there a discoverability problem (that is, are there commands that users would like to use but can’t find them)?  None of these are questions that can be answered by usage data.  To answer these questions, you need to use other research methodologies.

The true irony of Voss’s lack of understanding of how to appropriately apply usage data is found in another recent blog post of his about the need for statisticians, in which he says the following:

The whole web industry is accumulating vast quantities of data and storing it, magpie-like, as if it has intrinsic value, aided by ever-falling prices for storage. But the data isn’t valuable. It doesn’t mean anything until somebody who knows what they’re doing looks at it, sifts through it, and produces a tool that lets others use it to draw valid and useful conclusions.

He’s right: data isn’t valuable until someone who knows what they’re doing looks at it and helps draw valid and useful conclusions.  It’s always amusing when bloggers officially enter the realm of self-parody.

The summer intern program at VMware is robust.  My user experience team gets a few summer interns every year, and sets them loose on several forward-thinking projects.  They’ve got 10-12 weeks to go through the whole design process for their project.

This summer, I worked with one of our interns towards the end of her project.  She wanted to collect some user feedback about her work, and wasn’t sure how to go about it.  Her mentor suggested that she work with me to see where I could help her collect this feedback. After we had decided on our methodology, I assigned her the task of creating a walkthrough of her design to present to our users.

Together, we iterated on her walkthrough several times.  In her first attempt, she filled the entire time available to us by presenting her ideas and designs, not leaving enough time to gather feedback.  I gave her advice on what kind of research questions were appropriate and what kind of feedback she could get, and helped her hone her walkthrough so that she would get feedback that was directed and actionable.  I also gave her advice on her presentation style, how to answer questions, and what to do when she felt stuck.  Her final walkthrough was tight and focused, and she did a great job with it.

At the end of her internship, she gave a presentation to the whole user experience team about her project.  She talked about how she started, how she iterated on her design, how she collected feedback from stakeholders, where our research fit into her design.  At the end, she shared some quotes from team members which will stick with her.  One of them was from me:

Every minute you talk is a minute that your user isn’t giving you data.

(At that point, another designer leaned over to me and said, “yeah, that’s something you’d say”.  How well they know me!)

One of the things that we talked about as she worked on her walkthrough was brevity.  When you’re collecting data from your user, your goal is to collect as much actionable data as possible in the time that you have available.  This is just as true for a 5-minute survey as it is for a 2-hour usability study.  If you are talking, your user isn’t.  You can hear yourself talk any time, whereas you only have a very short window of opportunity with your user.  Therefore, when you’re collecting user data, you have to carefully craft what you present to them to maximize the data that they are able to give you.

Her first attempt at her walkthrough required a full hour for her to go through it.  What she actually presented to our users took about 10 minutes.  The rest of the time, we were able to ask questions and probe our users for feedback.  It took a lot of hard work to hone the focus of the walkthrough down to something that was so much shorter than the original, but it ultimately paid off.  We learned a lot, and were able to identify some important design improvements that could be made.

If I had to pick one message that a user experience intern would take away from working with me as we conducted research, I think this is a pretty good one.  I was very pleased to see her call that out as one of her lessons learned here at VMware.

Several times lately, I’ve been approached by a couple of the newer designers on my team who are conducting research for VMware for the first time.  In our conversations, it’s become clear that there’s a difference between what you learn about conducting research in school and what you learn when you’re actually out in the field conducting research.

Finding participants for research is always harder than you think it will be, and it takes longer than you think it will.  Not only do you have to get the right participant to take part in the study, scheduling them is difficult because you’ve got to work with your own schedule and theirs too.  When you’re asking others for leads, such as a Program Manager or a Technical Account Manager, you’re going to have to build a relationship with that person and help them understand your research goals to be able to get the right people.

You will always have more research questions than you can answer in the time available.  You’re going to have to prioritize your research questions.  While you’re actually conducting the research, you’re going to find additional research questions, which you might decide should be prioritized higher than the research questions that you’ve already identified.  Be aware that this will happen, and be flexible about it.

You have to get comfortable with silence.  Don’t rush in to fill space.  Let your participant think, let them consider what they’ll do or say next.  But don’t let the silence go on for too long, and don’t forget to prompt a participant who isn’t talking at all.  It’s a fine line to walk, and it takes some practice.

Conducting research isn’t just about having a solid research plan and protocol.  There’s a lot of mechanics surrounding conducting the research that can make or break your research.  The best research protocol won’t give you meaningful results if you can’t talk to the right people or if you have too many questions for the time available.

When I mentioned that I was going into the lab to do a usability study, I got a mail asking:

How do you prepare for a usability study?  What materials do you produce?

In this case, I got a request from the Zimbra team to help them understand the behavior of their users.  I met with the team a few times to understand their needs, and then prepared a research plan.  For this study, the research plan was a single page.  It contained the research questions and how I planned to answer them, as well as a discussion of the type of users that I will recruit to help us answer these questions.  Once the team bought off on the research plan, I got down to the real work.

First, I produced a draft of the task list.  The task list is the list of stuff that I’m going to ask my participants to do in the study.  Creating a task list is at least as much art as it is science.  You have to create tasks that feel natural and appropriate for the environment that you’re studying, and you have to avoid leading language.  In the case of Zimbra, this isn’t exactly easy.  I was interested in several scenarios in the calendar, but here are some words that I can’t use in a task list: meeting, appointment, event, recurring, repeating.  How do I tell a participant to create a meeting without actually telling them a word that they’ll see on the screen as they do the task?

The task list had several iterations.  I iterated on it a couple of times by myself.  Once I felt like I had a good draft, I sent it to the Zimbra team to ensure that I wasn’t missing something that we wanted to study.  I met with my fellow researchers to ask them for feedback on it.  I did a pilot study to check for time and flow (more on that in a minute).  All of this got incorporated into the final task list.  For this study, I ended up doing three major revisions of the task list from my first draft through to the final document that I used in the study.

The task list turned into the moderator script.  The moderator script is a superset of the task list, and it covers everything that I say during the study.  It also notes the different ways that a participant could complete the given task.  Having this information immediately at hand helps me to follow what the participant is doing during the study.  It also makes it easy as I’m taking notes, since I can just jot down that the participant took Path A through the interface.

Then there’s a bunch of ephemera associated with simply running the test.  I use a checklist between participants to make sure that I get the testing environment set up correctly for each person.  I’ve got an end-of-day checklist too, which reminds me to do things like print off any needed materials for the next day’s participants, ping those participants to remind them what time they’re scheduled to come visit my lab, and send any schedule updates to the team.  For my own note-taking purposes, I create a Word document for each participant, which contains all of the notes that I take during that participant’s session.

I also have an observer survey.  I ask any member of the team who comes in to observe a participant to note their top three observations during the study.  This helps me if I miss something.  It also allows me to see the study through the eyes of someone who isn’t necessarily well-versed in a usability study.  Their comments often help me to craft the report of research results afterwards because I have some additional insight into how they see both their application and their users.

I always do a pilot study before I actually begin the study.  If I’m running the study on live code that isn’t going to change during the study, then I do this the day before the study is to begin.  If I’m using a prototype (paper, Flash, Flex, whatever), then I do it a few days before the study is set to begin.  This pilot allows me to make sure that the testing environment is working properly, the study flows properly, and everything fits into the allotted time.  I always uncover one problem during the pilot study.  It never fails.  If it’s live code, then the issue is usually just with my task list, which I can update quickly.  If it’s a prototype, then an issue in the prototype usually takes a little bit more time to fix, which is why I do the pilot earlier to accommodate for that.

Then I conduct the study, which is the easiest part of this.  During the study, I don’t do much other than collect data.  I save that all for what happens after the study, which is probably a blog post on its own (if there’s any interest, that is).