user experience

user research begets more research

A truism of my job is that user research begets more research.  There’s two major reasons this is true: you always have more research questions than you can answer in any given amount of time, and you always learn something when you’re conducting research that you want to learn more about.

The first problem is the most apparent when you set out to do research.  You begin by identifying what questions you want to answer.  Your list of questions grows as you identify themes and trends.  As you share your research questions with others, they have more questions to add to it.

Once you feel like you’ve got a good list of questions, you then have to prioritize them.  Some questions are more important than others.  When I’m prioritizing my list of research questions, I try to determine what action will be taken if I answer a given question.  If I think that it would simply be nice to know a given piece of information, then it’s immediately struck from my list.  I craft my list such that every research question has something actionable that comes out of it.

Once the prioritization is done, then you have to decide what methodology you will use to answer those questions.  The methodology that you will use is dependent on many factors that are external to your research questions, such as the time that you have available, your budget, access to appropriate users, and support from your management.  Even in the best of circumstances when you have an infinite budget, lots of access to the right people, and lots of time, the methodology that you choose is unlikely to answer all of the research questions that you have identified.  There are a lot of factors in play, and you’ll have to make a decision about the best way to proceed.  You will have to leave some of your research questions unanswered, oftentimes with the hope of being able to revisit them at a later point to answer them.

As you’re conducting your research, you will learn new things.  This is, after all, why you’re conducting your research in the first place.  Inevitably, your new information will create new research questions.  Sometimes this will happen early in your research, and you’ll have an opportunity to tweak your methodology in an attempt to try to answer this new question.  In this case, you have to choose whether your new research question is one that you can answer using your current methodology, and whether it’s more important to answer this question than one of the ones that you’ve already identified.  You might have to remove an existing research question to make way for the new one.

In many cases, new research questions arise as you’re analyzing your data.  Perhaps you can’t answer one of your existing research questions because you need to know something else.  Perhaps you learn something entirely new that you don’t understand, so you know that you want to conduct additional research about it.

Perhaps a different way to look at this truism is to say that you have to accept that you’ll never answer all of the questions that you can identify.  I always have a running list of research questions.  That list always gets longer.  When I left my previous employer, I shared my running list of research questions with my colleagues in the hopes that perhaps it would be useful to them.  One of the hardest things about leaving my previous position was the day that I deleted that running list of questions.  I’d invested so much effort into my research there, and there was still so much left that I could learn.  I put off deleting that file for days.  Conversely, one of the most awesome days that I’ve had since I joined VMware was the day when I created my new running list of research questions that I’d like to answer some day.

Research begets more research.  While I sometimes find it frustrating that I’ll never answer all of the questions that I have, it’s really one of the things that I love about my job: there’s always something new to learn.  The day when I run out of research questions is probably the day that I’m taken off of life support.

Nadyne, user experience

the user experience of daily deals

1 Comment

Scouring the Internet for a good deal isn’t new.  It’s easy(ish) to compare prices online and find the best one.  Some websites have been compiling daily lists of good deals.  Consumerist has posted morning deals to its website for ages, and MyPoints has its daily deals too.

Now, daily deals are all the rage.  They’ve evolved from the lists of deals that someone has dug up into offering specific deals tailored to someone’s location or interests.  There’s Groupon and its ilk, with their focus on location.  There’s also the interest-based ones such as Fab and Bookperk.  Each of these daily deals has a different user experience.

So far, I’ve purchased a couple of local daily deals.  A Groupon came up for Books Inc, a bookshop that is about four blocks from my home.  PurpleTie, the dry cleaner that services my office, also offered a Groupon that I jumped on.  Groupon made headlines with their Nordstrom Rack coupon, which I took advantage of (along with the rest of the world).  In total, I’ve bought six Groupons.  Likewise, I’ve purchased some from LivingSocial: a manicure, a gift certificate to Amazon, a deal for Amoeba Music.

All of those daily deal website emails get filtered off to a folder, and I scan the subject lines to see if there’s anything of interest.  Generally, though, they go unopened.  I find them pretty repetitive.  I don’t want to try every restaurant in the San Francisco Bay Area, and I really don’t need multiple photography classes.  I only open them if one catches my eye, both for being close enough to my home and of matching my interests.  I probably don’t even open one of these emails per week.

There are several reasons for this.  Aside from their repetitive nature, most of these daily deals are unknowns.  In exchange for a (sometimes steep) discount, I’m usually taking a risk on an unknown.  If you look over the list of ones that I have actually purchased, I’m much more likely to purchase them if it’s for someplace with which I’m already familiar.  Another aspect is that I’m not actually purchasing something, but rather a voucher to redeem at some later point.  This means that I have another hoop to jump through later.

Take the example of the LivingSocial manicure voucher that I purchased.  The user experience of redeeming the voucher was such a huge hassle.  The merchant wouldn’t accept walk-ins.  I wouldn’t mind that if booking an appointment were a more reasonable process.  They didn’t even call back for three weeks when I tried to book  the appointment, and no indication given that there would be such a delay.  When I finally went to the appointment, the experience was horrible.  Even at a discount, the manicure wasn’t worth the cost.  As a result, I’m now much less likely to purchase from LivingSocial again.

On the other hand, I’m much more engaged with the interest-specific daily deals.  There are two that I read every time they appear in my inbox: Fab and Pop Market. I’ve purchased six things through Fab in six weeks, and have been very happy with all of my purchases.  Fab bills itself as “daily design”, and I think that I’ve figured out why they’ve gotten so much of my business lately.

While Fab is a daily deal mail, their offers actually last for 3 days.  This means that I can see something on the website and let it simmer.  I’ve missed a couple of things because I didn’t jump fast enough, of course, but it’s been rare.  I don’t feel pressured to make a decision at this very instant, which is another reason that I often let other daily deals pass me by.  I generally consider my purchases (which is why I’m still wibbling over which new Mac to buy), so unless a daily deal is an absolute must-have, I tend to let them go.  With three days, I have more time to consider whether it’s something that I should actually buy, which perversely means that I buy more.  Further, since there’s new items coming up daily, I have an opportunity to reconsider the deals from the previous two days, which has also resulted in additional purchases.

Pop Market is somewhat similar in its execution.  While Fab focuses on design, Pop Market is all about music.  Given my CD-buying habit, Pop Market is perfect for me.  They do a mix of daily deals and week-long deals.  They publish the artists for their daily deals in advance, which gives me some insight as to whether I’ll be interested.  For their week-long deals, they’re centered around a theme (this week: anniversary editions, Sun Records alumni, complete albums collections).  My main complaint is that their deals are often good but not great.  When I’m on the Pop Market page, I’ve also got a browser tab open to Amazon.  Sometimes Pop Market wins, other times Amazon wins.  I can’t rely on Pop Market having the best price, which seems to defeat the purpose.

Additionally, these sites are giving me something now1, whereas Groupon et al are giving me something much later.  In many respects, Groupon is just giving me another item that I have to add to my task list: remember to use the Groupon before it expires.  On Fab and Pop Market, I’m buying an item.  On Groupon, I’m buying an opportunity to purchase a service or item.  My to-do list is long enough without having to worry about using a voucher before it expires.

In all, the user experience of the interest-based daily deal websites has been significantly better than the location-based ones.  I wonder if that will change.

  1. This isn’t actually as true as I want it to be.  Fab.com’s shipping is glacially slow.  Most items arrive a month after the purchase.  This is, by far, my biggest complaint about the service. ↩
user experience

usage data tells you what happened, not why it happened

A blog post that ends with “Microsoft UI has officially entered the realm of self-parody” is going to get quite a lot of mileage.  I lost count of how many times I saw it go by on my twitter stream.  Laurie Voss posted a response to Steven Sinofsky’s MSDN blog post about the improvements to Windows Explorer that are coming in Windows 8.

Voss takes a look at the data that Sinofsky posted about the usage of various commands in Windows Explorer, and is less than impressed at how they’ve applied this data to the new design of Windows Explorer.  He has two major complaints.  First, he complains that even by Microsoft’s own data, many of the commands that are elevated in the new Windows Explorer design are ones that aren’t commonly used.  Second, he complains that Microsoft’s data says that the menu bar within the Windows Explorer is very infrequently used, so what’s the point of doing it at all?

Both of these complaints are a very common misuse of usage data.  Usage data only tells you what happened in the past.  It doesn’t tell you why it happened, nor does it tell you what will happen in the future.  Furthermore, usage data can often not be broken down very far, so we don’t know what types of users and usages it represents.  I’ve written about this before in my blog post about the usage fallacy:

Usage data is directional. It doesn’t tell you what action to take, it tells you that there might be an action to take.

This certainly applies here.  For example, Voss is upset that few users use the menu bar, instead using contextual menus.  Do we know why they don’t use the menu bar?  Are there commands that they’re more likely to use in the menu bar?  Is there a discoverability problem (that is, are there commands that users would like to use but can’t find them)?  None of these are questions that can be answered by usage data.  To answer these questions, you need to use other research methodologies.

The true irony of Voss’s lack of understanding of how to appropriately apply usage data is found in another recent blog post of his about the need for statisticians, in which he says the following:

The whole web industry is accumulating vast quantities of data and storing it, magpie-like, as if it has intrinsic value, aided by ever-falling prices for storage. But the data isn’t valuable. It doesn’t mean anything until somebody who knows what they’re doing looks at it, sifts through it, and produces a tool that lets others use it to draw valid and useful conclusions.

He’s right: data isn’t valuable until someone who knows what they’re doing looks at it and helps draw valid and useful conclusions.  It’s always amusing when bloggers officially enter the realm of self-parody.

Nadyne

being female online is being a target online

9 Comments

I’ve been online since 1993.  Through it all, I’ve used my real name, and I’ve never tried to hide my gender.  I’ve taken part in various online communities, mostly geek and music.  One of my earliest lessons in being online is that being female makes you a target for threats.  I’ve received rape threats, most notably after I’ve mentioned that I financially support RAINN.  I’ve received death threats.  The threat that bothered me the most was a threat from a rather unhinged music fan against my beloved cat, since said music fan lived in the same city that I did and so I felt like there was a chance of him actually trying to do something.

Threats are the reason why I moderate comments here.  When I first started blogging, I didn’t bother.  Spam was my only concern, when I first started blogging, and modern spam filters do a reasonable job of managing it.  I started moderating comments when I got one particular commenter who saw it as his mission to threaten to rape me.  At first, moderation didn’t change his comments, but it did keep an echo chamber from forming.  Finally, he got bored, and stopped posting.  Maybe he’s still reading, I don’t know, but he’s not making the threats any longer.  I still get threats on occasion, most often if I’ve seen an uptick in traffic because I’ve gotten linked elsewhere.

Sadly, it came as no surprise to me today when I saw another female blogger say that she’s gone into hiding as a result of death threats:

I got a death threat.

I got another death threat.

And then I got one telling me that if I went to a popular blogger’s conference in November, they would find me, they would kill me, and they would kill everyone around me.

[…]

As a reasonably prominent female face on the internet, I always knew the day would come when it would get very, very ugly.

I don’t know why being female online sparks this kind of response.  I don’t know why there’s an echo chamber that creates an ever-escalating environment.

And I don’t really know what to do about it, other than continue to be myself.  Yes, I’m female.  Yes, I’m a software engineer.  Yes, I have opinions, and I share them unapologetically.

user experience

every minute you talk is a minute your user isn’t giving you data

The summer intern program at VMware is robust.  My user experience team gets a few summer interns every year, and sets them loose on several forward-thinking projects.  They’ve got 10-12 weeks to go through the whole design process for their project.

This summer, I worked with one of our interns towards the end of her project.  She wanted to collect some user feedback about her work, and wasn’t sure how to go about it.  Her mentor suggested that she work with me to see where I could help her collect this feedback. After we had decided on our methodology, I assigned her the task of creating a walkthrough of her design to present to our users.

Together, we iterated on her walkthrough several times.  In her first attempt, she filled the entire time available to us by presenting her ideas and designs, not leaving enough time to gather feedback.  I gave her advice on what kind of research questions were appropriate and what kind of feedback she could get, and helped her hone her walkthrough so that she would get feedback that was directed and actionable.  I also gave her advice on her presentation style, how to answer questions, and what to do when she felt stuck.  Her final walkthrough was tight and focused, and she did a great job with it.

At the end of her internship, she gave a presentation to the whole user experience team about her project.  She talked about how she started, how she iterated on her design, how she collected feedback from stakeholders, where our research fit into her design.  At the end, she shared some quotes from team members which will stick with her.  One of them was from me:

Every minute you talk is a minute that your user isn’t giving you data.

(At that point, another designer leaned over to me and said, “yeah, that’s something you’d say”.  How well they know me!)

One of the things that we talked about as she worked on her walkthrough was brevity.  When you’re collecting data from your user, your goal is to collect as much actionable data as possible in the time that you have available.  This is just as true for a 5-minute survey as it is for a 2-hour usability study.  If you are talking, your user isn’t.  You can hear yourself talk any time, whereas you only have a very short window of opportunity with your user.  Therefore, when you’re collecting user data, you have to carefully craft what you present to them to maximize the data that they are able to give you.

Her first attempt at her walkthrough required a full hour for her to go through it.  What she actually presented to our users took about 10 minutes.  The rest of the time, we were able to ask questions and probe our users for feedback.  It took a lot of hard work to hone the focus of the walkthrough down to something that was so much shorter than the original, but it ultimately paid off.  We learned a lot, and were able to identify some important design improvements that could be made.

If I had to pick one message that a user experience intern would take away from working with me as we conducted research, I think this is a pretty good one.  I was very pleased to see her call that out as one of her lessons learned here at VMware.

user experience

what they don’t tell you about conducting research

Several times lately, I’ve been approached by a couple of the newer designers on my team who are conducting research for VMware for the first time.  In our conversations, it’s become clear that there’s a difference between what you learn about conducting research in school and what you learn when you’re actually out in the field conducting research.

Finding participants for research is always harder than you think it will be, and it takes longer than you think it will.  Not only do you have to get the right participant to take part in the study, scheduling them is difficult because you’ve got to work with your own schedule and theirs too.  When you’re asking others for leads, such as a Program Manager or a Technical Account Manager, you’re going to have to build a relationship with that person and help them understand your research goals to be able to get the right people.

You will always have more research questions than you can answer in the time available.  You’re going to have to prioritize your research questions.  While you’re actually conducting the research, you’re going to find additional research questions, which you might decide should be prioritized higher than the research questions that you’ve already identified.  Be aware that this will happen, and be flexible about it.

You have to get comfortable with silence.  Don’t rush in to fill space.  Let your participant think, let them consider what they’ll do or say next.  But don’t let the silence go on for too long, and don’t forget to prompt a participant who isn’t talking at all.  It’s a fine line to walk, and it takes some practice.

Conducting research isn’t just about having a solid research plan and protocol.  There’s a lot of mechanics surrounding conducting the research that can make or break your research.  The best research protocol won’t give you meaningful results if you can’t talk to the right people or if you have too many questions for the time available.

Nadyne, software

the road away from Quicken, part 4: using iBank

2 Comments

Now that I’ve got my data imported into iBank, it’s time to actually put it through its paces on my easiest set of financial data (the household accounts).

The UI is a single window.  I’ve never thought myself a fan of Quicken’s windows-everywhere approach, but I’m not sure if I like the single window either.  My household account list has 10 accounts in it, which takes up the majority of the space available.  I don’t want to lose the summary pane, since that’s where the upcoming scheduled transactions are listed, and I very  much want to see them at all times.  I seem to spend a lot of time scrolling through the left navigation pane, and it’s getting on my nerves.

There’s several other UI annoyances.  iBank should win a special award for Most Gratuitous Use of Cover Flow.  I can mostly ignore that, no matter how much it makes me want to snigger.  The main UI looks like a chequebook register, which is natural, but poor use of spacing and fonts make it difficult to read for any length of time.  Poor use of fonts also haunts the reporting function.

Entering data

Entering data manually seems difficult.  While the main window looks like a chequebook register, there’s no blank line.  You’re forced to either use the “new transaction” button in the bottom bar (not the most discoverable location) or cmd-N.  Normally, I like cmd-N for this sort of thing, but it gets annoying if you’ve got a whole account that needs manual entry.  I also don’t like that cmd-N creates a new transaction inline wherever you’ve currently got selected instead of at the top or bottom.  I kept on creating new transactions before I realized where they were located.

From years of Quicken, I’m used to hitting the + key to get the date of a transaction to increase by one.  Actually, I don’t have to hit the shift key, just hitting = results in this.  Muscle memory made me try that, leaving me with = in the date field.  I know that this is a minor detail, but it’s something that I’ve come to rely on, and the absence of this detail is also annoying.

Using scheduled transactions

I started to add in my scheduled transactions.  There are two ways to enter a scheduled transaction.  One is to create an instance of a transaction, and then use a menu selection (or contextual menu selection, if you’re so inclined) to create a scheduled transaction from that.  When using this option, I didn’t see a confirmation that my scheduled transaction had been saved. It took a couple of minutes of looking around to see that it’s in the summary area, in the lower-left corner of the window (you know, the least discoverable area of the screen).  Also, in the left navigation bar, there’s a category called “manage” which has an item “scheduled transactions”.  In either case, the confirmation that the scheduled transaction has been created is pretty subtle, which doesn’t make me feel entirely confident in it.  On the other hand, creating a new scheduled transaction isn’t something that I do frequently, so perhaps this wouldn’t be so annoying after I’ve got everything set up.

The other way to do it is in the list of scheduled transactions.  That’s in the left navigation bar under the “manage” category, which I only found after I’d been trying to create scheduled transactions via the direct manipulation method and wasn’t seeing the confirmation messages.  Adding a scheduled transaction here allows more flexibility than doing so from an existing transaction.  Using an existing transaction assumes that you want it on a monthly basis.  However, there’s thing missing from the scheduled transactions: the ability to have a flexible amount for the scheduled transaction.  For example, my power bill is never the same every month, but it’s always due on the same date.  I’ll have to manually update the transaction amount when I actually make it.  This isn’t a big deal (after all, I have to make that manual step today), but it feels odd to not be able to specify that the amount isn’t fixed.

I haven’t figured out the difference between “transaction templates” and “scheduled transactions”.  Or, rather, I haven’t figured out why both of them exist in the UI.  In fact, I don’t get most of what’s in the left-hand navigation bar.

Reconciling accounts

After I got my new transactions entered, it was time to reconcile.  On my first attempt at reconciling a bank statement, I discovered that I had to reconcile all of my old transactions.  All of my imported information doesn’t have the reconciliation information.  This made reconciling last month’s statement impossible.  On the one account that I tried, I didn’t want to have to go through all six years of statements and do them individually, so I tried to choose all of the old transactions.  I had to click through six bloody years of transactions individually.  And then the numbers didn’t match up, so it was a bloody half-hour of wasted effort.  I deleted the statement, and the account is still unreconciled.

More research told me that I was going to have to go through all of the statements to do this to make sure that all of the individual transactions imported properly.  You can imagine the amount of obscenities that I started spewing at this point.  I’ve only got six years of data on this one, and the idea of having to go through all of my bank statements for that period of time is making my skin crawl.  This might be a dealbreaker for using iBank.

Creating a budget

Although I was pretty upset at the idea of having to go back and reconcile all eleventy billion transactions manually, I also tried to set up a budget.  In doing so, this made me realize that I need to invest time into fixing my categories.  It correctly imported the names of all of my categories, but set them as expenses.  That took a couple of minutes to fix, and then I went through the budget wizard to get started.  Most of it was self-explanatory, until I got to the screen titled “starting cash in envelopes”.  Hunh?  That sent me to the manual to learn that iBank’s budget can use the so-called envelope system of budgeting.

I ended up having to create three budgets to get one that was correct.  The second screen, “select accounts to budget” tripped me up.  It gives a list of all of my accounts, and they’re all checked by default.  The manual is no help, telling me to “[c]heck off the ones you want to include”.  Well, okay, they’re all part of my financial picture, so I want them included, right?  Wrong.  One of my accounts tracks my mortgage.  If I want to be able to include my monthly mortgage payment in my budget, I have to remove my mortgage account from the list of accounts.  Mortgage interest is already included, since it’s got a category, but I can’t capture this payment to my mortgage if I list it as an account to include in my budget.  Conceptually, I can’t grasp this at all.

I set up my budget as an annual budget.  While most of my transactions are monthly, there are some important ones that are biannual or annual.  My car insurance and property tax payments are biannual, for example.  However, this means that the only view that I get of my mortgage is of the year.  This makes it hard to see if I’ve met my budget for, say, my power bill this month.  I can only get the picture of my year-to-date, not this month.

Issues in setting up my budget aside, I don’t like how it shows my budget information.  The visual display of the information is poorly-done.  There’s a progress bar for each individual expense category, which has five states: empty (no transactions), green (on track), yellow (ahead of forecast, but still under budget), red (over budget), and a checkmark (exactly at the budget).  It’s difficult to read a line and see what’s going on.  Overall, viewing my budget and figuring out what’s going on is something that I can’t do in the budget view.  I’d hoped that I could create a report, but there only seem to be four built-in reports that I can’t do anything with.

Conclusion

I’ve spent three weeks using iBank seriously, and I end up with a headache each time I try to use it.  I don’t mean that in an “ugh, this thing sucks” kind of way, I mean that my head physically aches after using it.  Some of it is simply trying to get up and running, some of it is trying to figure out what they’re conceptually doing (the budget thing is entirely perplexing), and some of it is poor visual choices.

After three weeks, I just don’t see how I can use this seriously.  There’s the major hump to get over of having to reconcile all of my accounts for all of their transactions.  I don’t even want to imagine what that time investment is like.  Additionally, the childish budget design and difficulty of using scheduled transactions make it unsuitable for my needs in planning.

This is part four of an ongoing series about trying to move away from Quicken 2007 for Mac.  The previous three parts of the series are here:

Nadyne, software

the road away from Quicken, part three: importing data into iBank

This is part three of an ongoing series about trying to move away from Quicken 2007 for Mac.  The previous two parts of the series are here:

As I’ve mentioned before, I maintain two separate Quicken accounts: one with my personal data, and one with my household data.  The household data is the smaller dataset of the two, so I’m using that as my test case for importing into potential Quicken replacements.

iBank importer: choose file to importiBank has a 30-day free trial available on its website, so I downloaded and installed it.  On first launch, it detected that I didn’t already have a data file, and gave me the opportunity to import my data file from Quicken.  I didn’t expect that I’d have explicitly export my Quicken data into a QIF.  Exporting to a QIF took a few minutes, and resulted in a 360-KB QIF file.  Then I dragged the file into the iBank importer, which (oddly) gave me a teensy preview of the file.  I’m not quite sure what the point of this preview is.

After selecting my QIF for import, I was then prompted to save my document.  I was surprised that I was expected to decide the filename here, and not entirely sure what “document” means.  I’ve been using Quicken since the dawn of time or thereabouts, so I’m not used to thinking about how it stores its data.  I’m not sure what lies behind the design decision to force the user to think about both the name and location of their datafile.  Shouldn’t they at least provide a reasonable default?

After going through the last question of currency, it started importing.  iBank showed a determinate progress indicator, but didn’t give me an estimate of how long it would take.  My file took about a minute to import.

After importing, I was given a list of my accounts and account types, and asked to choose the most appropriate type for each account listed below.  It noted that iBank supports more account types than QIF1, but didn’t give any guidance as to why I might want to select a different type of account.  For example, my mortgage imported as the “liability” account type, and iBank has both “liability” and “loan”.  Why would I choose one over the other?  Can I change this later?  This drove me to check the “Quick Start Guide”, which didn’t help, so I had to go grab the full manual.  There, I learned that a loan has a payment schedule and a liability doesn’t.  So I changed the account type for my mortgage to loan, and also changed a couple of accounts from chequing to savings, and left the others unchanged.

At this point, I was out of time, so I quit iBank for the evening.  Overnight, I realized that my account data isn’t the only data that I’ve got in Quicken, and wrote that blog post.  As I was writing it, I opened iBank quickly to make sure that I had a couple of details right, and was surprised to see that iBank prompted me with the “Welcome to iBank” screen that I got on my first run.  On doing some research, I discovered that it’s a known bug with a workaround.  Additionally, I noticed that iBank wasn’t saving my window size and location information.  That was pretty annoying.  The workaround that seems to have resolved the document-prompt issue also seems to have addressed this issue.   While I’m glad that it’s a known issue with a workaround, it undermines my confidence in the application to hit such a bug first thing out of the gate.

My confidence is shaken, but still I’ll soldier on.  Now that I’ve gotten my transaction data into iBank, the next step is to try to actually use it.  Wish me luck …

  1. But not Quicken? I’m not sure what it means here, or why it would draw a distinction between Quicken and QIF. ↩

a Macintosh girl in a Microsoft world

© 2010-2024 go ahead, mac my day All Rights Reserved