There Is No Cat - Following up the followup

Sunday, March 30, 2003

Following up the followup

Dorothea follows up on my post below, as well as one by Shelley Powers, in response to her post about why software doesn’t work. Normally, I would just post this sort of thing in comments, but since Dorothea doesn’t enable comments on her site, I’ll post it here.

She posits that if she ran a company and was offered a choice between disruptive usability experts for a week and an embedded programmer for six months, she would choose the embedded programmer. I disagree, and here’s why.

If I were running the company and you offered me someone for six months, I would turn it down. It would take me a good three months just to train that person to do the job correctly. It would be another three months before said person felt comfortable enough with the job to offer suggestions and enhancements. Now just as my newly-trained worker has reached the point where they can be truly useful to me, you’re going to take him/her back? Hell no! I’d much rather hire someone I think will stay. The disruption caused by losing a trained worker will be much greater than having a couple of researchers here for a week.

Dorothea’s comments about "talking to" people betray a lack of understanding of what such research involves. If you’re thinking of focus groups, Dorothea, that’s not what I’m talking about. The kind of research I’m talking about involves being minimally disruptive, and more observation than conversation, particularly in initial stages. Ideally, it’s like cinéma verité documentary filmmaking (or direct cinema, as the Maysles Brothers called it), where the camera operator eventually blends into the background. When you just "talk to" people about what they do, you get a pretty superficial level of insight. People often misunderstand what they’re actually doing, or have difficulty putting it into words. A trained observer can pick up things that even people with a strong sense of self-evaluation miss.

As for whether employing the embedded programmer/direct experience approach would result in better software, I would expect it would improve upon the cult of the programmer approach Dorothea mentions in her initial post, but then almost anything would be an improvement on that. Whether it would be better than software produced with the assistance of trained researchers, I’m not so sure. I think it would take an unusually empathetic programmer to take advantage of the embedding approach, and while I’ve known some who fit that description, I’ve also known plenty who wouldn’t be capable of the kind of insights Dorothea expects. I’d be more inclined to place my bets on researchers, who specialize in teasing out the sort of information required, as being more likely to produce useful information on a consistent basis.

Posted at 8:31 PM

Comments

A lot could be said about this, but a key thing is how release cycles are done. Too often, the beta tests happen too late in the cycle to really incorporate user feedback in the design, no matter how you get it.

Research in advance may help, but I’d suspect a lot of usability issues are hard to predict.

XTreme Programming tackles this by keeping the "release cycle" very short, something like six weeks. This is, however, hard to do in commercial software where the business end plays so heavily.

But there has to be a way to get real users to touch the product early on -- maybe more prototypes or something.

Posted by markj at 9:30 PM, March 30, 2003 [Link]

Too true that by the time a program gets to beta, too much is set in cement to make a significant difference. As for usability issues being hard to predict, I agree with that as well. But different techniques are useful at different stages. When you’re defining the application and laying out the requirements, contextual inquiry can help you tease out what kind of challenges users face with their current tools and what they need to make their jobs easier. Those insights can be used to define the spec.

Once you’ve defined what an application is supposed to do and how it’s supposed to improve the work lives of your potential customers, you need to design the application. Systems Engineers, Information Architects, or just the programmers, call it different things depending on what your environment is like, but someone has to take a creative leap at this point. Some kinds of testing can help. Card sorting, for example, can help define how an interface is organized. Paper prototypes are cheap and can test various rough interfaces. Better to figure this stuff out before you spend money and time programming.

Finally, you loose the programmers on the project. As in war, no plan ever survives contact with the enemy, but hey, the programmers try, and they even have some good ideas. This is where what people think of as usability testing comes in, whether you do it with the one-way mirrors and such or whether you do guerilla testing in someone’s office. But you need to do this early and often, and the programmers have to be willing to incorporate the feedback.

Iterate, iterate, iterate! Repeat testing until the application does what you think it does and users can use it to do those things.

I’ve never worked in an eXtreme Programming environment, but the way you describe it, it sounds like iteration is the heart of it. That’s the effect that short, rapid release cycles will have, anyway.

Posted by ralph at 11:23 PM, March 30, 2003 [Link]