On Automated vs. Human Testing

A lot of people are pointing to Joel Spolsky’s Talk at Yale: Part 1 of 3

and it’s a great read. If you read it further, I guess closer to the end of Part 1, he makes some very valuable comments on automated testing and the need for human intervention, using Vista as an example:

Speaking of the old approach to testing – “they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.”

And because of the reliance on automated scripts – “And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. “

“nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.”

Now that sounds a little harsh but it’s also very true. No one will argue (ok, I’m sure some will) that UAC is a better way for managing security, or that putting all the but the reasons that Joel mentions above are precisely the reasons why Chris went off on his rants about the Vista UI last year. Now Ed did counter with some great points but many did agree and the result (6 months after the release of Vista) was still the same.

I’m all for automated testing and test suites, etc – but let’s remember that software is only useful if people can use it. And the people who write it aren’t always the ones who find the bugs. (great post by Goran Zidar on that)

I often think of this when I wonder why we haven’t done more implementing new VFP 9 technology in various products (including Foxfire!). After all, many of the features are completely awesome to put in. But the user experience has to count for something. And while putting a variety of code into the Comments field is one way and while sure, you can build a dialog with a bunch of checkboxes on it – I don’t know if you come away making an interface easier to use. Now, most of my readers here are developers – so we want the cool features and the UI on these dialogs certainly works the way most of us would expect it. But I’m taking the leap here from a development IDE to an end user product. Just as a user doesn’t always get all the different format options that can be dropped into the Picture box, nor do I think they will get all of the other features. So the user experience has to be considered priority and that is not something that can happen properly with automated testing.

I’m doing some work with outsourced resources from Russian but I’ve also seen the same results with local contractors. Early on, I have learned that if I want a dialog to say something, I have to spell it out completely – as in word for word, spelling out each dialog. If I don’t, I know others will come back and clobber me for it.

Likewise with good user interfaces, it’s something everyone needs to struggle with. What is the easiest way for users to discover a feature? (some might argue that users shouldn’t have to discover a feature but in this case, I equate discover to inductive user interface). Now some don’t get IUI and use obviously bad examples to describe its shortcomings. But that’s not the point.

It’s all about consistency and while I’m sure there are some automated tests that ensure buttons are lined up perfectly and the right fonts are used, they still don’t take the “human impact” into consideration. I don’t think any automated test would have shown that Apple’s UI decisions would be as impactful as they have been in the software world.

Those that are looking for automated tests to do everything should look into good usability tests. Morae‘s a useful tool here but sometimes, it’s simply enough to set up a Snag It  or a GoToMeeting and just record what people are going through. 

Blogged with Flock

Update: Argh! And just as I posted this from Flock, I was presented with another UI inconsistency. WHO was it that decided that any open-source or non-MS tool should reverse the order of the buttons in a dialog so that the most commonly used option was on the furthest RIGHT instead of the Cancel?

2 thoughts on “On Automated vs. Human Testing”

  1. Andrew – I’d argue that programmers/developers shouldn’t be designing UI at all. We know too much, we like computers too much, and we don’t see the world the way most of our users do. UI should be designed by designers.

  2. But how realistic is that?

    While it would be nice for a programming team to be made up of:

    a) a designer
    b) a programmer/developer
    c) a documenter

    That could add a fairly big overhead to a project.

    Sounds like a good discussion for a FoxShow – you up for it?

Comments are closed.