The Usability Test
At the end of the last part of our tale, Bruce and I were feeling pretty good about the state of Scheduling in SuperDuper! We’d solved the technical challenges involved, the UI for setting up the times looked good, worked well, and seemed logical.
In short, we were ready to try things out on “real users”.
In a large organization, Usability tests are done in a very formal way (you can see this in the comments for my last scheduling-related post). The user is given a series of tasks and a (typically instrumented) build, and put in a room where they can be extensively observed. Everything they do is logged, video cameras look at different aspects of the process, sometimes even physiological measurements are taken. And the team, either on the other side of a one-way mirror or in a control room away (sometimes far away) from the test, gets to watch.
What you see is always a surprise. No matter how much you plan and design and argue and rework, something always comes up when you usability test. And it’s never pretty.
Well, here at Shirt Pocket, we can’t afford that kind of usability test.
Instead, we select users who are good at approaching things naïvely. Each one is given the build, and a task, and then I watch and listen very carefully — often using an iSight — while they perform the task. And takes notes. And stay very, very quiet. After all, we’re hunting usability rabbits…
Well, the first tester was — of all people — my brother Paul. Paul’s a good choice, because he’s a very typical Macintosh user, and he indicates in lots of ways, which makes his reactions easy to read.
The task in this case was pretty easy — something like “Schedule a backup of all files on your drive at a time and frequency of your choosing”.
Right off the bat, lots of confusion and the very worst question of all: “How the hell do I do that?”
And, once he found the settings in Options, and checked the box, and it didn’t seem to do anything, he thought he was done and then found out he had to manually save settings, and didn’t know why, and… well, it kept going downhill.
Not good. Not good at all.
(In a real usability test, the designers and developers at this point are usually screaming at the user — who can’t hear them — in the remote location, begging them to find the stuff they’d worked so hard on. It’s really very frustrating to have a lot of work that seemed so good fail so completely.)
So, I thanked Paul, hoping his reaction was an anomaly, and moved onto my next tester. Who reacted almost identically. And a third. Same thing.
The fourth, a user who had previously used saved settings extensively, had virtually no problem at all.
75% failure. Many, many points of confusion. Most of the nice touches in the UI — and there were many — were ignored because the major portion of it was fundamentally flawed: it assumed you knew about saved settings and were comfortable with them, and the vast majority of users know nothing about saved settings and don’t want to know anything about them.
An unmitigated disaster. I’d really screwed the pooch. Gah. And now I had to tell Bruce how poorly it went, and fix it…


that’s what I did:
This is a real shot of a recent build of SuperDuper! v2.0, running an actual backup. I’ve broken the backup into different major “Phases”, each of which has a bar. A phase is grey when pending, blue when currently in progress, green and checked if successful, and red with an X if it failed (not shown).
Each phase has a number of steps, and each step changes to show whether it’s pending, current, successful or failed as well. The wording of the step changes, too, following the maxim: going to do it, doing it, done!
Everything follows our ‘no placeholder’ policy, and tells you exactly what drive it’s operating on, and what it’s doing. We’re also giving significantly more information about the process of copying (and will likely provide even more in the final version).
So, in the end, it seems pretty simple, and I think it feels pretty great in use. (The static picture here doesn’t show you the nice compositing Bruce does as the steps complete, nor the way that a step automatically expands when there’s more info to show, but you’ll see all that soon enough.)
What do you think?