Pilot Usability Study
Due: before class on April 24th
- 1 Overview
- 2 Prototype
- 3 Participants
- 4 Benchmark Tasks
- 5 Measures and Observations
- 6 Procedure
- 7 Results
- 8 Write-Up
- 9 Submission
The goal of this assignment is to learn how to perform a simple usability test and to incorporate the results of the test into design changes in your prototype. In the real world, this kind of "pilot" study would be used to redesign your experiment before running the study with a larger pool of participants.
You will be performing this test using the latest version of your interactive prototype. As you are working on the pilot study you should also be working to add more functionality to your prototype based on the feedback you received in your interactive prototype presentation. You should include a brief bulleted list in your study write-up of any changes made to your prototype.
You will need to find three unique participants (i.e., volunteers who are not in this class and who aren't your friends) to work through your benchmark tasks. Also remember that participation must be voluntary. You should get the participants to sign an informed consent form and obtain other demographic information (e.g., age, sex, education level, major, experience with your type of tasks and application, etc.) Do not re-use subjects who participated in your lo-fi tests.
Your test will use three tasks. They should include 1 easy task, 1 medium task, and 1 difficult task. These tasks should give good coverage of your interface; if they don't then this is a good chance to redesign your tasks. These tasks do not need to be the same as the tasks in your lo-fi prototype study.
Measures and Observations
Although it will be hard to get statistically significant bottom-line data with only three participants and a rough prototype, you should measure some important dependent variables to get a feel for how this is done (i.e., task time, # of errors, etc.).
You should concentrate on process data. For example, you should instruct your participant to think aloud. You should make a log of critical incidents (both positive and negative events). For example, the user might make a mistake and you notice it or they might see something they like and say "cool". Set up a clock that only the observers can see (one or more of you should observe), and write down a log containing the time and what happened at that time when a critical incident occurred.
If you happen to have access to a video camera, it is fine to use it (but make sure your subjects consent) -- note the time that you start taping so that you can find your critical incidents later on tape. You may wish to use an audio recorder if you don't have a video camera.
You will give the participant a short demo of the system. Do not show them exactly how to perform your tasks. Just show how the system works in general and give an example of something specific that is different enough from your benchmark tasks. You should write-up a script of your demo and follow the same script with each participant.
The participant will then be given task directions for the first task that tells them what they are trying to achieve, not how to do it. When they are finished, you will give them the directions for the next task and so on. Each participant will perform all 3 tasks. You will want to keep the data separate for each task and participant.
Your participants should test your application on a hardware device, not in the emulator.
You must report your results (values of dependent variables, summary statistics, and summaries of the process data) and in the "Discussion" section you should draw some conclusions with respect to your interface prototype. You should also say how your system should change if those results hold with a larger user population. This should be the most important part of the write-up. We want to understand how you would fix your system as a result of what you observed.
Your group should turn in one paper (before lecture on the due date) and on the Wiki. Your write-up should follow this outline with separate sections for the top-level items (number of pages per section are approximate). It should be at most 5 pages, plus appendices and sketches that describe what you did. List the responsibilities of each group member.
Introduction (5 points)
- Introduce the system being evaluated and state the purpose and rationale of the test (1 paragraph)
Implementation and Improvements(15 points)
- Bulletted list of feature improvements since the interactive prototype (1/2 page at most). The descriptions do not need to be detailed.
Method (10 points)
- Participants (who -- demographics -- and how were they selected) (1 paragraph)
- Apparatus (describe the equipment you used and where) (1 paragraph)
- Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
- Procedure (1 paragraph) describe what you did and how
Test Measures (5 points)
- Describe what you measured and why (bullet points encouraged)
Results and Discussion (25 points)
- Results of the tests (1/2 page)
- What you learned from the user study (1/2 page). Document any changes that you plan to make in your prototype as a result of the study.
Appendices (5 points)
- Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
- Raw data (i.e., entire merged critical incident logs)
Hand in Printout in Class
Print your assignment and hand it in at the begining of class on April 24th..
Add Link to Your Group's Page
Edit your group's page to add a link to a new wiki page for this assignment. The wiki syntax should look like this:
[[PilotStudy-Group:ExampleGroup-FirstNameLastName|Pilot Study (FirstName LastName)]]
Again replace ExampleGroup with your group's name and FirstNameLastName with your name. Look at Group:ExampleGroup for an example. Then click on the link and enter the information about your assignment. Be sure to clearly address everything mentioned in the writing guidelines above. Writeups for all group members should be posted here.
Add Link to Your Finished Assignment
One you are finished editing the page, add a link to it here with your group name as the title link. The wiki syntax will look like this: *[[PilotStudy-Group:ExampleGroup|Group:ExampleGroup]]. Hit the edit button for this section to see how I created the link for the ExampleGroup.