HW 4 - Evaluation

From CS260 Fall 2011
Jump to: navigation, search


Revisit the interaction technique you implemented for either HW1 or HW2. Run a small user study (either in person or online, through Mechanical Turk), analyze and write up the results.

Note: this homework has a shorter deadline (next week Monday); you can work either individually or in pairs.


This homework asks you to carry out a small evaluation of either:

  • The Bubble Cursor, the interaction technique you implemented for HW1. The bubble cursor should be evaluated online (we'll subtract points for a HW1 evaluation conducted in person).
  • The Gesture Menu, which you implemented for HW2. The menu should be evaluated in person.

The Bubble cursor evaluation will take more effort as it requires you to deploy your task online. Collect simple survey data on user preferences for either option.

Option 1: Bubble Cursor

Review the evaluation conducted in the the original paper: Grossman, T. and Balakrishnan, R. 2005. The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area. In Proceedings of the CHI 2005, p. 281-290.

The authors conducted two evaluation procedures: one on single-dimension alternating pointing (i.e., the classic Fitts setup), and a second on a more realistic 2D pointing task with distractors. You will gather performance data on pointing using this second setup (as developed in HW 1; do not worry about "object pointing" as described in the paper).

At a minimum, you should vary the following independent variables:

  • Cursor type (Bubble vs Normal)
  • Movement amplitude (distance between targets - at least 2 values)
  • Target Width (at least 2 values) and/or Effective Target Width (at least 2 values) (your pick)

Collect Data about the following dependent variables:

  • Movement Time
  • Error Rate

Option 2: Gesture Menus

Conduct a study that compares menu selection performance At a minimum, you should vary the following independent variables:

  • Gesture control scheme:
    • If you implemented two different gesture modes, compare them to each other
    • If you implemented only one gesture mode, compare it to mouse or game controller input.
  • Menu depth (how many levels down in the hierarchy the target is - at least 2 values)

Collect Data about the following dependent variables:

  • Task Completion Time
  • Error Rate (user selects incorrect menu item)
  • Misses (users perform selection gesture but system does not recognize it)

Qualitative Data

For both options, also gather some qualitative data. At a minimum, collect:

  • Likert scale data about user preference for the technique
  • An open-ended question that asks the users to reflect on the relative advantages and disadvantages of the two techniques you compared.

Recruiting Participants

You can either perform this study in person (HW2) or online (HW1). To conduct the study online, crowdsource the study on Mechanical Turk.

If you conduct the study in person, recruit at least 5 users. You may not count yourself or your group partner as a user. Asking other CS260 students to participate in your study (in exchange for your own participation in their study) is encouraged.

A within-subjects design (where each user goes through multiple different combinations of independent variables) is fine.

To conduct the study online, you will have to export your application in a way that can be accessed from a web browser (e.g., using Silverlight). There are many free web hosts available, but you can also contact cs260@imail if you do not have access to a server to host these files. You will have to figure out how to get users to submit their experimental data collected on the client side back to your web server. You will be able to recruit a larger number of users, but might have to break tasks into smaller subsets to keep remote users motivated to complete your tasks. Leave yourself enough time to experiment with different approaches - getting good data out of Mechanical Turk is possible, but subtle design decisions can have large unforeseen impacts.

Analyzing Your Data

Once you have collected your data, produce appropriate graphs and descriptive statistics that compare movement time and error rate for the two cursor types or two menu types. Also use the appropriate statistical tests to report inferential statistics: determine whether the observed differences are significant (i.e., whether we can expect them to generalize from your sample of users to the larger population).

We suggest you use R, an open source statistical analysis environment for this purpose. See our hints on Getting Started With R.

Your writeup should report both the results of your tests, and include appropriate graphs that depict these results. A common visualization of group differences is a bar graph with error bars that show 95% confidence intervals.

For background on appropriate tests, consult David Martin, Doing Psychology Experiments:

Submission Instructions

You will submit your assignment on this wiki.

Create a Wiki Page for this assignment

Begin by creating a new wiki page for this assignment. Go to your user page that you created when you made your account. You can get to it by typing the following URL into your browser:


Replace FirstName and LastName with your real first and last names. This will take you to the page you created for yourself when you created your wiki account. If you have trouble accessing this page, please check that you created your wiki account properly.

Edit your user page to add a link to a new wiki page for this assignment. The wiki syntax should look like this:

[[Homework4-FirstNameLastName|Homework 4]]

Again replace FirstName and LastName with your name. Look at my user page for an example. Then click on the link and enter the information about your assignment. You should upload the files described below and describe any extra functionality you implemented and want us to review.

Upload Project

  • Submit both your raw data (as an upload) and your analysis writeup (as a wiki page - below).
  • Create a zip file of your raw experimental data. Rename the zip file to firstname-lastname-hw4.zip (e.g., bjoern-hartmann-hw4.zip)
  • Upload the zip file to the Homework4-FirstNameLastName page you just created:
    • Create a new file link like this: [[File:firstname-lastname-hw4.zip]]
    • Save the page, then click on the File link you just created to upload the zip file.

Describe your analysis on the wiki

  • On the Homework4-FirstNameLastName page you just created, write up your method, results, and discussion
    • describe the experimental setup and your user population - any information that may help another researcher replicate your experiment
    • report summary statistics and appropriate test results
    • show graphs and figures for the most important results (upload as JPG, GIF, or PNG pictures to the wiki)
    • interpret the results in a discussion paragraph.

Add Link to Your Finished Assignment

One you are finished editing the page, add a link to it at the bottom of the page with your full name as the link text. The wiki syntax will look like this: *[[Homework4-FirstNameLastName|FirstName LastName]]. Hit the edit button for the last section to see how I created the link for my name.

Links to Finished Assignments

Add your submission below this line.