From CS160 Spring 2013
Jump to: navigation, search

Assignment: Pilot Usability Study

Member Contributions

  • Avneesh Kohli - Server-side programming for user account/recommendations service, some android UI layout cleanup, consent form design, significant portion of written report
  • Erika Delk - Display season and episode numbers, led two user interviews, wrote up Results and Discussion section
  • Oulun (Orion) Zhao - Connected episode detail page with api
  • Matthew Chang - Improved feedback in the UI, fixed favourites not updating, led one user interview


The mobile application that our group has been developing is called Watch; it attempts to overhaul the television watching experience by simplifying the way you find, track and share your favorite TV shows. Among several features we have planned and started developing for this app, the core features include the ability to search for shows, browse amongst currently trending shows, find out more details about a particular show, and to start tracking your favorite shows. Having already built out this core functionality, we’re proceeding with a pilot usability study to get a sense of how users react to the experience we’ve provided through these features. Specifically we’re looking for any usability problems that users which would appear while using a live prototype on a hardware device, that might not have shown up in low-fidelity mockups and designs. Additionally, we’re also looking for any additional suggestions that users might have on how to improve the user experience, given that they’ll now have a tangible prototype to test out.

Implementation and Improvements

Below are some of the main improvements and bug fixes we’ve managed to complete since the original interactive prototype:

  • Users are now able to view details about all episodes of their tracked TV shows
  • We’ve condensed our list of episodes into a list of seasons, which expand to show all episodes within a season
  • We’ve fixed a bug where favoriting a show wouldn’t show up on the tracking list
  • Our search functionality now supports queries that are comprised of multiple words
  • A user account system has been implemented on a server (not yet hooked up to the app)
  • A recommendation system between users has been implemented on our server (UI component has not been added to the app yet)
  • We’ve re-architected our layout code so the layouts should appear normally on most phone and 7” tablets



For this pilot usability study, we selected 3 unique participants that we believe represent a varying and strong portion of our target user base for our mobile application. Our first participant was a 19-year old, male, college student who watches his television shows primarily through various Internet streams on his laptop. We feel that this participant would give us a good indication of how a younger, very tech-savvy user would interact with our app. Our second participant was a 26-year old female who holds a full time job, and is a casual television watcher. This participant would help give us the perspective of a female, who was adequately tech-savvy, and someone who consumed content primarily through the television versus the computer. Our final participant was a 35 year old mother with young children, who managed both her own TV choices as well as those of her children. We felt this participant would give us good information on a user that would be tracking a lot of shows as well as dealing with a wide variety of shows.


The equipment that was used to conduct the pilot usability study was a Galaxy Nexus smartphone running Android 4.1. Since our app does require an internet connection, we ensured that we were connected to the available wireless network at the Starbucks coffee shop where we conducted our studies. We also had the users fill out a consent form to participate in the usability study. For our timing information, we used a stop-watch.


Task 1 (Easy): Find a show called “White Collar” using the search interface and view details about the show.

  • For this task, we verified that the user was able to find the show “White Collar” by typing it into the search bar, and then view the details page by clicking the show’s title in the list of results

Task 2 (Medium): Favorite a show called “Arrow” to start tracking it

  • For this task, we verified that the user was able to find the show “Arrow”, either by browsing or by searching, and then added the show to their favorites list by clicking the details button on the favorites page.

Task 3 (Hard): Find me details about the 2nd episode of the 1st season of Game of Thrones

  • For this task we verified that the user added Game of Thrones to their favorites list either by searching for it or browsing and selecting the “favorite option.” We then checked that the user navigated to the “track” tab, and was able to view the season and episode list by selecting the show name. Then, by expanding the list and selecting the correct season and episode number, they were able to view the episode details page.


Our group went to a Starbucks to try and find a diverse group of people from which we could select to participate in our usability study. We would approach an individual and explain to them the premise of mobile application, and asked them if they would be willing to participate in a usability study to help enhance the experience of the app. If they agreed, we had them fill out a consent form, and gathered further information about their television watching experience to get a sense of what kind of perspectives they would be able to provide us. One person would explain the main features of the application. They would then proceed to have the user go through completing the three tasks described above. Two members of the group observed the user and took notes and measurements, while the last manned a stopwatch. We thanked the user after completing all of the tasks, and compiled our notes.

Test Measures

The following are points of measurement that we captured during our usability tests:

  • The length of time from opening the application to completing of the task, as measured by our timekeeper with a stopwatch.
  • The number of taps required to complete each task, and the difference between the number of taps recorded and the number of taps we expected each task to take. A maximum of 2 taps would be * assessed any scrolling actions taken.
  • The number of clicks on the back button, which would indicate the user making a mistake or being exploratory in the app.
  • The number of clarifying questions the participant asked, and the general amount of confusion they showed.

Results and Discussion

Our tests showed that all of the tasks were completed relatively quickly, taking under two minutes even for the hard task. Furthermore, the number of tasks the participant took was close to the maximum number we estimated, with an average of 5 more taps total than we expected. Most of these extra taps were used switching between tabs in the application, as there seemed to be some confusion about the starting point and the flow of control. The only issue with the back button was after the user had “favorited” show--because clicking the button does not open a new page, the participants seemed to be confused about what actions to take afterwards. Generally, they would use the back button to navigate back to the application’s search page and then try all tabs, until they found the track tab at which point they continued their task. This vagueness regarding the consequences of clicking the “favorite” button and the uncertainty of the flow of control at that point in the chain of execution and was the main source of confusion and questions. Participants asked things like, “now what do i do?”, “where do I go?”, “did this do anything?”, etc. Interestingly, the demographic information of the user had little effect on their ability to navigate the application.

We got a lot of useful information from these interviews. One of the first complaints we got was with regards to the loading speed of the application. Even though it only takes a couple of seconds to query the web API and parse the data, all of the participants seemed impatient and did not like having to wait. The second piece of the feedback we got consistently were questions about the distinction between the “browse” and “search” tabs--it seems that it was unclear to most what the difference was. All commented that episode names would be more useful to see than episode numbers. A couple wanted to know how they would go about deleting a show from their list of favorites. Two explicitly stated that having the favorites button redirect to the track tab would be useful. The interface was deemed intuitive, but not very aesthetically pleasing.

Taking all of this feedback into consideration, we have determined that we will try to make querying and parsing the API faster (possibly through caching). We will also display the episode names instead of the the episode number, and add an explicit “delete from favorites” option on the episode details page. We will also either merge the search and browse functionalities, or try to add more feature to the actions so that the distinction between them is more clear. Furthermore, we will change the app’s flow of control, so that clicking the “favorite” button redirects the user to the tack tab. Lastly, we will try to work more on the UI to provide a unique theme and the general beautification of the application.


Consent form that was used for participants: [http://cl.ly/0E0G2r0d3V2W]

Raw Data

Minimal number of taps needed:

  • Task 1 (easy): 2
  • Task 2 (medium): 3
  • Task 3 (hard): 8
User Age Gender TV Habits Task 1 (taps/time (sec)/back/questions) Task 2 (taps/time (sec)/back/questions) Task 3 (taps/time (sec)/back/questions)
User 1 19 Male Streams shows from internet 2/36/0/2 5/35/0/0 11/90/2/1
User 2 26 Female Casual watcher 3/45/0/1 4/62/0/1 9/101/3/2
User 3 35 Female Watches shows on Netflix with children 4/62/0/3 4/55/0/0 10/96/3/1