HW 3 - Gesture Recognition

From CS294-84: Interactive Device Design
Jump to: navigation, search


  • You will work in pairs.
  • You will learn how to connect and read several simple continuous sensors and an NFC reader.
  • You will write some utility code to visualize sensor readings and recognition code.
  • You will write gesture recognition code that can distinguish between several events extracted from continuous sensor readings.
  • You'll put it all together in a game.


Your pair will receive the following loaner sensors - you will have to return them at the end of the assignment:

  • Adafruit ADXL335 accelerometer
  • Adafruit PN532 NFC Breakout board kit - includes 4050 level shifter and two MiFare tags.

You can also get the following additional loaner sensors if your design makes a good case for them:

  • Interlink Force Sensitive Resistor
  • Flex sensor
  • Potentiometers

Also feel free to build your own continuous sensors as we started to do in class.

Requirements for the utility

  • Write a utility that graphs continuous sensor data from all three axes of your accelerometer.
  • For defined thresholds, visualize whether the incoming sensor data matches the event or not.
  • Optional: give the user interactive control perform signal conditioning on the fly - filter, change thresholds, etc. to help you understand your sensor data.
  • Optional: give the user the ability to define a gesture by recording a performance - and then match against that gesture with DTW or any other recognition algorithms you like.

You can write this utility in any programming language you choose. We can help you if you choose Processing. You can get started with sending sensor data from Arduino to Processing by looking at the following Arduino example: Examples - Communication - Graph.

If you want inspiration what your tool could look like, here's an earlier research project from Bjoern's grad school days: http://www.youtube.com/watch?v=FlvRrptT1kk Obviously, in one week, we don't expect you to produce something as complex. The very end of that video also has some game control examples.

Requirements for the game

  • Game Control: The game should have either:
    • At least one dimension of continuous control and at least one discrete recognized event. Example: Breakout - the continuous dimension is the X-control of the shuttle; the discrete event is firing guns to break blocks (if you've collected the special "firing" power).
    • At least two dimensions of discrete control (e.g., up, left, down, right) that you extract from continuous input. Example: Pacman.
  • Score Keeping: The game should remember players' scores: at the beginning of a game, players tag in with an NFC tag to load a their total score and high score. At the end of the game, the players tag out to write the score onto the tag (alternative, less neat implementation: score gets saved into a database indexed by player's tag ID).
  • You don't have to invent a new game - take an arcade classic. You have to implement the game - but you are allowed to take and extend open source code.
  • The main requirement is that in the end the game is playable and is enjoyable to play with your controller.
  • You're allowed to use simple switches for auxiliary functions, but the point of the assignment is to deal with analog sensor data.

Requirements for Physical construction

  • Learn how to make connecting cables, e.g., with ribbon cable - don't just stick all your components onto a breadboard wave the breadboard through the air.
  • Think about designing some handheld shape that has the accelerometer / other sensors embedded.

Turn in instructions

Create a Wiki Page for this assignment

Begin by creating a new wiki page for this assignment. Go to your user page for one user in your group.You can get to it by typing the following URL into your browser:


Replace FirstName and LastName with your real first and last names. This will take you to the page you created for yourself when you created your wiki account. Log into the wiki, then look for your name on the top line of the page - clicking on the name will take you to your user page, too. If you have trouble accessing this page, please check that you created your wiki account properly.

Edit your user page to add a link to a new wiki page for this assignment. The wiki syntax should look like this:

[[Homework3-FirstNameLastNam1eFirstNameLastName2|Homework 3]]

Replace FirstName and LastName1 with your own name, First and LastName2 with your teammate's. Look at my user page for an example. Then click on the link and enter the information about your assignment. You should upload the files described below and describe any extra functionality you implemented and want us to review.

(Your teammate should also add the link to their page).

Write up your project on this wiki

On the Homework3-FirstNameLastName1FirstNameLastName2 page you just created, write 1-3 paragraphs about your implementation, as before.

  • Include images (How to include images).
  • Reflection: what you learned from this assignment (also let us know if this was too easy or too hard)
  • Record a short video of you tagging in to the game and playing a level. Narrate the video. (How to include video)

One you are finished editing the page, add a link to it at the bottom of the page with your full name as the link text. The wiki syntax will look like this: *[[Homework3-FirstNameLastName1FirstNameLastName2|FirstName LastName1 and FirstName LastName2]]. Hit the edit button for the last section to see how I created the link for my name.

Turn in your code using the Class Revision Control System

Create a directory for your group for HW3:

svn mkdir
berkeleyhci_devicedesign/assignments/hw3/<teamname> -m "creating new group"

(all on one line)

Where <teamname> is a concatentation of your firstnames or usernames, e.g. bjoern-paul.

Then checkout the directory you just created:

svn checkout
berkeleyhci_devicedesign/assignments/hw3/<teamname> hw3

Then add your files with svn add and commit them with svn commit.

Only check in your source files (.ino for Arduino, .pde for Processing) - don't check in any binaries. Feel free to check in images, but keep them <2mb each. Don't check in videos.

If you're new to Subversion, you may want to start with a GUI client like Versions on OS X or TortoiseSVN on Windows.

Links to Finished Assignments

Add your submission below this line.