Individual Programming Assignment 3

From CS160 Spring 2014
Jump to: navigation, search

Due: before class on Mar 31

20 points - Programming

Overview

In this assignment you will learn how to:

  • Build an application using sound as input
  • Use media capture APIs like Android's Camera

You will build a clapper camera.

By now you've probably seen the classic commercial for the Clapper:

The media player is loading... (YouTube link)

While its aesthetics may be rooted in the 1980s, clapping hasn't gone out of style. You will build an Android app that listens for two claps. When it hears two claps, it takes a photo. This is like a self-timer on a camera for a mobile device, except that you get to control it with claps instead of having a fixed timer. Once the picture is taken, your app will give the user the option to send or share it.

You will submit your source code, the executable, screenshots and a narrated video. It is your responsibility to ensure that the executable has all the resources it needs to execute.

Instructions

  • Application Functionality: Your application must
    • recognize when the a user claps twice in a short period of time
    • take a photo when after recognizing a the claps
    • allow the user to send or share the photo after it's been taken (you may use an implicit intent for this)
  • Video
    • Submit a narrated video demo of your application, 60-90 seconds in lengths, that shows and describes the features.

Grading Criteria

Full credit (20 points) will be given if your application compiles, runs, contains the functionality as detailed in the instructions, and your documentation (writeup, photos, video) is complete.

  • 5 Points - Usability
  • 7 Points - App robustly detects a quick series of 2 claps in real time
    • Partial credit for only detecting one clap
    • Partial credit if the app does not appear to be robust-- e.g., it detects claps when you are speaking and not clapping
  • 5 Points - App takes a photo without using a camera implicit intent (a camera intent would require the user to tap a button to take the photo)
  • 3 Points - Ability to share or send photo from the app

Up to 5 extra points will be given if you implement additions that make the application more interesting, more usable, or more aesthetically pleasing. For example, you can implement the following:

  • Detection of different audio characteristics to do different actions in the app (different numbers of claps, vocal pitch recognition, etc.). Feel free to experiment with other components of TarsosDSP!

Notes

Implementation

Sample Solution

The media player is loading... (YouTube link)

Listening for the audio signal

Here are some useful tips on sound programming: http://husk.eecs.berkeley.edu/courses/cs160-sp14/index.php/Sound_Programming

And here is a repository of sample sound-based Android code: https://github.com/srubin/cs160-audio-examples

In order to detect claps, you're going to need to process audio from the microphone. To process the raw audio samples on Android, use the AudioRecord class. Setting this up can be a bit tricky, so we're providing some code to get you started. First, you need to enable audio recording in your app's AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />

You need to set up an AudioRecord object.

static final int SAMPLE_RATE = 8000;
int bufferSize = AudioRecord.getMinBufferSize(SAMPLE_RATE, AudioFormat.CHANNEL_IN_MONO,
        AudioFormat.ENCODING_PCM_16BIT);
buffer = new byte[bufferSize];
recorder = new AudioRecord(MediaRecorder.AudioSource.MIC, SAMPLE_RATE, AudioFormat.CHANNEL_IN_MONO,
        AudioFormat.ENCODING_PCM_16BIT, bufferSize);

You can then spawn a background thread and read the audio data from recorder once you've called recorder.startRecording().

When you read samples from the recorder, you will get an length bufferSize array of bytes. These represent consecutive signals from the recorded audio, which are taken at a sampling rate of SAMPLE_RATE samples per second. Each value represents the amplitude of the signal at that sample time; the array of samples are in the time domain.

Detecting claps

The original clapper detected a clap by running a 2200 Hz to 2800 Hz bandpass filter on the input audio signal (see this story for more details). We are providing some basic signal processing tools that you can use in your project. This library is a stripped down version of TarsosDSP (we've modified it to allow some of the components to work on Android). You could also detect a clap using TarsosDSP's PercussionOnsetDetector, if you tune its parameters appropriately.

Get our modified version of TarsosDSP here (you can download the JAR and add it to your libs folder in your Eclipse project): https://github.com/srubin/TarsosDSP/releases/tag/TarsosDSPAndroid-v2

The original code and documentation for TarsosDSP can be found here: https://github.com/JorenSix/TarsosDSP

The general way to work with any of the classes in TarsosDSP that implement AudioProcessor is to first create an AudioEvent from your audio data, and then call process on the AudioProcessor. Once the event has been processed, you can check the contents of the AudioEvent buffer, or in some cases, listen to particular events/callbacks, like the PercussionOnsetDetector's OnsetHandler and its handleOnset() method.

For example, to set up a low-pass filter for frequencies below 1000 Hz on a buffer of bytes using TarsosDSP:

// Setup:
// (note: this is NOT android.media.AudioFormat)
be.hogent.tarsos.dsp.AudioFormat format = new be.hogent.tarsos.dsp.AudioFormat(SAMPLE_RATE, 16, 1, true, true);
LowPassSP lowpassFilter = new LowPassSP(1000f, SAMPLE_RATE);
 
// Later, when you have the buffer of bytes
AudioEvent audioEvent = new AudioEvent(format, buffer.length());
audioEvent.setFloatBufferWithByteBuffer(buffer);
lowpassFilter.process(audioEvent);
byte[] filteredBuffer = audioEvent.getByteBuffer();

There are many techniques you can use to detect claps, some more basic and some more complex. While we are not grading based on the complexity of the detection algorithm you use, we expect your detection to be reasonably robust (the Clapper itself had problems with loud music and barking dogs).

Taking a photo

You'll need to programatically take a photo with the Android API. See notes from section 5 about how to set up the camera in Android. The docs are useful here, as well: http://developer.android.com/reference/android/hardware/Camera.html

Submission Instructions

Create a Wiki Page for this assignment

Begin by creating a new wiki page for this assignment. Go to your user page. Edit your user page to add a link to a new wiki page for this assignment. The wiki syntax should look like this:

[[ProgrammingAssignment3-FirstName_LastName|Programming Assignment 3]]

Again replace FirstName and LastName with your name. Look at my user page for an example. Then click on the link and enter the information about your assignment.

Writeup and Photos / Videos

Describe (briefly) what design decisions you made and why you made them.

Include screenshots of key screens.

Then record a short (<90 second) video of you using your application. Narrate and explain what you are doing. No video editing is necessary - you can send us a single take. To record your screen, try screencast-o-matic.com, or Quicktime Screen Recording on a Mac.

You should then upload your source as one file (described below) and in the wiki page, describe any extra credit functionality you implemented and want us to review.

Upload Project

  • Your submitted project must include the full source code as well as the executable of the working application and the video.
  • Create a zip file of your project tree (the top-level folder that includes the Eclipse project. Rename the zip file to ipa3.zip
  • Upload the zip file to an EECS instructional machine

scp ipa3.zip cs160-xx@star.cs.berkeley.edu:

  • Once you've uploaded the file, ssh into the machine and submit your assignment:

ssh cs160-xx@star.cs.berkeley.edu

submit ipa3

  • Upload or link your video file to the ProgrammingAssignment2-FirstNameLastName page you just created. Post a link to the video on YouTube (the mediaplayer plugin is having trouble with video files that are not uploaded to YouTube.)
    • YouTube link: Add a link to your submission page like this: <mediaplayer>http://youtube.com/myvideolink</mediaplayer>.

Add Link to Your Finished Assignment

One you are finished editing the page, add a link to it at the bottom of the page with your full name as the link text. The wiki syntax will look like this: *[[ProgrammingAssignment3-FirstName_LastName|FirstName LastName]]. Hit the edit button for the last section to see how I created the link for my name.

Links to Finished Assignments

Add your submission below this line: