Program HITs with the Mechanical Turk API

From CrowdsourcingSeminar
Jump to: navigation, search

Overview

  • Amazon Mechanical Turk is the canonical example of a microwork marketplace. In this assignment, you'll learn how to use any of the available Mechanical Turk APIs to post work to Amazon and accept or reject worker results.
  • Programmatically post HITs to MTurk; retrieve results; accept or reject work.
  • Post at least $1 worth of tasks programmatically and gather data.
  • Post at least two different types of tasks, either using provided templates, or writing your own:
    • One task that is hosted on the MTurk server.
    • One task that is hosted on your own web server (Google App engine, the EECS server, an Amazon EC2 instance, or another server of your choice. Contact the course staff if you absolutely cannot find an external spot to host HTML).

Collecting responses takes time. Starting on Monday morning will likely be too late. Moreover, the API is often complex and difficult to understand immediately. Give yourself plenty of time to learn how it works.

Resources

There are several APIs available to post work to Mechanical Turk. Select the one that's most appropriate for your intended project application or one that you'd like to learn more about. All those posting to Turk should take a look at the developer's guide here to get a sense of what's going on. There are a number of concepts specific to Mechanical Turk that are somewhat byzantine; it's worthwhile to learn them.

The Amazon web services API lets you post work to Turk via REST and SOAP (in other words, using POST requests; in other words, by encoding requests into URLs and visiting them). You can use this API directly via your favorite language. If you go this route, you'll need signature generation code to authenticate your requests to Amazon. The main reason to go this route is if you want to build your own interface to Turk via a language not supported below.

Here are some community-provided tools for interfacing with Turk, including a set of command-line tools for those on Linux/OS X systems

TurKit was written by Greg Little et al at MIT and has been used in several papers. There's an online version; it's not hard to use, and it lets you write programs that invoke Turk as a single function. There are drawbacks; it's not exactly appropriate for dynamic programs, though, since it halts every time the program makes a request to Turk, and can't execute multiple threads of work at once.

QuikTurKit is an API written by Jeff Bigham on top of TurKit to support real-time requests. The course staff has not tried it, but we are curious to see the applications you can develop using this system.

Amazon offers four official APIs in Ruby, .NET, Perl, and Java.

There's also a Python API (Boto) that's not officially supported by Amazon but which has seen some good use.

There is a PHP Turk API written by Matt Can and Anand Kulkarni; ask them for it. Matt and Anand are around to answer any questions about it. The 2008 PHP API on Sourceforge is out of date and no longer works out of the box, but there is a new unverified PHP API up here.

Houdini is a beta requester API for Turk; the course staff haven't tried it, so we're happy to hear your evaluations of it.

Crowdflower offers an API that allows simplified access to MTurk, Zynga, and several other platforms. It requires a few hours for your access to be approved, but it's been reported as considerably easier to use.

Questions and Answers

If you have any questions about interacting with Turk, ask them here (edit this section and add a new subsection), or feel free to answer someone else's.

What to Submit

  • One to two paragraphs about your experience: what kind of tasks did you post? what was your experience using the MTurk API? What was easy, what was cumbersome? Did this assignment spark any ideas for research projects?


Submission Instructions

You will submit your assignment on this wiki.

Create a Wiki Page for this assignment

Create a wiki page called Assignment2-FirstNameLastName and link to it from your personal account page. Replace FirstName and LastName with your real name. Then click on the link and enter the information about your assignment.

Describe your project plan on the wiki

On the page you just created, write up your experience as described above.

Add Link to Your Finished Assignment

One you are finished editing the page, add a link to it at the bottom of the page with your full name as the link text. The wiki syntax will look like this: *[[Assignment2-FirstNameLastName|Firstname Lastname]]. Hit the edit button for the last section to see how I created this link for me.

Links to Finished Assignments

Add your submission below this line.