From CrowdsourcingSeminar

Contents 1 CS298-52: Crowdsourcing Seminar 2 Publications from this course 3 Format 4 Preliminary Schedule 5 Potential additional topics

CS298-52: Crowdsourcing Seminar

• UC Berkeley, Spring 2011
• Mondays, 3:30-5pm, 380 Soda Hall
• 1-3 units, CCN 27226
• Organizers: Profs. Bjoern Hartmann, Mike Franklin, Tapan Parikh; Graduate Students Anand Kulkarni & Kuang Chen

Crowdsourcing and human computation have rapidly become interesting topics of research in Computer Science. In this reading seminar, we will survey and discuss the latest literature, with the following overarching themes:

• Algorithms, patterns and systems for crowd programming
• Organizational behavior and economic aspects of crowdsourcing
• Participatory sensing & crowdsourcing for development
• Demographics & labor issues of crowdsourcing
• Applications in active learning and AI

If you want to read and discuss papers only, sign up for 1 unit. You may take this course for up to three units if you are also working on a research project related to crowdsourcing as part of the class. Students are encouraged to collaborate on projects and choose projects that can be finished and submitted as papers to UIST 2011, the ACM Symposium on User Interface Software and Technology.

Questions? Email: crowdsourcing-seminar-staff@lists.eecs.berkeley.edu

Publications from this course

• What’s the Right Price? Pricing Tasks for Finishing on Time. Siamak Faridani, Björn Hartmann, and Panos Ipeirotis. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Full Paper (6 pages)
• Pricing Mechanisms for Online Labor Markets. Yaron Singer and Manas Mittal. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Full Paper (6 pages)
• MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Prayag Narula, Philipp Gutheim, David Rolnitzky, Anand Kulkarni, Björn Hartmann. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Poster
• Turkomatic: Automatic, Recursive Task and Workflow Design for Mechanical Turk. Anand Kulkarni, Matthew Can, Björn Hartmann. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Poster

Format

For each class period, students will submit short reading responses online by midnight on the day before class. Reading responses are submitted directly to this wiki.

In addition to commentaries, students will be asked to lead one class discussion. The discussant should read all student commentaries before class and integrate them into the discussion.

A small number of assignments will ensure students are familiar with current crowdsourcing platforms and programming tools. For students working on research projects, additional assignments are designed to keep you on track towards a successful paper submission.

Preliminary Schedule

Jan 24: Introduction [ Slides ], Project Opportunities

In-class Reading 1: Crowd Control. Leah Hoffman, Crowd Control, Communications of the ACM Vol. 52 No. 3, Pages 16-17

In-class Reading 2: Five Rules of the New Labor Pool. Jeff Howe, Wired 14.06

Assignment due by Friday Jan 28, 5pm: Create Wiki Account

Assignment due by Friday Jan 28, 5pm: Submit Class Petition

Jan 28: No class, but items due

Due: Create Wiki Account

Due: Submit Class Petition

Jan 31: Surveys and Taxonomies (Submit Response | Bjoern's Slides | Discussion Slides | Discussants: Anand Kulkarni, Siamak Faridani)

Paper 1: Human Computation: A Survey and Taxonomy of a Growing Field. Alexander J. Quinn, Benjamin B. Bederson. CHI 2011.

Paper 2: Analyzing the Mechanical Turk Marketplace. Panagiotis G. Ipeirotis, ACM XRDS, December 2010.

Assignment due before class on Feb 7: Create and Perform HITs on Mechanical Turk

Assignment for project students, due before class on Feb 7: Project Abstract

Feb 7: Applications in HCI ( Discussion Slides | Discussant: Sally Ahn, Pablo Paredes)

Paper 1: VizWiz: nearly real-time answers to visual questions. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. UIST 2010.

Paper 2: Soylent: a word processor with a crowd inside. Michael S. Bernstein , Greg Little , Robert C. Miller , Björn Hartmann , Mark S. Ackerman , David R. Karger , David Crowell , Katrina Panovich. UIST 2010.

Due: Create and Perform HITs on Mechanical Turk

Due: for project students: Project Abstract

Assignment due before class on Feb 28: Program HITs with the Mechanical Turk API

Assignment for project students, due before class on Feb 14: Project Abstract Part 2

Feb 14: Systems for Crowdsourcing (Discussant: Reynold Xin, Philipp Gutheim)

Paper 1: CrowdDB: Answering Impossible Queries. Michael Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, Reynold Xin

Paper 2: CrowdSearch: Exploiting Crowds for Accurate Real-time Image Search on Mobile Phones. Tingxin Yan, Vikas Kumar, Deepak Ganesan, Mobicom 2010

Due for project students, due before class on Feb 14: Project Abstract Part 2

Feb 16: No Class - Jeff Bigham (VizWiz) talk at 12pm, BiD Lab 354/360 HMMB

Feb 21: No class - President's Day"

Feb 28: Annotations and Participatory Sensing (Discussants: Chulki Lee, Wesley Willett)

Paper 1: Utility data annotation with Amazon Mechanical Turk. Alexander Sorokin, David Forsyth. First IEEE Workshop on Internet Vision at CVPR 2008.

Paper 2: Participatory Sensing: A citizen-powered approach to illuminating the patterns that shape our world Jeffrey Goldman et al. 2009.

Due: Program HITs with the Mechanical Turk API

Assignment: Prototype I

Mar 7: Crowd Programming Tools (Discussants: Manas Mittal, Kristal Curtis)

Paper 1: TurKit: Human Computation Algorithms on Mechanical Turk. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. UIST 2010.

Paper 2: Generalized task markets for human and machine computation. Dafna Shahaf and Eric Horvitz. 2010.

Mar 14: Task and Workflow Design (Discussant: Beth Trushkowsky, Kurtis Heimerl)

Paper 1: Exploring Iterative and Parallel Human Computation Processes. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. HCOMP 2010.

In-class discussion on taxonomies.

Due: Prototype I

Mar 21: No class - Spring Break

Mar 28: Experiments (Discussants: Nicholas Kong, Beth Trushkowsky)

Paper 0: (holdover from 3/14): Exploring Iterative and Parallel Human Computation Processes. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. HCOMP 2010.

Paper 1: The Online Laboratory: Conducting Experiments in a Real Labor Market. Horton, John J., Rand, David G. and Zeckhauser, Richard J. SSRN, 2010.

Paper 2: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. Jeffrey Heer , Michael Bostock. CHI 2010.

Mar 29: No Class - Leila Janah, CEO Samasource, talk at 4pm, B100 Blum Hall Creating Sustainable Livelihoods in Poor Regions

Apr 4: Economics (Discussants: David Rolnitzky, Travis Yoo)

Paper 1: Financial Incentives and the "Performance of Crowds". Mason, W., & Watts, D. J. HCOMP 2009.

Paper 2: Algorithmic Wage Negotiations: Applications to Paid Crowdsourcing. John Horton and Richard Zeckhauser. CrowdConf 2010.

Apr 8: No class, but assignment due for project students

Due for project students: Draft of Paper (Either UIST or HCOMP format)

Apr 11: Discussion of UIST Drafts, Taxonomy (Kurtis Heimerl leads taxonomy discussion)

Apr 18: Applications in International Development (Discussants: Prayag Narula, Ariel Chait)

Paper 1: Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India. Shashank Khanna, Aishwarya Ratan, James Davis, WIlliam Theis. ACM DEV 2010.

Paper 2: The Development Potential of the Virtual Economy. Vili Lehdonvirta, Helsinki Institute for Information Technology. (Read Exec Summary + Ch 2-5)

Apr 22: No Class - UIST Deadline

Apr 25: Last class Labor Issues and Incentives (Discussants: Yaron Singer, Kuang Chen; Guest Speaker: Aaron Shaw)

Paper 1: Designing Incentives for Inexpert Human Raters. Aaron Shaw, John Horton, Daniel Chen. CSCW 2011.

Paper 2: Sellers' problems in human computation markets. M. Six Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. HCOMP 2010.

Apr 29: No Class - HCOMP Deadline

Due: Final Paper (Either UIST or HCOMP format)

Potential additional topics

Participatory sensing

Participatory Sensing: A citizen-powered approach to illuminating the patterns that shape our world Jeffrey Goldman et al. 2009.

Traffic Sensing through mobile phones, e.g. Berkeley Mobile Millenium Project.

Games with a Purpose

Social Search and Q&A

Aardvark WWW'10

Stack Overflow, Lada Adamic's Q&A work.

Advanced crowdsourcing interfaces

CrowdFlow: Integrating Machine Learning with. Mechanical Turk for Speed-Cost-Quality Flexibility Quinn and Bederson, 2010.

Evaluating crowdsourcing algorithms

Towards a Theory of AI Completeness Shahaf and Amir, Commonsense 2007.

Databases

CrowdSearch

Consult the User's Guide for information on using the wiki software.