CS298-52: Crowdsourcing Seminar
- UC Berkeley, Spring 2011
- Mondays, 3:30-5pm, 380 Soda Hall
- 1-3 units, CCN 27226
- Organizers: Profs. Bjoern Hartmann, Mike Franklin, Tapan Parikh; Graduate Students Anand Kulkarni & Kuang Chen
Crowdsourcing and human computation have rapidly become interesting topics of research in Computer Science. In this reading seminar, we will survey and discuss the latest literature, with the following overarching themes:
- Algorithms, patterns and systems for crowd programming
- Organizational behavior and economic aspects of crowdsourcing
- Participatory sensing & crowdsourcing for development
- Demographics & labor issues of crowdsourcing
- Applications in active learning and AI
If you want to read and discuss papers only, sign up for 1 unit. You may take this course for up to three units if you are also working on a research project related to crowdsourcing as part of the class. Students are encouraged to collaborate on projects and choose projects that can be finished and submitted as papers to UIST 2011, the ACM Symposium on User Interface Software and Technology.
Questions? Email: firstname.lastname@example.org
Publications from this course
- What’s the Right Price? Pricing Tasks for Finishing on Time. Siamak Faridani, Björn Hartmann, and Panos Ipeirotis. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Full Paper (6 pages)
- Pricing Mechanisms for Online Labor Markets. Yaron Singer and Manas Mittal. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Full Paper (6 pages)
- MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Prayag Narula, Philipp Gutheim, David Rolnitzky, Anand Kulkarni, Björn Hartmann. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Poster
- Turkomatic: Automatic, Recursive Task and Workflow Design for Mechanical Turk. Anand Kulkarni, Matthew Can, Björn Hartmann. Proceedings of the Third AAAI Human Computation Workshop (HCOMP'11). Poster
For each class period, students will submit short reading responses online by midnight on the day before class. Reading responses are submitted directly to this wiki.
In addition to commentaries, students will be asked to lead one class discussion. The discussant should read all student commentaries before class and integrate them into the discussion.
A small number of assignments will ensure students are familiar with current crowdsourcing platforms and programming tools. For students working on research projects, additional assignments are designed to keep you on track towards a successful paper submission.
- In-class Reading 1: Crowd Control. Leah Hoffman, Crowd Control, Communications of the ACM Vol. 52 No. 3, Pages 16-17
- In-class Reading 2: Five Rules of the New Labor Pool. Jeff Howe, Wired 14.06
- Assignment due by Friday Jan 28, 5pm: Create Wiki Account
- Assignment due by Friday Jan 28, 5pm: Submit Class Petition
Jan 28: No class, but items due
- Paper 1: Human Computation: A Survey and Taxonomy of a Growing Field. Alexander J. Quinn, Benjamin B. Bederson. CHI 2011.
- Paper 2: Analyzing the Mechanical Turk Marketplace. Panagiotis G. Ipeirotis, ACM XRDS, December 2010.
- Assignment due before class on Feb 7: Create and Perform HITs on Mechanical Turk
- Assignment for project students, due before class on Feb 7: Project Abstract
- Paper 1: VizWiz: nearly real-time answers to visual questions. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. UIST 2010.
- Paper 2: Soylent: a word processor with a crowd inside. Michael S. Bernstein , Greg Little , Robert C. Miller , Björn Hartmann , Mark S. Ackerman , David R. Karger , David Crowell , Katrina Panovich. UIST 2010.
- Due: Create and Perform HITs on Mechanical Turk
- Due: for project students: Project Abstract
- Assignment due before class on Feb 28: Program HITs with the Mechanical Turk API
- Assignment for project students, due before class on Feb 14: Project Abstract Part 2
Feb 14: Systems for Crowdsourcing (Discussant: Reynold Xin, Philipp Gutheim)
- Paper 1: CrowdDB: Answering Impossible Queries. Michael Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, Reynold Xin
- Paper 2: CrowdSearch: Exploiting Crowds for Accurate Real-time Image Search on Mobile Phones. Tingxin Yan, Vikas Kumar, Deepak Ganesan, Mobicom 2010
- Due for project students, due before class on Feb 14: Project Abstract Part 2
Feb 16: No Class - Jeff Bigham (VizWiz) talk at 12pm, BiD Lab 354/360 HMMB
Feb 21: No class - President's Day"
Feb 28: Annotations and Participatory Sensing (Discussants: Chulki Lee, Wesley Willett)
- Paper 1: Utility data annotation with Amazon Mechanical Turk. Alexander Sorokin, David Forsyth. First IEEE Workshop on Internet Vision at CVPR 2008.
- Paper 2: Participatory Sensing: A citizen-powered approach to illuminating the patterns that shape our world Jeffrey Goldman et al. 2009.
- Due: Program HITs with the Mechanical Turk API
- Assignment: Prototype I
Mar 7: Crowd Programming Tools (Discussants: Manas Mittal, Kristal Curtis)
- Paper 1: TurKit: Human Computation Algorithms on Mechanical Turk. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. UIST 2010.
- Paper 2: Generalized task markets for human and machine computation. Dafna Shahaf and Eric Horvitz. 2010.
Mar 14: Task and Workflow Design (Discussant: Beth Trushkowsky, Kurtis Heimerl)
- Paper 1: Exploring Iterative and Parallel Human Computation Processes. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. HCOMP 2010.
- In-class discussion on taxonomies.
- Due: Prototype I
Mar 21: No class - Spring Break
Mar 28: Experiments (Discussants: Nicholas Kong, Beth Trushkowsky)
- Paper 0: (holdover from 3/14): Exploring Iterative and Parallel Human Computation Processes. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. HCOMP 2010.
- Paper 1: The Online Laboratory: Conducting Experiments in a Real Labor Market. Horton, John J., Rand, David G. and Zeckhauser, Richard J. SSRN, 2010.
- Paper 2: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. Jeffrey Heer , Michael Bostock. CHI 2010.
Mar 29: No Class - Leila Janah, CEO Samasource, talk at 4pm, B100 Blum Hall Creating Sustainable Livelihoods in Poor Regions
Apr 4: Economics (Discussants: David Rolnitzky, Travis Yoo)
- Paper 1: Financial Incentives and the "Performance of Crowds". Mason, W., & Watts, D. J. HCOMP 2009.
- Paper 2: Algorithmic Wage Negotiations: Applications to Paid Crowdsourcing. John Horton and Richard Zeckhauser. CrowdConf 2010.
Apr 8: No class, but assignment due for project students
Apr 11: Discussion of UIST Drafts, Taxonomy (Kurtis Heimerl leads taxonomy discussion)
Apr 18: Applications in International Development (Discussants: Prayag Narula, Ariel Chait)
- Paper 1: Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India. Shashank Khanna, Aishwarya Ratan, James Davis, WIlliam Theis. ACM DEV 2010.
- Paper 2: The Development Potential of the Virtual Economy. Vili Lehdonvirta, Helsinki Institute for Information Technology. (Read Exec Summary + Ch 2-5)
Apr 22: No Class - UIST Deadline
Apr 29: No Class - HCOMP Deadline
Potential additional topics
- Participatory Sensing: A citizen-powered approach to illuminating the patterns that shape our world Jeffrey Goldman et al. 2009.
- Traffic Sensing through mobile phones, e.g. Berkeley Mobile Millenium Project.
Games with a Purpose
Social Search and Q&A
- Aardvark WWW'10
- Stack Overflow, Lada Adamic's Q&A work.
Advanced crowdsourcing interfaces
- CrowdFlow: Integrating Machine Learning with. Mechanical Turk for Speed-Cost-Quality Flexibility Quinn and Bederson, 2010.
Evaluating crowdsourcing algorithms
- Towards a Theory of AI Completeness Shahaf and Amir, Commonsense 2007.
Consult the User's Guide for information on using the wiki software.