Applications in International Development

From CrowdsourcingSeminar
Jump to: navigation, search

Reading Responses

Kristal Curtis

Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India: This paper argues that wrt MTurk, an untapped segment of the population in India is the less educated service class, since most Indian Turkers are highly educated relative to the country's overall workforce. They then investigate the factors that serve as barriers for these people. The most important takeaway from their study was that the UI is very difficult for these people to work with. They also made an interesting case that MTurk is best seen as a source of supplemental income (rather than primary income). I found their analysis of the projected take-home pay of this type of worker to be very interesting -- it seems that due to the cost of cybercafes, many workers barely make enough money for the activity to be worthwhile. It would be interesting to see if Indian Turkers as a whole were more productive if Indians were able to function as requesters; presumably, tasks from Indians would be more accessible for other Indians.

The Development Potential of the Virtual Economy: This work explores the opportunities for employment created by today's internet climate. Many workers overseas capture a lot of revenue doing tasks related to "artificial scarcities" such as getting a company's Facebook page to have more "like"s or playing games like World of Warcraft to capture virtual goods and then sell them off to people in the developed world. It'd be great to transfer some of this energy to working towards natural scarcities.

Philipp Gutheim

The "MTurk for Low-Income Workers in India" paper presents a set of HIT design and user interface design criteria on Mechanical Turk to lower the barriers for a potential low-income worker group which is currently excluded from completing tasks yet capable of delivering the work. While the authors provide a great set of contributions for their paper, I believe the actual new inside is the last mentioned "user study with 49 low-income workers measuring the impact on work quality of different HIT design techniques (instruction translation, video tutorial etc.). I believe this is a quite interesting insight, though I believe to a certain extend this would have also happened if the user study would have been conducted with US workers. First, I expect the gain in quality due to a better understanding of what has to be done. I would argue that US worker might be more familiar with this type of work. Tagging friends on facebook, surfing on the web, reading articles etc might have provided them with a slightly better context of what needs to be done for image tagging tasks. Second, I expect parts of the gain in quality is simply because guidelines are clearer. It is easier to understand what is expected from the worker. Requesters sometimes tend to be a little bit short and vague in their HIT instructions (also: the paper's test HIT instructions weren't shown). In that sense, I would have expected an increase in quality for U.S. workers as well.

Along the line, a paper called "txteagle: Mobile Crowdsourcing" presents a mobile based crowdsourcing employment solution for individuals in South East Africa to complete simple text-based jobs on their cell phone and in turn receive airtime. The paper outlines a set of potential tasks the author envisions that can be done. It is interesting to see that txteagle - by now a Boston/SF-based startup - pivoted to on the ground marketing research leveraging their large installed worker base to provide western companies who plan to enter emerging markets with a tool to test their marketing strategy. Although USSD is a smart way to avoid the costs sending the questions and answers back and forth, it limits the set of tasks significantly. Unfortunately, the predominant part of work that is currently outsourced in platforms such as Mechanical Turk requires users to have way more advanced cell phones that would support mobile browsing etc...and even in that case, it would be rather impractical to complete some of the tasks.

The report "Development Potential of the Virtual Economy" is an interesting and very informative report. I found two things particularly interesting: First, I think it is interesting yet nothing desirable that the gaming service industry has become such a huge market. Although this provides individuals in developing economies with an income who might otherwise be unemployed but at the same time it does not create any value (from a global perspective) at all. It just shifts money from one side to the other. Second, it is really interesting to read the market analysis for the crowdsourcing market. In essence, my impression is that (in it's current shape) open crowdsourcing platforms such as MTurk have failed to be economically sustainable while "closed" specialized crowdsourcing/outsourcing services are growing rapidly.

Beth Trushkowsky

In "Evaluating and Improving the Usability", the authors determine that existing mechanical turk tasks are inaccessible to low-income indian, and workers and offer techniques to improve task user interfaces. For me, a takeaway from the paper was the extent to which people of different cultures and backgrounds have difficulty with seemingly straightforward mturk tasks. I'm much less cynical now about turkers potentially spamming my tasks or putting forth little effort. In general, the paper highlights the importance of clear instructions for tasks. However I wish that turkers wouldn't try tasks they don't fully understand (although I guess that's what qualification tests are for). Also, while I appreciate the altruistic nature of the authors' work, is it suggesting that we as requestors should aim to make our tasks more accessible?

The "virtual economy" paper describes likens microtask markets to online-gaming services or virtual goods productions, and describes the various players in such an economy, e.g., transformers, aggregators, workers. In particular, the authors note that microtask labor markets have emerged to fulfill a natural scarcity of human effort caused by the rise of the digital economy. Also, I was incredibly surprised/saddened that the online gaming and virtual goods production markets are so successful.

Siamak Faridani

The first paper "Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India" authors seem to have two main contributions. They argue that earning from mechanical turk can complement a family's annual salary and they show the effect of localization on the quality of the work by Turkers. They show that video tutorials have no significant impact on the quality of work but language localization improves the quality dramatically. The study is done in India where English is at least the official language I can imagine if it was done in other developing countries with no English familiarity the magnitude of the effect would be even more clear. I think authors could invest more on interviews and field work. I also think this paper could have been broken into two separate but perhaps more focused papers. One on the effect of crowdsourcing as an alternative earning source and the other on the effect of language localization and UI on the quality.

One issue that is always missing in these type of publications is that they all argue that you can do Mturk on the side and earn more money but I haven't really seen a work that looks at crowdsourcing as a sustainable and constant source of earning for families. In other words can we hire a group of people in india, set up work stations and provide them with HITs? are we going to have enough jobs for them? can we provide them with basic benefits (health insurance, sick days, and holiday breaks)? if not it is not really a sustainable way to help developing countries as it may reduce the level of poverty a little but for any sustainable development we need to keep these factors in mind.

I found the report, Development Potential of the Virtual Economy, a very interesting compilation of facts and numbers but I disagree with most of the conclusion. From the statistics it is clear that in game markets are huge but I am not sure it will eventually translate to development for the Chinese. First of all the actual cut that goes to the worker is only a fifth of the actual transaction (reminds me of the coffee bean example in the beginning of the report) secondly, it is an illegal activity and I don't think they pay taxes (so the revenue will not contribute to building roads, pipes or energy production). I am not sure but I don't think money laundrying has ever helped any developing country! It again goes back to a sustainable environment for labor and the in-game market is not a sustainable revenue source. Like any arbitrage opportunity sooner or later it will be closed down.

The micowork market is dwarfed when compared to the gaming opportunities. At the same time it *may* result in real development. Both papers entirely miss the opportunity of having sound macro economics arguments.

Dave Rolnitzky

Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India

This article discusses the usability barriers of AMT, and provides some new design recommendations to enable low-income workers to participate and earn money on AMT. The authors find that there are significant usability barriers for low income workers.

Overall I felt that this was a very solid article, and especially relevant for our course project MobileWorks, particularly the portion of the study that dealt with the CAPTCHA tasks. The article talked about the difficulties of some basic text entry functions, like when to put in a space between letters or words, using upper vs. lowercase, etc. These are all issues that came up during our pilot study. Also, it reinforced the importance of having a localized version of the interface. I guess I wasn't surprised by the stat presented that home PC penetration is less than 10% -- but given this small number I found it interesting that the authors would spend so much time redesigning for such a small group of users. Seems like there is a big role for mobile here.

Prayag Narula

Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India

The paper explores the possibility of using MTurk as an ICTD platform. It was interesting read though I was left less than satisfied with the quality of interviews and field work. Exposed the problems with MTurk’s User Interface which I feel is not just India specific. Their median average income of $0.2/hour is too less even for India. Would’ve liked a discussion on that.


The report of extensive field work in virtual economy including gaming, micro-tasks and cherry-blossoming. The article on gaming was way too big and worrisome. I also don’t agree with the conclusion that this can be a widely used ICTD platform just because of the fact that this is artificial scarcity.

I liked the article on micro-tasks which gave a holistic view of the potential. The author mentions that micro-tasks can be a multi-billion dollar market. MobileWorks would like that.

Manas Mittal

Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India

This is an interesting paper. It provides (somewhat) real statistics for things such as average education level, median earnings, etc., and identifies some of the bottlenecks (such as task description, cultural context, etc) which are useful and interesting to know.

One thing I noticed was that in their initial test of 7 workers (which provides some motivating ideas), it is not clear if the workers were eager to do the tasks in the first place. For example, was it clear to them that they could make $50 a month with this? Will that make them try harder? Also, in these scenarios, I think it will be easy for the subjects to get intimidated by the experimenters who are conducting the test (especially if the experimenters are foreigners), and the experimenters should provide more background about this. For the specific example, I'd be curious to find out how the Samasource interface looks, and what their error rates are. This would help us figure out what % of the error rate is just due to the cognitive hardness of the task (i.e., "vacuum cleaner for a lamp") and worker capabilities, as opposed to interface issues.

This also gave me some ideas about how these interfaces should be built. It seems that current interfaces tend to be very one-way, i.e., tasks are posted and workers do their best to do it. It would be interesting to let workers talk about the tasks (facilitate a discussion) even in a semi-moderated way. I was just filiing my taxes the other day and Turbotax had this thing called the TurboTax community where people answer people's questions, in addition to some inputs from turbotax people. It might be very useful to build something like that for more complicated tasks.

Knowledge Map of the Virtual Economy

Most of the activities talked about "Cherry Blossoming" are ethically wrong and systems and networks will ideally evolve to exclude such behavior. I do think that there is a market for preventing of Cherry Blossoming in the form of protecting your brand from fake, malicious reviewers (I know of someone who is employed by BMW to do both - prevent bad reviews on Facebook walls and write bad ones for Mercedes Benz).

The game examples led me to wonder why game companies don't just pull a zynga and start selling whatever the chinese are selling after spending hours building things. I am assuming that this is already happening, and this market exists because the companies want more than what the chinese want. If you think about it, this is crazy. It is the epitome if wasted time, and yet, market dynamics support and encourage this (creation of artificial scarcity).

Wesley Willett

Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India. This paper more systematically documents the observation that I think we've probably all made that, due of language, technical expertise, and cultural barriers, Indian workers perform most Mechanical Turk tasks more poorly than western workers. While their findings are somewhat obvious (simplified tasks written in workers' native tongue and designed to circumvent shortcomings of Amazon's platform improve results), someone needed to document them and I think they've done a relatively good job. The authors clearly articulate a short list of design recommendations that anyone designing for this class of workers should probably take into account.

That said, it's depressing that their interventions still only resulted in reasonable answers about 65% of the time, and that even video tutorials were unable to improve performance on such simple tasks. This raises continued questions about the viability of relying on these workers. Moreover, their suggestions that tasks be translated manually into regional dialects presents overhead issues (although maybe this could be Turked as well), and Mechanical Turk doesn't make it easy to allocate tasks to workers based on language and precise geographic region. Finally, I found their discussion of earning potential to be fairly uninformative, since their numbers are heavily tied to the pay, timing, and completion rates from their particular study.

The Development Potential of the Virtual Economy makes the assertion that hundreds of thousands of workers are employed in the "third-party gaming services industry", harvesting resources and building up characters in virtual worlds. The numbers cited here were shocking to me since they imply that this secondary market within online games may be an order of magnitude larger than the market for MTurk-style microwork. This suggests that there may be a far more interesting economy in massively multiplayer games that we've almost completely ignored in our discussions. The sheer size and complexity of the entities involved in these markets (including companies which employ hundreds of workers to perform goldmining along with administrators to handle translation, logistics, fraud checks, etc.) is startling.

Chulki Lee

Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India improved usability especially for those who have basic digital literacy. The paper discussed not only efficiency of a design but also its impact, like long-term education for users, low-income workers in this case. I agree that usability and many possible barriers (assumptions beyond tasks) can be critical to create more opportunity for them. I would like to more research about the role and potential of MTurk in ICTD can be discussed more broadly and throughly.

The Development Potential of the Virtual Economy overviews the virtual economy and its segments such as microwork and cherry blossoming. I like reading regulatory and policy section discussing diverse topics.

Sally Ahn

The Mechanical Turk for Low-Income Workers in India paper describes how microtasks on Mechanical Turk can be redesigned to allow better access for low-income workers in India. They report their findings from a survey on MTurk that reveals that MTurk is a significant source of income for many college graduates in India. They address common difficulties these workers face in completing MTurk tasks (eg. language barrier, complex user interface, etc.). While I appreciate this effort to facilitate the tasks for low-income workers, I agree with the authors' caution that targeting these workers with microtasks may create some negative impact. I was shocked that most of these workers were college graduates with a bachelor's degree or higher; it concerns me that these workers may be wasting their skills and talent by engaging in these tedious and non-stimulating tasks. Nevertheless, I appreciate the authors' good intent to improve the user interface for supplementary income from MTurk.