Mechanical Turk and Artificial Reviews

So I've been very interested in the whole mTurk phenomenon, with crowdsourcing small tasks. I decided to start doing tasks myself in order to see what things were best for this type of work, but frankly, I was appalled by some of the uses. I've seen solicitations for newsletter signups, which are bound to get someone spammed. I've seen solicitations for "please post 3 comments to the following blogs" to artificially get a conversation going. I've seen requests for Facebook follows/likes/etc., requests to break CAPTCHA's, requests to "download college material from this site," and "fill out a free credit report online." (If that last one isn't a serious identity scam, I don't know what is.)



Now I am able to see that this is part of the engine behind artificial website rankings and reviews, to pump up garbage, and game the ranking metrics. Here's a terrible one for academics: just today, I saw someone use it to artificially bump up his SSRN downloads, which I happened to report to both SSRN and to mTurk. I also 1-starred the book who's PR company was soliciting 5 star reviews on Amazon.

This brings me to an interesting point, though, because a whole community of people has evolved around mTurk, and some of them have moral stances, flagging questionable tasks like this. (See http://turkers.proboards.com/) The sad thing is though, not everyone has this kind of moral compass or perhaps they need the money/just don't care. I can understand those motives even if I don't condone it. So I have to say that this experience has really changed my view of what is "real" on various ratings and review sites. We might think those things are easy to moderate out, but some of the requests are really so specific as to require "at least 200 facebook friends" or other qualifications, and of course the sanctioning power of reputation effects doesn't matter if someone has already created an artificial online presence or several. (See what I mean by googling up "black hat SEO" where people will proudly talk about creating multiple artificial identities and "even using them to talk with one another.")

Ok, that last one is a little involved for the mTurkers, and could still happen even before the internet. But if there are people who will post bogus reviews for $0.05 each, at least there are a few people who will probably feel the moral outrage at this kind of thing, like myself, and post a negative comment on some of those same sites. At least that's what I'm hoping.

But I can easily see how this is going to become an increasingly difficult business to check and to moderate. It is going to require new and perhaps more sophisticated methods of doing marketing research, for example, because it is just so easy to post a "help me vote for funding for X medical disease online" task on mTurk--I actually saw this one--and if one just trusts the survey results, you might not be getting anything near a real sample. I'm just waiting until I see one along the lines of "buy one share of this company stock."

That said, mTurk does have some great potential, and I'll have more later on the useful and interesting tasks you can do with mTurk. However, I have come to the conclusion that if Amazon wants to police this better, they need to moderate requestors better (ala Amazon stores), and/or create an "approve this mTurk task" as an ongoing HIT to clear each and every request, which would work nicely with new or otherwise questionable requestors. I am sure that if they police things, new blackhat sites will spring up, but since Mechanical Turk is associated with the Amazon brand, they do need to be careful not to let it be the wild west, or they could suffer their own negative reputation effects. 

 Read More
A nice academic blog on Mechanical Turk: http://behind-the-enemy-lines.blogspot.com/ 
Interesting labor regulation issues from the crowdflower blog: http://blog.crowdflower.com/2010/06/regulating-distributed-work-part-three-why-its-a-good-idea/