6 ms·

My Half Workday as a Turker

245 points by alceufc 12y ago

ggchappell 12y ago
Interesting article. One hears about MT, but not so much from this perspective. It's also interesting that the linked page has two scroll bars (tried both Chrome & FF).
- stopbits 12y ago
  This page is just a container for a google doc in an iframe. Interesting CMS strategy.
  - dugmartin 12y ago
    He should set overflow: hidden on the body
  - krilnon 12y ago
    It seems to work reasonably transparently most of the time. I was in one of the author's classes in ~2010 and I think the class schedule was put online in the same way.
geronimox 12y ago
"At the time, I was hoping Dmitry would reject my HIT so I could go on a tirade about how unfair it was, and try to get him banned from MTurk or whatever." I found this quite amusing. You know why this HIT was approved? Because requesters are also punished for rejecting too many HITs! He also would have been banned had Amazon discovered the scheme he was pulling.
- compare 12y ago
  This is a serious problem that requesters face. Often the workers who spam everyone with fake work will report anyone who rejects their work as retribution.
alexchamberlain 12y ago
Very interesting. Has anyone used Mechanical Turk to get work done?
- viggity 12y ago
  i've used it a couple times in a lame attempt to find leads. IIRC, I paid $.40 to find a name/phone number/website of construction equipment rental companies in 50 different cities. They did exactly what I wanted them to do, although I never did anything with the results. One guy seemed to pick up most of the jobs, so I tipped him an extra $10.
- jimmar 12y ago
  I've collected experimental data using Mechanical Turker. In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. You have to kick people out who aren't paying any attention at all and just want to click through your survey to get paid. (Similar techniques also have to be employed for undergraduate participants who just want to get done with your survey so they can earn their 10 points of extra credit). If nothing else, Turkers are good at making sure your data collection instruments are working the way that they are supposed to.
  - pavel_lishin 12y ago
    > In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. Do you ask any 'trick' questions that are the cognitive equivalent of a captcha?
    - VLM 12y ago
      I've turked for fun and you'd be shocked how many incredibly long psych surveys have scale of 1 to 10 questions like "enter a 5 as the answer to this question if you expect to get paid". Also if a survey asks "how much do you love the new windows phone" you can fully expect the opposite question perhaps on the next page "how much do you hate the new windows phone". Presumably they correlate the data somehow? I've never run into a serious college professor category of trick question. Turk is a grind game if you don't need the money. If you're susceptible its going to be a little moth to flame for you and all that matters is your meaningless number goes up, or if you're easily bored like me, you'll turk for like 20 hours one week and then not think about it again, until this HN question, in fact. I will say that any researcher type would be horribly disappointed by turking because I did it while watching TV, or drinking a beer, and I have a low tolerance not being much of a drinker. Speaking of beer, I am told there is, or was, something of a fad of trying to turk fast enough to pay for your booze. There is no way humanly possible to turk fast enough to drink at a bar, but cheap freshman class beer in a dorm is totally realistic. Can you pay for that six pack on mturk before you finish drinking it? So... I don't think anyone could get homework assignments done on mturk. At least if you want to pass the class or have graduated from grade school. The linked article example is far beyond anything I ever wrote for mturk. The most insulting and infuriating turk jobs I ran into were not on the author's list, basically being paid to be spammed to. Here's 20 full page variations of a home equity mortgage spam, now pick your favorite one for a nickle. Or call this number and talk to this scammer for a minimum of 5 minutes and then come here and fill out a survey for like 10 cents, at least on the surface its for "quality control" but you know its really to give you the hard sell. Life is cheap on the turk, and its full of work hits that pay the equivalent of a buck an hour or less, which is eventually super demotivating / depressing. You hear about cool high paying hits but end up filling out 20 page surveys about B.S. for like ten cents, and getting sick of it.
abakker 12y ago
I've used MTurk to have interviews transcribed several times, and almost always had flawless results. (sometimes jargon or names trips them up.) I've always been incredibly impressed at the speed they get done, which is substantially better than I've ever achieved doing them myself. I usually aim to pay $10 for a 10 minute segment, which should amount to more than $10/hour for a good typist.
danielweber 12y ago
Last time I looked at MTurk, it was offering money if I would complete some simple task, and, oh, by the way, download this crapware onto your computer and run it.
- crazypyro 12y ago
  To be fair, those types of HITs are forbidden by the terms (often removed quickly) and, in my experience, tend to be rare.
- tectec 12y ago
  That sounds a lot like someones research project: http://threatpost.com/research-project-pays-people-to-download-run-executables/106773 http://threatpost.com/research-project-pays-people-to-downlo...
bitlord_219 12y ago
What a dehumanizing experience.
- runj__ 12y ago
  Yes, working: how awful. I hope you're trying to be on the workers side but either way this is a terrible thing to say. The way you look down upon people trying their very best to make a living is not helping them at all.
  - andreasvc 12y ago
    It's dehumanizing to work and get paid little or nothing at all. You seem to be reading much more in the comment you're replying to than was actually said.
    - iLoch 12y ago
      Depends on the sort of tasks you're doing. Someone above mentioned receiving ~$0.27/min of translations. This is a relatively high figure in comparison to a lot of tasks, but if the task you're doing isn't entirely mind numbing then being a Turker isn't as bad as it sounds. $0.27/minute is something like ~$16/hr - if you're working at a gas station with a computer handy, you'd be able to more than double your wage. Are the tasks undervalued? Not sure it's fair for us to say. The HITs keep coming in and they continue to be completed, so something is working.
      - tormeh 12y ago
        Depends what sort of gas station you're at, I guess. My sister worked at a gass station one summer. She was mostly alone on the job, kept an eye on everyone who filled gas, made fast food, made sure the shelves were well stocked with products and cleaned. Not much room for going to the toilet much less sitting at the computer, but then her wage was something like 30 USD/h.
    - plaguuuuuu 12y ago
      First world problems though. It's little or nothing by your standards. In India it would be a very different story.
incision 12y ago
I dabble on MTurk every now and then. It might sound strange, but I find certain kinds of survey type simple tasks incredibly relaxing and some of the HITs clearly connect to what seems to be interesting research. Pretty much everything the author says here is spot-on and has been for a long time. MTurk is positively flooded with HITs for generating fake reviews for products and fake content for websites. The best paying HITs by far are translation tasks - transcribing Arabic and Farsi seems to pay the best ($0.30 or more per minute of transcription). I aim for academic projects with preference for things which require qualification (simple math / reading comprehension tests usually) as they generally pay quite a bit more. I've earned an average of $0.27 per HIT and have never had one rejected. Some of the more interesting / unusual tasks I've seen: * Pinpoint various joints (shoulder, elbow, knee) in each frame of a clip showing a baseball player swinging a bat. * Drop a pin on an estimate map location based on scenery shown in a short video clip (This was long before MapCrunch/GeoGuessr). * Manipulate the camera in a scene of flat-shaded objects to bring them into 'correct' perspective. * Choose 'preferred' structure designs (little houses) which appear to have been created by some sort of genetic algorithm.
- VLM 12y ago
  "Some of the more interesting tasks I've seen" I get distracted by categorization jobs, like look at this blueprint and if there is a blueprint number in the title box, enter it. Well, I end up admiring the print. Oh look a giant gear for some kind of mining thing. Why, what an interesting looking architectural drawing. Similar thing happens with categorize document numbers. Apparently some municipality decided to turn millions of paper scanned pages of contract bids into piles sorted by contract, so I'd end up sitting there reading all about bids to put in some drainage culvert. Who would ever have guessed theres so many steps and processes and backfill and compaction are critical and theres so many inspectors... I mean you can half A a drainage culvert in like one paragraph but if you want it done right its 50 pages of PDF. These jobs are probably impossible without an excellent short term memory or dual (or more) monitors. The ones that annoyed me the most were "leave blank if no number" so I'd scour the document for what seemed like forever just to make sure. Inevitably leading me to becoming fascinated by drainage culvert design. Its very D+D like. Whats behind this dungeon door? Oh, I see, a blueprint of a giant gear from some mining equipment. Fascinating. Like archeology but I'm getting paid for it. Well, not much pay.
zackmorris 12y ago
I think this is a perfect example of how money today is worth more or less than it actually is. Another way of saying that is that money is nonlinear or even non-constant. For example, say you are broke and wandering the streets of a big city and really want to buy a bus ticket to another city for $20 or whatever. No matter how much your net worth is, you can't get that $20 easily. You could try asking various businesses to do some work for cash, you could try a song and dance routine, etc, but your choices all amount to various forms of begging. Meanwhile you can be sitting in an office somewhere and make that money in less than 3 hours even at minimum wage, even if you do little or no work. You can even sell something on craigslist if you’re home. I suppose in desperation you could sell blood plasma, but that’s one of very few lifelines. Nobody thinks $20 is worth much, but when you don’t have it, it’s very expensive indeed. To me, the root of the problem is whether you are resource rich or resource poor. So the web, by virtue of being intangible, is almost quintessentially resourceless. The trend seems to be lower and lower wages for increasingly onerous labor. In other words, if you have money, you get a real world return greater than the value of your money. But if you don’t have money, then getting it requires an expenditure of resources and effort larger than the value of the money itself. At some point in the near future, acquiring money will be so expensive from a labor standpoint that it will be cheaper to simply do things yourself and live outside of mainstream society by bartering goods and services. This really bothers me, because that shouldn’t be the goal of progress. The paradox is that even though every new fiverr and mturk create more jobs, they lower the value of work. Right now this is affecting developing countries by creating a race-to-the-bottom economy, but the futurist in me looks all the way to the end and sees how so many jobs today (especially the non-production ones like administrative/clerical work) will eventually be automated by technology and thereby make the purchase of capital through labor even more expensive. Does anyone see a way out of this? Did I miss something fundamental?
- presidentender 12y ago
  In the aftermath of the Bubonic plague in Europe, the shortage of labor led to higher wages for serfs, eventually ending any semblance of what we'd call 'feudalism.' I'm not advocating a mass die-off of human beings, but any other means of limiting the supply of low-quality labor would have the same effect. If some sort of back-to-the-land subsistence thing became popular among the youth, those who didn't partake might find that a McJob paid more by virtue of there being fewer people willing to do it. Sort of a "shrug" by the lower-class Atlas instead of the elite.
pc86 12y ago
For those of you like me getting a white screen with no content, the direct link is https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxvz18Lv-ej8U9QXEgPlsg/pub?embedded=true https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxv...
- robertfw 12y ago
  FYI, Privacy badger is what caused my browser to whitescreen, as it was blocking the pass off to google docs
aceperry 12y ago
Kind of reminds me of the saying, "pay peanuts, get monkeys."
crazypyro 12y ago
I used MTurk off and on through my teens[0] (15-18) and found that the majority of my earnings came in through the 50 word or less summaries. You get to a point where you've read enough PR department press releases that you can write the entire summary (at least to the required standards, which in my experience, is "barely comprehensible English") by looking at the headline. I was able to finish 50 word summaries in under 30 seconds. They values range from .25-.50 and they seem to come from a lot of submitters. I've had a theory that out in the world somewhere is someone selling a guide to using MTurk to create article summaries and the guide suggests the price range of $.25-.50 because for the entire time that I've been on MTurk (many years now), every time I come back the price range is the same with small fluctuation. Past the relatively low amount of summaries that are posted daily, money can be made by watching the right forums where people post high paying HITs (the name of the most popular forum eludes me at the moment) that are typically 5-15 minute surveys that pay .5-3.00 each. By far, the biggest barrier to making a decent wage for me was the number of available HITs. The ones that pay over pennies each are few while the ones that are horrible time invests are everywhere. If I had an unlimited number of summary HITs, I could have made well over minimum wage. [0] Made about $450 over the time, but this was sporadic, random working periods.
- ski 12y ago
  I'm curious, have you found the ability to write 50 word summaries in under 30 seconds to be a useful skill later in life? It sounds like good practice in concise communication.
  - e12e 12y ago
    I'd say the above comment is about average, to average+ for hn... so probably not? (Only being partially sarcastic) [edit: I now realize that that's not enough data, we'd also need how long it takes to write the average hn comment, and how long op took writing his :-) ]
  - Natsu 12y ago
    I'm surprised someone hasn't automated jobs like that, honestly.
spikels 12y ago
Jargon Alert - HITs (Human Intellegence Tasks) are simply the individual tasks (or units of work) that the worker is paid for completing.
- tripzilch 12y ago
  Thank you! I had to ctrl-F down 75% of this thread because it wasn't explained in the article either.
pessimizer 12y ago
An journal article I submitted a few years ago: Amazon Mechanical Turk: Gold Mine or Coal Mine? http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057 http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057
sparkzilla 12y ago
I checked Mechanical Turk as an example of how to not to price for Newslines, my crowdsourced content site. It seems that most publishers use crowdsourcing as a way to save money, and while that may work for some tasks, you will not get consistent work done. Rather than crowdsourcing being a way to find cheap writers, content business should see crowdsourcing as a way to find a diverse pool of writers who have different interests or knowledge, who can dip in and out of work, and then pay them fair money for their time. I pay $1 per post for writers to write 50-100 word posts that follow a very specific format. We provide feedback to the writer to try to make them make money faster. The way I figure it is, if the writer makes money then I can make money. If they stick with you, you don't have to train new people. People won't leave after half a day. We have already paid out thousands of dollars and our site is growing fast. Writer satisfaction is high. We currently have a waiting list of hundreds of writers wanting to sign up. http://newslines.org/newslines-rewards/ http://newslines.org/newslines-rewards/
- e12e 12y ago
  Initially I found $1/[edit: item] strangely low as an example of "high pay". Having a look at what newslines is actually about, I still think it is low, but not as low as I intially thought (I'd assumed you wanted contribution from domain experts, who'd I guess you'd price at at least 100/hour). I'd assume anyone that can write well, should be able to make at the very minimum 30/hour writing -- That leaves on average 2 minutes/newsline. I suppose the lesson is that there are many skilled poor people out there. (Note, this isn't meant as negative criticism, just some observations)
  - sparkzilla 12y ago
    Thank you for your observations, I appreciate the tone. In our case the writers are working to a formula, which for most people is a lot easier than creative writing. So far most of our writers are actually work-at-home moms who are happy to make around $10 an hour, for a task they see as being quite enjoyable - writing news summaries about their favorite celebrities, while sitting at home. They can finish a task in a few minutes, depending on the topic, and bank $10-$20 a day extra cash (or more) fairly easily. Then they can come back the next day for more. On our side we get a 24hr/day pool of talent that wants to keep adding content, without the need for office space and other HR issues.
ChuckMcM 12y ago
This was a great read. I found it particularly interesting that the author came into each encounter 'cold', which is to say without much in the way of additional tooling. They did install a word counter extension but much of their tools were ad-hoc or non-existent. That suggested that there may be an interesting 'toolware' market for Turkers. For example, document editing; Lets say we build a simple document editing platform (copy editing not typesetting) which includes components that are dropbox like (shared storage), Google docs like etc. And design a workflow around that toolset. So pick a HIT, get a document id, it shows up in your shared storage, use the included tool to edit it, when you're done, click 'done', and have the whole thing resubmit back for evaluation. That might allow you to focus on editing and not get hung up on a bunch of getting it done issues. Not sure how practical that is, but if there are common Turk workflows it might be useful to build some common tooling for them.
- _delirium 12y ago
  > That suggested that there may be an interesting 'toolware' market for Turkers. In the early days of Mechanical Turk (2005-06), when almost all the HITs were seeded by Amazon themselves and were of a few common types that kept being replenished, there were browser extensions to greatly improve the UI for the common tasks (the default UI is very bad). One of the more profitable tasks was fixing the location of addresses on the new-defunct A9 BlockView (a product similar to what Google would later introduce as Street View). They'd present you about 10 photos of a streetfront that were supposed to be near an address, and you were supposed to pick the one centered around the address (e.g. the store or house entrance). Or else indicate none of the above, if the alignment was seriously off so that the correct address was out of the frame. If you used the regular website UI, you could make maybe $3-4/hr, but most of this was tedious clicking: accept HIT, click a tiny radio button, scroll down, submit HIT, request next HIT, repeat. The browser extension implemented the obvious UI improvement: use the images instead of tiny radio buttons as click targets, and then auto-submit and request the next one on click. I think in one version you could also just hit a number 0-9 on the numpad instead of clicking. With that extension I was making $20-30/hr for a little bit.
joereggan190 12y ago
http://freerice.com/content-group/skyhd-watch-netherlands-vs-argentina-live-online-stream http://freerice.com/content-group/skyhd-watch-netherlands-vs...
mkhpalm 12y ago
I wonder what pays more? Panhandling on the street corner or working on mechanical turk?
lilsunnybee 12y ago
It's been over a year and a half since i've spent any serious time on Mechanical Turk trying to supplement low income, so some of my experience might be dated. With a bit of a learning curve it's plenty doable to find decently paying jobs in transcription, writing, etc. on MT, especially if you have a strong grasp of the English language. But even with higher-skilled, higher-paying jobs on that platform, the overall experience of working on MT can often end up being plenty demoralizing and unfair in many many ways, such as qualifying for skilled work and also being compensated for work performed. The most frustrating jobs i've ever performed on the platform were usually approx 5-10 minute tasks that seemed to be decently paying and straightforward, asking you to categorize items based on some arbitrary but seemingly simple system, but later all 5 or more submissions you might have made could be blanket denied / payment refused. In that case not only do you not get payed, but having HIT submissions rejected really seriously hurts your reputation and what jobs you're allowed to apply for on MT; often many more high paying jobs on the system use high acceptance rate and flawless work history as the lowest bar for entry, and so MT workers can be screwed over incredibly by having work rejected through no fault of their own. To not get seriously screwed newcomers are somehow supposed to know which types of HIT's to definitely avoid that could kill their rating, but without enough information to actually make that judgement. A task might seem easy and give the illusion that it's okay to send 5 or 10 submissions in a row, only to later have them all rejected and serious reputation damage done. I think it's interesting but also really sad how this mirrors a lot of the unfairness and power imbalance in the larger working world as well. Reputation is everything, but if you just happen to get unlucky to start out your career working for clueless bosses / clients with unrealistic expectations, there's a good chance future job and career prospects are going to be seriously hindered, if not completely derailed. Transcription work was usually very well paying i found, but doing one long transcription task for a $30 payout is a huge risk the way things were set up. Never mind that you worked for 4 or 5 hours on it; whoever assigned the job can reject it for whatever random reason they want and you get nothing. As nice as it can be to earn a nice wage for more high-skilled tasks, the possibility of being denied any payment for a days work with little recourse is really frustrating and demoralizing; the way Mechanical Turk is structured if anyone's taking a negative financial hit for work done, it's almost without fail going to be the worker. Though this CMU professor obviously put some work into this write-up, i don't see how somebody can possibly get an accurate picture of what it's like trying to earn an income on one of these platforms by just devoting 4 hours and extrapolating based on that really limited experience. It is really easy to miss all but the most obvious and glaring problems that way, just from not sticking around long enough to even run across them. There are plenty of MT workers who have spent much more substantial amounts of time on the platform, that are much better able to communicate benefits and pitfalls for the everyday worker. They might not be professors at prestigious universities like CMU, but having a decent amount of hands-on experience should really be the low bar for discussing pros/cons of the platform seriously; just as one example assumptions about longer-term take-home pay in the write-up were pretty naive: many better paying tasks (especially web browsing tasks and surveys paying more than a pittance) are often in limited supply and cannot be repeated either at all, or only a few times per day / week, etc. Even the best employers providing the most decent fairly-compensated work come and go, making a steady healthy income sometimes very difficult to achieve even in the best of circumstances. Mechanical Turk has experienced a lot of success from the start largely because they created a service that makes it so easy for virtually anybody with internet access and a basic computer, to work from home and perform simple tasks / other types of work when and wherever they want, and actually get paid for it. By any decent standards of developed-world countries the pay is shit though, and worker protections and benefits are nonexistent; workers can be denied payment and even have their reputation ruined for no good reason, with little in the way of an appeals process. But even now services like MT are still a pretty new thing. Maybe in 10 or even 20 more years most of the bugs will get worked out, and workers of every skill level can be a on more equal footing when negotiating with employers, settling disputes etc.
pgl 12y ago
Can anyone suggest any alternatives to Mechanical Turk that offer a better experience for the worker?
- garretthunyadi 12y ago
  Check out CrowdFlower. The workers are mturk workers, but I believe that there's more attention placed on the worker.
ccb_ 12y ago
Like Jeff, I'm a professor who works on crowdsourcing. For one of my current projects, I am working on a tool to make the Amazon Mechanical Turk marketplace fairer for workers. I have developed a Chrome plugin that tracks the length of time it takes to complete a task, and a web site that aggregates the information across many workers. Crowd-Workers.com allows workers to discovery higher paying work by sorting tasks by estimated hourly rate. Read more about the tool: http://www.cis.upenn.edu/~ccb/publications/crowd-workers-poster.pdf http://www.cis.upenn.edu/~ccb/publications/crowd-workers-pos... Try out our Chrome plugin and our web site: http://crowd-workers.com/landing http://crowd-workers.com/landing

245 points by alceufc 12y ago

Interesting article. One hears about MT, but not so much from this perspective. It's also interesting that the linked page has two scroll bars (tried both Chrome & FF).

stopbits 12y ago
This page is just a container for a google doc in an iframe. Interesting CMS strategy.
- dugmartin 12y ago
  He should set overflow: hidden on the body
- krilnon 12y ago
  It seems to work reasonably transparently most of the time. I was in one of the author's classes in ~2010 and I think the class schedule was put online in the same way.

geronimox 12y ago

"At the time, I was hoping Dmitry would reject my HIT so I could go on a tirade about how unfair it was, and try to get him banned from MTurk or whatever." I found this quite amusing. You know why this HIT was approved? Because requesters are also punished for rejecting too many HITs! He also would have been banned had Amazon discovered the scheme he was pulling.

compare 12y ago
This is a serious problem that requesters face. Often the workers who spam everyone with fake work will report anyone who rejects their work as retribution.

alexchamberlain 12y ago

Very interesting. Has anyone used Mechanical Turk to get work done?

viggity 12y ago
i've used it a couple times in a lame attempt to find leads. IIRC, I paid $.40 to find a name/phone number/website of construction equipment rental companies in 50 different cities. They did exactly what I wanted them to do, although I never did anything with the results. One guy seemed to pick up most of the jobs, so I tipped him an extra $10.
jimmar 12y ago
I've collected experimental data using Mechanical Turker. In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. You have to kick people out who aren't paying any attention at all and just want to click through your survey to get paid. (Similar techniques also have to be employed for undergraduate participants who just want to get done with your survey so they can earn their 10 points of extra credit). If nothing else, Turkers are good at making sure your data collection instruments are working the way that they are supposed to.
- pavel_lishin 12y ago
  > In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. Do you ask any 'trick' questions that are the cognitive equivalent of a captcha?
  - VLM 12y ago
    I've turked for fun and you'd be shocked how many incredibly long psych surveys have scale of 1 to 10 questions like "enter a 5 as the answer to this question if you expect to get paid". Also if a survey asks "how much do you love the new windows phone" you can fully expect the opposite question perhaps on the next page "how much do you hate the new windows phone". Presumably they correlate the data somehow? I've never run into a serious college professor category of trick question. Turk is a grind game if you don't need the money. If you're susceptible its going to be a little moth to flame for you and all that matters is your meaningless number goes up, or if you're easily bored like me, you'll turk for like 20 hours one week and then not think about it again, until this HN question, in fact. I will say that any researcher type would be horribly disappointed by turking because I did it while watching TV, or drinking a beer, and I have a low tolerance not being much of a drinker. Speaking of beer, I am told there is, or was, something of a fad of trying to turk fast enough to pay for your booze. There is no way humanly possible to turk fast enough to drink at a bar, but cheap freshman class beer in a dorm is totally realistic. Can you pay for that six pack on mturk before you finish drinking it? So... I don't think anyone could get homework assignments done on mturk. At least if you want to pass the class or have graduated from grade school. The linked article example is far beyond anything I ever wrote for mturk. The most insulting and infuriating turk jobs I ran into were not on the author's list, basically being paid to be spammed to. Here's 20 full page variations of a home equity mortgage spam, now pick your favorite one for a nickle. Or call this number and talk to this scammer for a minimum of 5 minutes and then come here and fill out a survey for like 10 cents, at least on the surface its for "quality control" but you know its really to give you the hard sell. Life is cheap on the turk, and its full of work hits that pay the equivalent of a buck an hour or less, which is eventually super demotivating / depressing. You hear about cool high paying hits but end up filling out 20 page surveys about B.S. for like ten cents, and getting sick of it.

abakker 12y ago

I've used MTurk to have interviews transcribed several times, and almost always had flawless results. (sometimes jargon or names trips them up.) I've always been incredibly impressed at the speed they get done, which is substantially better than I've ever achieved doing them myself. I usually aim to pay $10 for a 10 minute segment, which should amount to more than $10/hour for a good typist.

danielweber 12y ago

Last time I looked at MTurk, it was offering money if I would complete some simple task, and, oh, by the way, download this crapware onto your computer and run it.

crazypyro 12y ago
To be fair, those types of HITs are forbidden by the terms (often removed quickly) and, in my experience, tend to be rare.
tectec 12y ago
That sounds a lot like someones research project: http://threatpost.com/research-project-pays-people-to-download-run-executables/106773 http://threatpost.com/research-project-pays-people-to-downlo...

bitlord_219 12y ago

What a dehumanizing experience.

runj__ 12y ago
Yes, working: how awful. I hope you're trying to be on the workers side but either way this is a terrible thing to say. The way you look down upon people trying their very best to make a living is not helping them at all.
- andreasvc 12y ago
  It's dehumanizing to work and get paid little or nothing at all. You seem to be reading much more in the comment you're replying to than was actually said.
  - iLoch 12y ago
    Depends on the sort of tasks you're doing. Someone above mentioned receiving ~$0.27/min of translations. This is a relatively high figure in comparison to a lot of tasks, but if the task you're doing isn't entirely mind numbing then being a Turker isn't as bad as it sounds. $0.27/minute is something like ~$16/hr - if you're working at a gas station with a computer handy, you'd be able to more than double your wage. Are the tasks undervalued? Not sure it's fair for us to say. The HITs keep coming in and they continue to be completed, so something is working.
    - tormeh 12y ago
      Depends what sort of gas station you're at, I guess. My sister worked at a gass station one summer. She was mostly alone on the job, kept an eye on everyone who filled gas, made fast food, made sure the shelves were well stocked with products and cleaned. Not much room for going to the toilet much less sitting at the computer, but then her wage was something like 30 USD/h.
  - plaguuuuuu 12y ago
    First world problems though. It's little or nothing by your standards. In India it would be a very different story.

incision 12y ago

I dabble on MTurk every now and then. It might sound strange, but I find certain kinds of survey type simple tasks incredibly relaxing and some of the HITs clearly connect to what seems to be interesting research. Pretty much everything the author says here is spot-on and has been for a long time. MTurk is positively flooded with HITs for generating fake reviews for products and fake content for websites. The best paying HITs by far are translation tasks - transcribing Arabic and Farsi seems to pay the best ($0.30 or more per minute of transcription). I aim for academic projects with preference for things which require qualification (simple math / reading comprehension tests usually) as they generally pay quite a bit more. I've earned an average of $0.27 per HIT and have never had one rejected. Some of the more interesting / unusual tasks I've seen: * Pinpoint various joints (shoulder, elbow, knee) in each frame of a clip showing a baseball player swinging a bat. * Drop a pin on an estimate map location based on scenery shown in a short video clip (This was long before MapCrunch/GeoGuessr). * Manipulate the camera in a scene of flat-shaded objects to bring them into 'correct' perspective. * Choose 'preferred' structure designs (little houses) which appear to have been created by some sort of genetic algorithm.

VLM 12y ago
"Some of the more interesting tasks I've seen" I get distracted by categorization jobs, like look at this blueprint and if there is a blueprint number in the title box, enter it. Well, I end up admiring the print. Oh look a giant gear for some kind of mining thing. Why, what an interesting looking architectural drawing. Similar thing happens with categorize document numbers. Apparently some municipality decided to turn millions of paper scanned pages of contract bids into piles sorted by contract, so I'd end up sitting there reading all about bids to put in some drainage culvert. Who would ever have guessed theres so many steps and processes and backfill and compaction are critical and theres so many inspectors... I mean you can half A a drainage culvert in like one paragraph but if you want it done right its 50 pages of PDF. These jobs are probably impossible without an excellent short term memory or dual (or more) monitors. The ones that annoyed me the most were "leave blank if no number" so I'd scour the document for what seemed like forever just to make sure. Inevitably leading me to becoming fascinated by drainage culvert design. Its very D+D like. Whats behind this dungeon door? Oh, I see, a blueprint of a giant gear from some mining equipment. Fascinating. Like archeology but I'm getting paid for it. Well, not much pay.

zackmorris 12y ago

I think this is a perfect example of how money today is worth more or less than it actually is. Another way of saying that is that money is nonlinear or even non-constant. For example, say you are broke and wandering the streets of a big city and really want to buy a bus ticket to another city for $20 or whatever. No matter how much your net worth is, you can't get that $20 easily. You could try asking various businesses to do some work for cash, you could try a song and dance routine, etc, but your choices all amount to various forms of begging. Meanwhile you can be sitting in an office somewhere and make that money in less than 3 hours even at minimum wage, even if you do little or no work. You can even sell something on craigslist if you’re home. I suppose in desperation you could sell blood plasma, but that’s one of very few lifelines. Nobody thinks $20 is worth much, but when you don’t have it, it’s very expensive indeed. To me, the root of the problem is whether you are resource rich or resource poor. So the web, by virtue of being intangible, is almost quintessentially resourceless. The trend seems to be lower and lower wages for increasingly onerous labor. In other words, if you have money, you get a real world return greater than the value of your money. But if you don’t have money, then getting it requires an expenditure of resources and effort larger than the value of the money itself. At some point in the near future, acquiring money will be so expensive from a labor standpoint that it will be cheaper to simply do things yourself and live outside of mainstream society by bartering goods and services. This really bothers me, because that shouldn’t be the goal of progress. The paradox is that even though every new fiverr and mturk create more jobs, they lower the value of work. Right now this is affecting developing countries by creating a race-to-the-bottom economy, but the futurist in me looks all the way to the end and sees how so many jobs today (especially the non-production ones like administrative/clerical work) will eventually be automated by technology and thereby make the purchase of capital through labor even more expensive. Does anyone see a way out of this? Did I miss something fundamental?

presidentender 12y ago
In the aftermath of the Bubonic plague in Europe, the shortage of labor led to higher wages for serfs, eventually ending any semblance of what we'd call 'feudalism.' I'm not advocating a mass die-off of human beings, but any other means of limiting the supply of low-quality labor would have the same effect. If some sort of back-to-the-land subsistence thing became popular among the youth, those who didn't partake might find that a McJob paid more by virtue of there being fewer people willing to do it. Sort of a "shrug" by the lower-class Atlas instead of the elite.

pc86 12y ago

For those of you like me getting a white screen with no content, the direct link is https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxvz18Lv-ej8U9QXEgPlsg/pub?embedded=true https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxv...

robertfw 12y ago
FYI, Privacy badger is what caused my browser to whitescreen, as it was blocking the pass off to google docs

aceperry 12y ago

Kind of reminds me of the saying, "pay peanuts, get monkeys."

crazypyro 12y ago

I used MTurk off and on through my teens[0] (15-18) and found that the majority of my earnings came in through the 50 word or less summaries. You get to a point where you've read enough PR department press releases that you can write the entire summary (at least to the required standards, which in my experience, is "barely comprehensible English") by looking at the headline. I was able to finish 50 word summaries in under 30 seconds. They values range from .25-.50 and they seem to come from a lot of submitters. I've had a theory that out in the world somewhere is someone selling a guide to using MTurk to create article summaries and the guide suggests the price range of $.25-.50 because for the entire time that I've been on MTurk (many years now), every time I come back the price range is the same with small fluctuation. Past the relatively low amount of summaries that are posted daily, money can be made by watching the right forums where people post high paying HITs (the name of the most popular forum eludes me at the moment) that are typically 5-15 minute surveys that pay .5-3.00 each. By far, the biggest barrier to making a decent wage for me was the number of available HITs. The ones that pay over pennies each are few while the ones that are horrible time invests are everywhere. If I had an unlimited number of summary HITs, I could have made well over minimum wage. [0] Made about $450 over the time, but this was sporadic, random working periods.

ski 12y ago
I'm curious, have you found the ability to write 50 word summaries in under 30 seconds to be a useful skill later in life? It sounds like good practice in concise communication.
- e12e 12y ago
  I'd say the above comment is about average, to average+ for hn... so probably not? (Only being partially sarcastic) [edit: I now realize that that's not enough data, we'd also need how long it takes to write the average hn comment, and how long op took writing his :-) ]
- Natsu 12y ago
  I'm surprised someone hasn't automated jobs like that, honestly.

spikels 12y ago

Jargon Alert - HITs (Human Intellegence Tasks) are simply the individual tasks (or units of work) that the worker is paid for completing.

tripzilch 12y ago
Thank you! I had to ctrl-F down 75% of this thread because it wasn't explained in the article either.

pessimizer 12y ago

An journal article I submitted a few years ago: Amazon Mechanical Turk: Gold Mine or Coal Mine? http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057 http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057

sparkzilla 12y ago

I checked Mechanical Turk as an example of how to not to price for Newslines, my crowdsourced content site. It seems that most publishers use crowdsourcing as a way to save money, and while that may work for some tasks, you will not get consistent work done. Rather than crowdsourcing being a way to find cheap writers, content business should see crowdsourcing as a way to find a diverse pool of writers who have different interests or knowledge, who can dip in and out of work, and then pay them fair money for their time. I pay $1 per post for writers to write 50-100 word posts that follow a very specific format. We provide feedback to the writer to try to make them make money faster. The way I figure it is, if the writer makes money then I can make money. If they stick with you, you don't have to train new people. People won't leave after half a day. We have already paid out thousands of dollars and our site is growing fast. Writer satisfaction is high. We currently have a waiting list of hundreds of writers wanting to sign up. http://newslines.org/newslines-rewards/ http://newslines.org/newslines-rewards/

e12e 12y ago
Initially I found $1/[edit: item] strangely low as an example of "high pay". Having a look at what newslines is actually about, I still think it is low, but not as low as I intially thought (I'd assumed you wanted contribution from domain experts, who'd I guess you'd price at at least 100/hour). I'd assume anyone that can write well, should be able to make at the very minimum 30/hour writing -- That leaves on average 2 minutes/newsline. I suppose the lesson is that there are many skilled poor people out there. (Note, this isn't meant as negative criticism, just some observations)
- sparkzilla 12y ago
  Thank you for your observations, I appreciate the tone. In our case the writers are working to a formula, which for most people is a lot easier than creative writing. So far most of our writers are actually work-at-home moms who are happy to make around $10 an hour, for a task they see as being quite enjoyable - writing news summaries about their favorite celebrities, while sitting at home. They can finish a task in a few minutes, depending on the topic, and bank $10-$20 a day extra cash (or more) fairly easily. Then they can come back the next day for more. On our side we get a 24hr/day pool of talent that wants to keep adding content, without the need for office space and other HR issues.

ChuckMcM 12y ago

This was a great read. I found it particularly interesting that the author came into each encounter 'cold', which is to say without much in the way of additional tooling. They did install a word counter extension but much of their tools were ad-hoc or non-existent. That suggested that there may be an interesting 'toolware' market for Turkers. For example, document editing; Lets say we build a simple document editing platform (copy editing not typesetting) which includes components that are dropbox like (shared storage), Google docs like etc. And design a workflow around that toolset. So pick a HIT, get a document id, it shows up in your shared storage, use the included tool to edit it, when you're done, click 'done', and have the whole thing resubmit back for evaluation. That might allow you to focus on editing and not get hung up on a bunch of getting it done issues. Not sure how practical that is, but if there are common Turk workflows it might be useful to build some common tooling for them.

_delirium 12y ago
> That suggested that there may be an interesting 'toolware' market for Turkers. In the early days of Mechanical Turk (2005-06), when almost all the HITs were seeded by Amazon themselves and were of a few common types that kept being replenished, there were browser extensions to greatly improve the UI for the common tasks (the default UI is very bad). One of the more profitable tasks was fixing the location of addresses on the new-defunct A9 BlockView (a product similar to what Google would later introduce as Street View). They'd present you about 10 photos of a streetfront that were supposed to be near an address, and you were supposed to pick the one centered around the address (e.g. the store or house entrance). Or else indicate none of the above, if the alignment was seriously off so that the correct address was out of the frame. If you used the regular website UI, you could make maybe $3-4/hr, but most of this was tedious clicking: accept HIT, click a tiny radio button, scroll down, submit HIT, request next HIT, repeat. The browser extension implemented the obvious UI improvement: use the images instead of tiny radio buttons as click targets, and then auto-submit and request the next one on click. I think in one version you could also just hit a number 0-9 on the numpad instead of clicking. With that extension I was making $20-30/hr for a little bit.

joereggan190 12y ago

http://freerice.com/content-group/skyhd-watch-netherlands-vs-argentina-live-online-stream http://freerice.com/content-group/skyhd-watch-netherlands-vs...

mkhpalm 12y ago

I wonder what pays more? Panhandling on the street corner or working on mechanical turk?

lilsunnybee 12y ago

It's been over a year and a half since i've spent any serious time on Mechanical Turk trying to supplement low income, so some of my experience might be dated. With a bit of a learning curve it's plenty doable to find decently paying jobs in transcription, writing, etc. on MT, especially if you have a strong grasp of the English language. But even with higher-skilled, higher-paying jobs on that platform, the overall experience of working on MT can often end up being plenty demoralizing and unfair in many many ways, such as qualifying for skilled work and also being compensated for work performed. The most frustrating jobs i've ever performed on the platform were usually approx 5-10 minute tasks that seemed to be decently paying and straightforward, asking you to categorize items based on some arbitrary but seemingly simple system, but later all 5 or more submissions you might have made could be blanket denied / payment refused. In that case not only do you not get payed, but having HIT submissions rejected really seriously hurts your reputation and what jobs you're allowed to apply for on MT; often many more high paying jobs on the system use high acceptance rate and flawless work history as the lowest bar for entry, and so MT workers can be screwed over incredibly by having work rejected through no fault of their own. To not get seriously screwed newcomers are somehow supposed to know which types of HIT's to definitely avoid that could kill their rating, but without enough information to actually make that judgement. A task might seem easy and give the illusion that it's okay to send 5 or 10 submissions in a row, only to later have them all rejected and serious reputation damage done. I think it's interesting but also really sad how this mirrors a lot of the unfairness and power imbalance in the larger working world as well. Reputation is everything, but if you just happen to get unlucky to start out your career working for clueless bosses / clients with unrealistic expectations, there's a good chance future job and career prospects are going to be seriously hindered, if not completely derailed. Transcription work was usually very well paying i found, but doing one long transcription task for a $30 payout is a huge risk the way things were set up. Never mind that you worked for 4 or 5 hours on it; whoever assigned the job can reject it for whatever random reason they want and you get nothing. As nice as it can be to earn a nice wage for more high-skilled tasks, the possibility of being denied any payment for a days work with little recourse is really frustrating and demoralizing; the way Mechanical Turk is structured if anyone's taking a negative financial hit for work done, it's almost without fail going to be the worker. Though this CMU professor obviously put some work into this write-up, i don't see how somebody can possibly get an accurate picture of what it's like trying to earn an income on one of these platforms by just devoting 4 hours and extrapolating based on that really limited experience. It is really easy to miss all but the most obvious and glaring problems that way, just from not sticking around long enough to even run across them. There are plenty of MT workers who have spent much more substantial amounts of time on the platform, that are much better able to communicate benefits and pitfalls for the everyday worker. They might not be professors at prestigious universities like CMU, but having a decent amount of hands-on experience should really be the low bar for discussing pros/cons of the platform seriously; just as one example assumptions about longer-term take-home pay in the write-up were pretty naive: many better paying tasks (especially web browsing tasks and surveys paying more than a pittance) are often in limited supply and cannot be repeated either at all, or only a few times per day / week, etc. Even the best employers providing the most decent fairly-compensated work come and go, making a steady healthy income sometimes very difficult to achieve even in the best of circumstances. Mechanical Turk has experienced a lot of success from the start largely because they created a service that makes it so easy for virtually anybody with internet access and a basic computer, to work from home and perform simple tasks / other types of work when and wherever they want, and actually get paid for it. By any decent standards of developed-world countries the pay is shit though, and worker protections and benefits are nonexistent; workers can be denied payment and even have their reputation ruined for no good reason, with little in the way of an appeals process. But even now services like MT are still a pretty new thing. Maybe in 10 or even 20 more years most of the bugs will get worked out, and workers of every skill level can be a on more equal footing when negotiating with employers, settling disputes etc.

pgl 12y ago

Can anyone suggest any alternatives to Mechanical Turk that offer a better experience for the worker?

garretthunyadi 12y ago
Check out CrowdFlower. The workers are mturk workers, but I believe that there's more attention placed on the worker.

ccb_ 12y ago

Like Jeff, I'm a professor who works on crowdsourcing. For one of my current projects, I am working on a tool to make the Amazon Mechanical Turk marketplace fairer for workers. I have developed a Chrome plugin that tracks the length of time it takes to complete a task, and a web site that aggregates the information across many workers. Crowd-Workers.com allows workers to discovery higher paying work by sorting tasks by estimated hourly rate. Read more about the tool: http://www.cis.upenn.edu/~ccb/publications/crowd-workers-poster.pdf http://www.cis.upenn.edu/~ccb/publications/crowd-workers-pos... Try out our Chrome plugin and our web site: http://crowd-workers.com/landing http://crowd-workers.com/landing