Pakgamers Is Now DEAD! A Win For Requesters!

I was going to post a followup to Fredrick's post about Tim Edwards, but in doing some research I discovered that Pakgamers is now dead.

The final post on pakgamers

 As most of you know, I try to educate both workers and requesters on how to use Mturk so that it works out best for all of us. In doing this, I have to keep an eye on people who are out to scam Mturk. Since Amazon has put a halt on new accounts outside of the United States, one of the biggest scam forums for cheating Mturk has finally closed down. PAKGAMERS SCAMMER FORUM THREAD
If you read that entire thread, you will be shocked at the amount of dishonest practices that were going on overseas. Everything from how to make multiple accounts without getting caught to selling of American accounts overseas. This was a thorn in the side of honest workers for a long time and it is finally gone.

Now that Amazon Mechanical Turk is not accepting new accounts overseas, the quality of data is increasing dramatically. It is unfortunate because a lot of Asian workers were honest and hard working, but at the same time a majority of the accounts that were being created to cheat requesters were being created in India and Asia. I also need to state that there are many workers from outside the United States that have a better grasp on the English language than Americans. Many of them have a better work ethic and are diligent and honest workers.

For years the workers on TurkerNation have been reporting this forum to Amazon to show them what is going on and finally Amazon has taken steps to reduce this cheating and tighten the reigns on the workforce. This is not only a win for requesters on Amazon Mechanical Turk, it is also a win for workers. This adds an element of trust to the workforce that has never been possible in the past.

Add me on Twitter for updates to this blog. @pokernonsmoker

Thought Masters was just bad for non-blessed workers? It's even worse for requesters: the Tim Edwards story

It's a well known sentiment that the Masters qualification absolutely sucks for the majority of workers. Just to recap: Amazon pushes it on requesters, eliminating their chances at finding work, Amazon doesn't tell them how to get it, so even if they have one million HITs approved they might not be a “Master” and Amazon doesn't care at all about workers with any of these problems. Requesters can't even recommend workers as Masters, as a few have done for me. I have it on good authority that the Masters qualification is given to people who do random HITs on mTurk that Amazon puts up to test workers. I have no idea what these HITs pay, but if they pay little I doubt I've ever even noticed them. This really does explain why so many do not have Masters that deserve it. Take this how you will, a good journalist never names his sources. ;)

As a worker and requester (and consultant to requesters), though, I hate the Masters program doubly because as bad as it is for workers, it is even worse for requesters. Why?

• Requesters need to pay three times more to use Masters. Instead of paying 10¢ on every dollar, they pay 30¢.
• If a requester has a worker who they already know produces good work but that worker does not have Masters, that worker is barred from all of their HITs.
• Since there are a lot less Masters than Amazon says there are, work completed by Masters is done at a slower rate than that of work done by workers with sane custom qualification schemes or workers with system qualifications.

Well, points one and two are very well proven, but I get a lot of flack when I try to claim point three. Fortunately, Tim Edwards came to my aid.

Thanks to a lone spammer or several spammers, Mr. Edwards decided that he had had enough and that he was going to give Masters a shot. Amazon has been pushing it hard very recently, so it may have seemed like a wise thing to do. Unfortunately for Mr. Edwards, he might not have known a few things:

• He has several large datasets of workers. He could very easily have picked out hundreds of workers, the real Masters of his HITs, and qualified them with a custom qualification. I would have helped him out with this for as little as $30.
• There aren't as many Masters as Amazon shows on the worker site, as I explained in my previous post. This could change, but at the moment, there are around 5,000 of them. I have no idea how many of those are active accounts, or if that is even a real number.

Anyway, on to the fun part. Opportunities like this don't come around very often, and reeling from some very new workers being granted Masters while some of their most loyal fans are left wondering where they went wrong, I decided to take down the number remaining of Edward's HITs every fifteen minutes. Here are my results, with fun annotations!

Added 11/16: As ChapterFoe points out in the comments, these results, as shocking as they are, might not be conclusive. I've left this post alone for history's sake — but Masters are very unlikely to do do HITs that pay little (which I confirmed) so I may not have proved much by posting this at all. In any case, it is an interesting look at work done on Mechanical Turk. I still think work done by Masters will always be slower, but this level of slower might be pushing it and is not representative of the whole body of work done by Masters.

I'll post the full data once Edwards gets all his HITs back, but let me clue you in on some fun things right here and now:

• With 78000 HITs at 10¢, Edwards was spending 20% more than necessary, a whooping $1,560. For that price, I would have run his HIT ten times over.
• In the entire time Edwards was using Masters (twelve hours almost exactly), he got around 287 HITs done. If we figure each HIT takes one minute, using the crowd was actually 4.7x slower than if he had done all the work himself.
• For every hour of Masters HITs (he got around 24 HITs/hr with Masters), Edwards gets 1,610 by letting those with the mark of Cain (I kid!) work on his HITs. That's 67 times more work being done!

As to the Masters program? It's bad for workers and bad for requesters and needs some serious reforms. I think it's a good idea at heart, but Amazon's lack of transparency on the issue dooms it to blog posts like this one.

Until next time!

HITs not getting completed? You might have missed Amazon's new default Masters requirement. As they intended.

Today, in their infinite wisdom and right before the weekend so they could deal with all complaints on Monday, the Mechanical Turk team decided to roll out a fundamental changes to the Requester Web UI.

Specifically, they decided that choosing which workers you want to work on your HITs is now an "Advanced" option, much like choosing paper or plastic at the grocery store is an advanced option or deciding whether you want to buy brand name or generic medicine is for advanced users only. Of course, the things that are really advanced are woefully poorly described and left out of the Requester UI, but we all knew that already.

This correlated with a big change on the Worker's site: the Categorization Masters and Photo Moderation Masters were combined into one big group, which, much to the surprise of everyone, led to 20,686 "Masters"!

So, how does this all play into the requester UI? Amazon, the sleeping giant, decided that instead of policing their site and helping requesters get better quality work by making it easier to create custom qualifications, they just wanted more money. A noble goal (by some definitions of noble), and to achieve it, they made Masters the default option for all new projects, and hid how to qualify workers:

Doesn't it just feel wrong to select "Customize Worker Requirements"? You're basically being told your results will be poor quality. I've been looking at the mTurk worker site, and I've noticed that some poor university students who run HITs have been oblivious to the change, and are likely about to run over their budgets and thrown out of school, all the while wondering why their results were so slow to be delivered:

So, be aware that this has changed when posting new HITs with the web UI. If you use the mTurk API or CLT, this doesn't affect you. 

As to how workers are taking it, I have a 99.9% approval rating and 22,000 approved HITs and am seriously looking into finding another way to make ends meet. There's nothing like not being able to get a license that not even the issuing body will tell you the requirements for, and quite frankly I'm quickly becoming sick of it. This sleight of hand with the Requester UI is a dirty trick that will confuse requesters and do little else.

Even if you win the rat race, you're still a rat. ~ William Sloane Coffin

Until next time!