“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.”
January 8, 2026 at 8:47 PM
“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.”
I focus quite a bit on MTurk in the book I'm writing about ImageNet. Modern "AI" wouldn't exist if it weren't for MTurk, and, moreover, the blueprint ImageNet created for data labor.
Amazon is officially shutting down Mechanical Turk next month. It feels like the end of an era.
I've written about the platform on and off for close to a decade. We wouldn't have the AI tools we do now if it weren't for the low-paid, largely invisible labor of MTurkers
I've written about the platform on and off for close to a decade. We wouldn't have the AI tools we do now if it weren't for the low-paid, largely invisible labor of MTurkers
August 26, 2026 at 1:13 AM
I focus quite a bit on MTurk in the book I'm writing about ImageNet. Modern "AI" wouldn't exist if it weren't for MTurk, and, moreover, the blueprint ImageNet created for data labor.
Maybe in hindsight not the greatest idea to brag about the sheer volume of MTurk studies you're conducting?
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
Any sufficiently advanced file-drawer is indistinguishable from p-hacking
March 12, 2026 at 9:44 AM
Maybe in hindsight not the greatest idea to brag about the sheer volume of MTurk studies you're conducting?
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
The same exact survey is fielded on three platforms: Prolific, CloudResearch Connect, and MTurk.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
January 8, 2026 at 8:47 PM
The same exact survey is fielded on three platforms: Prolific, CloudResearch Connect, and MTurk.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
Even after aggressive screening, many contradictory items remain positively correlated on MTurk.
January 8, 2026 at 8:48 PM
Even after aggressive screening, many contradictory items remain positively correlated on MTurk.
Exciting new vibecession explainer:
It is caused by the Mturk shock in survey costs.
It is caused by the Mturk shock in survey costs.
April 27, 2026 at 6:24 PM
Exciting new vibecession explainer:
It is caused by the Mturk shock in survey costs.
It is caused by the Mturk shock in survey costs.
Over 96% of these clearly opposite item pairs are positively correlated on MTurk. In other words, many respondents give similar answers to statements that literally contradict each other.
January 8, 2026 at 8:48 PM
Over 96% of these clearly opposite item pairs are positively correlated on MTurk. In other words, many respondents give similar answers to statements that literally contradict each other.
Researchers have long argued about whether Amazon Mechanical Turk (MTurk) survey data can be trusted.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
January 8, 2026 at 8:47 PM
Researchers have long argued about whether Amazon Mechanical Turk (MTurk) survey data can be trusted.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
Chatbots already crippling MTurk…
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
February 2, 2026 at 1:31 AM
Chatbots already crippling MTurk…
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
She also mentions MTurk, Scale AI, Hive and Mighty AI.
September 24, 2026 at 10:19 AM
She also mentions MTurk, Scale AI, Hive and Mighty AI.
Emerson is a hack pollster that leans pretty far right and is erratic. It's because they use mturk. There is no way the race swings like that. Ignore outliers.
September 25, 2025 at 7:27 PM
Emerson is a hack pollster that leans pretty far right and is erratic. It's because they use mturk. There is no way the race swings like that. Ignore outliers.
But MTurk isn't shutting down because "AI" has just become _that_ good. The work is just moving to different platforms, to different subcontractors, to more levels of obscurity buried within AI supply chains.
August 26, 2026 at 1:13 AM
But MTurk isn't shutting down because "AI" has just become _that_ good. The work is just moving to different platforms, to different subcontractors, to more levels of obscurity buried within AI supply chains.
MTurk Studies
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
May 28, 2024 at 4:42 AM
MTurk Studies
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
We should no longer trust data collected on MTurk
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
January 8, 2026 at 8:26 PM
We should no longer trust data collected on MTurk
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
Prolific vs MTurk
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.”
January 8, 2026 at 9:46 PM
Prolific vs MTurk
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
Meta invents Mturk
Meta Tests Muse AI Agent Calls That Are Actually Made By Humans in a Call Center
"This has potential for so much negative PR. It could portray us as ‘their AI is not good enough so they still need humans’ kind of coverage for this launch."
www.404media.co
September 25, 2026 at 4:37 PM
Meta invents Mturk
The implication is severe: careless responding on MTurk isn’t rare noise; it’s systematic enough to flip the sign of relationships and generate results that are the opposite of what they really are.
Wow; this is pretty damning.
link.springer.com/article/10.3...
Wow; this is pretty damning.
link.springer.com/article/10.3...
January 8, 2026 at 8:49 PM
The implication is severe: careless responding on MTurk isn’t rare noise; it’s systematic enough to flip the sign of relationships and generate results that are the opposite of what they really are.
Wow; this is pretty damning.
link.springer.com/article/10.3...
Wow; this is pretty damning.
link.springer.com/article/10.3...
Looking at some pilot stuff for our next instrument and it looks like mturk might be just totally cooked if you give a shit about quality. bummer. I think LLMs really fucked this.
July 11, 2025 at 2:08 AM
Looking at some pilot stuff for our next instrument and it looks like mturk might be just totally cooked if you give a shit about quality. bummer. I think LLMs really fucked this.
Excited to co-organize the Bridging Human and LLM Annotations for Social Science tutorial at #ic2s2 tomorrow!
Join us at 1:30pm CEST to learn how to combine Qualtrics+Prolific, MTurk, & LLMs for automated annotation and valid estimates.
Code&materials: github.com/kristinaglig...
Join us at 1:30pm CEST to learn how to combine Qualtrics+Prolific, MTurk, & LLMs for automated annotation and valid estimates.
Code&materials: github.com/kristinaglig...
July 21, 2025 at 12:20 AM
Excited to co-organize the Bridging Human and LLM Annotations for Social Science tutorial at #ic2s2 tomorrow!
Join us at 1:30pm CEST to learn how to combine Qualtrics+Prolific, MTurk, & LLMs for automated annotation and valid estimates.
Code&materials: github.com/kristinaglig...
Join us at 1:30pm CEST to learn how to combine Qualtrics+Prolific, MTurk, & LLMs for automated annotation and valid estimates.
Code&materials: github.com/kristinaglig...
I mean, why should anyone trust Americans on anything? The center to center left in this country don't believe in anything except a weird form of Democratic Centralism where appointing supreme court justices is your little treat for mindlessly following "social consensus" as defined by mturk polls
January 19, 2026 at 4:19 AM
I mean, why should anyone trust Americans on anything? The center to center left in this country don't believe in anything except a weird form of Democratic Centralism where appointing supreme court justices is your little treat for mindlessly following "social consensus" as defined by mturk polls
I can’t decide if the file drawer or the absolute poverty of methodological effort it takes to run 2,000 MTurk studies is more concerning to me
Any sufficiently advanced file-drawer is indistinguishable from p-hacking
March 12, 2026 at 2:01 PM
I can’t decide if the file drawer or the absolute poverty of methodological effort it takes to run 2,000 MTurk studies is more concerning to me
Unsurprising but still big: MTurk is on its way out, killed by AI.
Mechanical Turk was a mainstay of social & survey research through the 2010s, as it allowed you to quickly buy access to many representative humans. It was pretty good at it, until LLMs came along and everyone started using AI
Mechanical Turk was a mainstay of social & survey research through the 2010s, as it allowed you to quickly buy access to many representative humans. It was pretty good at it, until LLMs came along and everyone started using AI
July 7, 2026 at 5:14 AM
Unsurprising but still big: MTurk is on its way out, killed by AI.
Mechanical Turk was a mainstay of social & survey research through the 2010s, as it allowed you to quickly buy access to many representative humans. It was pretty good at it, until LLMs came along and everyone started using AI
Mechanical Turk was a mainstay of social & survey research through the 2010s, as it allowed you to quickly buy access to many representative humans. It was pretty good at it, until LLMs came along and everyone started using AI
it's the modal reply of an mturk worker in kenya who has been told that the model is jailbreakable with emoji or is too easy to trick into ERP and who has to write two hundred separate refusal replies that correctly address the inappropriate request
August 28, 2025 at 5:28 AM
it's the modal reply of an mturk worker in kenya who has been told that the model is jailbreakable with emoji or is too easy to trick into ERP and who has to write two hundred separate refusal replies that correctly address the inappropriate request
September 10, 2025 at 1:39 PM
Tal trabalho heteromado é muitas vezes tornado invisível para a não devida compensação dos seus trabalhadores, como é o caso do Amazon MTurk, ou por ações que beiram a ilegalidade como é o caso da Infraestrutura Humana da Desinformação. +
September 11, 2024 at 3:53 PM
Tal trabalho heteromado é muitas vezes tornado invisível para a não devida compensação dos seus trabalhadores, como é o caso do Amazon MTurk, ou por ações que beiram a ilegalidade como é o caso da Infraestrutura Humana da Desinformação. +