RIP to everyone’s favorite p<0.05 factory
October 1, 2026 at 10:12 AM
RIP to everyone’s favorite p<0.05 factory
pouring one out for mturk today
September 30, 2026 at 10:49 PM
pouring one out for mturk today
“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.”
January 8, 2026 at 8:47 PM
“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.”
I focus quite a bit on MTurk in the book I'm writing about ImageNet. Modern "AI" wouldn't exist if it weren't for MTurk, and, moreover, the blueprint ImageNet created for data labor.
Amazon is officially shutting down Mechanical Turk next month. It feels like the end of an era.
I've written about the platform on and off for close to a decade. We wouldn't have the AI tools we do now if it weren't for the low-paid, largely invisible labor of MTurkers
I've written about the platform on and off for close to a decade. We wouldn't have the AI tools we do now if it weren't for the low-paid, largely invisible labor of MTurkers
August 26, 2026 at 1:13 AM
I focus quite a bit on MTurk in the book I'm writing about ImageNet. Modern "AI" wouldn't exist if it weren't for MTurk, and, moreover, the blueprint ImageNet created for data labor.
Maybe in hindsight not the greatest idea to brag about the sheer volume of MTurk studies you're conducting?
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
Any sufficiently advanced file-drawer is indistinguishable from p-hacking
March 12, 2026 at 9:44 AM
Maybe in hindsight not the greatest idea to brag about the sheer volume of MTurk studies you're conducting?
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
P.S. For a second I thought that Malte added the MTurk thing himself as a joke, but no. It's right there on the website. Major "grad student who never said no" energy.
RIP
there will never again be an MTurk poll
there will never again be an MTurk poll
October 1, 2026 at 11:55 AM
RIP
there will never again be an MTurk poll
there will never again be an MTurk poll
As MTurk closes today, the work isn’t disappearing. Some requesters are moving to other platforms, while others have created their own sites.
Here’s one to check out:
Piper - www.piperpays.com
You may remember them as PickFU on Mturk
Here’s one to check out:
Piper - www.piperpays.com
You may remember them as PickFU on Mturk
September 30, 2026 at 7:31 PM
As MTurk closes today, the work isn’t disappearing. Some requesters are moving to other platforms, while others have created their own sites.
Here’s one to check out:
Piper - www.piperpays.com
You may remember them as PickFU on Mturk
Here’s one to check out:
Piper - www.piperpays.com
You may remember them as PickFU on Mturk
The same exact survey is fielded on three platforms: Prolific, CloudResearch Connect, and MTurk.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
January 8, 2026 at 8:47 PM
The same exact survey is fielded on three platforms: Prolific, CloudResearch Connect, and MTurk.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts.
On MTurk, however, the results are the opposite.
Even after aggressive screening, many contradictory items remain positively correlated on MTurk.
January 8, 2026 at 8:48 PM
Even after aggressive screening, many contradictory items remain positively correlated on MTurk.
Exciting new vibecession explainer:
It is caused by the Mturk shock in survey costs.
It is caused by the Mturk shock in survey costs.
April 27, 2026 at 6:24 PM
Exciting new vibecession explainer:
It is caused by the Mturk shock in survey costs.
It is caused by the Mturk shock in survey costs.
Over 96% of these clearly opposite item pairs are positively correlated on MTurk. In other words, many respondents give similar answers to statements that literally contradict each other.
January 8, 2026 at 8:48 PM
Over 96% of these clearly opposite item pairs are positively correlated on MTurk. In other words, many respondents give similar answers to statements that literally contradict each other.
Researchers have long argued about whether Amazon Mechanical Turk (MTurk) survey data can be trusted.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
January 8, 2026 at 8:47 PM
Researchers have long argued about whether Amazon Mechanical Turk (MTurk) survey data can be trusted.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
This paper takes a simple approach to evaluating the quality of data currently produced by MTurk.
The author gives respondents pairs of questions that are obviously contradictory.
Chatbots already crippling MTurk…
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
February 2, 2026 at 1:31 AM
Chatbots already crippling MTurk…
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
link.springer.com/article/10.3...
(Heard someone call this an “existential crisis for the behavioral sciences”).
Now implications for public opinion research in contested environments… watch out.
Emerson is a hack pollster that leans pretty far right and is erratic. It's because they use mturk. There is no way the race swings like that. Ignore outliers.
September 25, 2025 at 7:27 PM
Emerson is a hack pollster that leans pretty far right and is erratic. It's because they use mturk. There is no way the race swings like that. Ignore outliers.
But MTurk isn't shutting down because "AI" has just become _that_ good. The work is just moving to different platforms, to different subcontractors, to more levels of obscurity buried within AI supply chains.
August 26, 2026 at 1:13 AM
But MTurk isn't shutting down because "AI" has just become _that_ good. The work is just moving to different platforms, to different subcontractors, to more levels of obscurity buried within AI supply chains.
MTurk Studies
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
May 28, 2024 at 4:42 AM
MTurk Studies
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
"We administered 27 semantic antonyms— ...(e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample."
Preprint: osf.io/preprints/ps...
#Methodology #Psychology #MetaSci
amazon mturk is shutting down at the end of the month
September 29, 2026 at 1:17 AM
amazon mturk is shutting down at the end of the month
We should no longer trust data collected on MTurk
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
January 8, 2026 at 8:26 PM
We should no longer trust data collected on MTurk
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
link.springer.com/article/10.3...
My guess is that other online data is going to drop in quality due to LLMs. This is going to be an existential crisis for the behavioral sciences.
Behavioural Science in 2026.
October 1, 2026 at 1:35 PM
Behavioural Science in 2026.
Prolific vs MTurk
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.”
January 8, 2026 at 9:46 PM
Prolific vs MTurk
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
Responses to semantic antonyms are significantly negatively correlated among Prolific and Connect participants (as you'd expect) but *positively* among MTurk participants, suggesting very careless responding among MTurkers!
#SocialPsyc
The implication is severe: careless responding on MTurk isn’t rare noise; it’s systematic enough to flip the sign of relationships and generate results that are the opposite of what they really are.
Wow; this is pretty damning.
link.springer.com/article/10.3...
Wow; this is pretty damning.
link.springer.com/article/10.3...
January 8, 2026 at 8:49 PM
The implication is severe: careless responding on MTurk isn’t rare noise; it’s systematic enough to flip the sign of relationships and generate results that are the opposite of what they really are.
Wow; this is pretty damning.
link.springer.com/article/10.3...
Wow; this is pretty damning.
link.springer.com/article/10.3...
Looking at some pilot stuff for our next instrument and it looks like mturk might be just totally cooked if you give a shit about quality. bummer. I think LLMs really fucked this.
July 11, 2025 at 2:08 AM
Looking at some pilot stuff for our next instrument and it looks like mturk might be just totally cooked if you give a shit about quality. bummer. I think LLMs really fucked this.
I mean, why should anyone trust Americans on anything? The center to center left in this country don't believe in anything except a weird form of Democratic Centralism where appointing supreme court justices is your little treat for mindlessly following "social consensus" as defined by mturk polls
January 19, 2026 at 4:19 AM
I mean, why should anyone trust Americans on anything? The center to center left in this country don't believe in anything except a weird form of Democratic Centralism where appointing supreme court justices is your little treat for mindlessly following "social consensus" as defined by mturk polls