Gabriel suggests that instinct is a trap, because a system flawlessly aligned to one person or group is also what would let it override everyone else. The measure of good alignment isn't obedience. It's fairness that holds up even when we disagree.

arxiv.org/abs/2001.09768

He closes by weighing three routes to such principles: global human rights, a Rawlsian veil of ignorance, and social choice theory.

The quiet reversal at the paper's core is what makes it worth sitting with. The instinct is to make AI more obedient, more precisely tuned to what we say we want.

His central claim then reframes the whole debate: the goal is not to discover the one true morality and install it in machines, but to identify principles people can accept as fair even when their moral beliefs differ. Alignment, he writes, is political, not metaphysical.

aligning to revealed preferences risks encoding people's malicious impulses and the quiet resignation of those who have learned to expect too little. Gabriel's move is to argue that the more robust option situates human direction inside a moral framework rather than optimizing any single signal.

our underlying interests or well-being, or our values. Each fails in a characteristic way. Aligning to literal instructions invites the King Midas problem, getting exactly what you asked for and not what you wanted;

He then shows there is no single obvious thing to align to, laying out six candidates: an AI could follow our instructions, our expressed intentions, the preferences our behavior reveals, the preferences we would hold if we were better informed and reasoning well,

Gabriel splits alignment into two tasks that can't be pulled apart: the technical work of building AI that reliably does what it should, and the normative work of choosing what that is.

The question worth sitting with is not whether young people will talk to AI, they already are, but whether we are building AI, and the human support around it, so that it points them back toward each other rather than away.

www.insidehighered.com/news/student...

But when a teenager concludes that a chatbot understands them better than the people in their life, the risk is not that they got help, it's that the easier comfort quietly replaces the harder work of building real relationships and reaching out to people who can truly hold what they're carrying.

This is important because there is a real difference between a tool that helps you think and a tool you mistake for a friend you can trust. AI can offer immediate, judgment-free support at the exact moment a young person feels they have no one, and that accessibility is not nothing.

Fewer than half of high school seniors rated their mental health as good or very good, and only a small minority of all respondents said they felt prepared for what comes next.

More telling than the frequency is the reason: many of the high schoolers who use AI this way said they do so because they don't have anyone else to talk to, and a notable share said AI is more supportive than their friends, both far above the rates their college counterparts reported.

A College Pulse survey commissioned by Uwill, of just over a thousand rising high school and college seniors, found that high school seniors were more than twice as likely as college seniors to use AI at least weekly to talk through relationships or emotional issues.

Thank you Louis Tay, Karina Vold, Ashton Anderson, Harsh Kumar, Daniel Harris, Joyce Lum, Renée Sirbu, Qianfeng Wen, Jinsoo Choi, and Lauren Moran for bringing so much curiosity and the good company. ✨