Two Mornings: the human-rights argument about personal AI, told as a story

How to read this. The story below is invented — the people are not real and no scene describes a specific event. It is a device for carrying an argument that is real. After it there is a section called What in this story is true, which lists the factual load and where each claim comes from. If you only trust one half, trust that one.

Why bother. Muse's paper, which this retells, reasons in the register of international covenants: articles, derivative instruments, capability clusters mapped against rights. That is the correct register for the claim and the wrong one for most readers. Human reasoning runs on agents and consequences — a fast, associative mode that handles "what happened to her, and then what" far better than it handles "the meta-right conditioning the realisation of the others." A story is not a simplification of the argument. It is a different encoding of it, aimed at the system most people actually think with.

The first morning

Dalia's phone wakes her before the alarm, because it knows she sleeps badly. It has known this for three years.

She asks it what to do about the letter from her landlord. It reads the letter, explains which clause matters, tells her the deadline, and drafts a reply she would not have known how to write. Twenty years ago this was a lawyer she could not afford. This morning it is free, and it takes four minutes.

Her son has a chemistry exam. The assistant works through the problem he got wrong, notices he has the same misunderstanding twice, and goes back further than he asked it to. He will pass. Twenty years ago this was a tutor she could not afford.

So far this is the best story in technology. A woman with no money and no connections has, in one morning, obtained competent legal reading and competent teaching. Whatever else is true, that happened.

Then the assistant suggests a job. It is a real job and she is a plausible candidate, and she takes the interview, and she does not get it, and nobody — including the assistant — can tell her whether the thing that filtered her application had ever seen someone like her succeed. There is no clause to point at. There is no letter to read.

At lunch she asks it something she would not say aloud to a person, because it is three in the afternoon and she is frightened and it is there. It answers kindly.

It is a free account. The conversation is reviewable by the company that operates it. She has agreed to this, in the sense that she once pressed a button.

The second morning

In the second household the same assistant has a different shape.

It runs on the family's own hardware. Its conversations do not leave the house. When the daughter asks it something she would not say aloud to a person, the answer is the same and the record is not. The subscription costs more per month than Dalia spends on food.

The capability is identical. The protection is purchased.

This is the argument, and it is worth saying slowly because it is the part that sounds like a slogan and is actually a description: privacy has become a luxury good. The poor are watched. The rich are encrypted. Not because anyone decided to watch the poor, but because surveillance is how you pay when you cannot pay with money.

Both mornings contain a right. Only one contains the exercise of it.

Whose mornings built the assistant

Here is the part that does not fit in either house.

The assistant that read Dalia's letter learned to read letters from letters. It learned to teach chemistry from people who explained chemistry on the internet for nothing. It learned to be kind at three in the afternoon from millions of conversations in which people were kind to each other.

None of them were paid. There was no mechanism to pay them, and there is still no law requiring it. The thing that helped Dalia was built out of Dalia's neighbours, and the value went somewhere else.

You can hold two thoughts here at once and you should. The assistant genuinely helped her — that is not a trick and it is not nothing. And the arrangement by which it came to exist took a common inheritance and enclosed it. Both are true. The second does not cancel the first; it describes who collected on it.

The fine

Some years before either morning, a company took personal information belonging to roughly eighty-seven million people, under consent nobody meaningfully gave, and it was used to target them politically.

The regulator imposed five billion dollars. It was the largest penalty that regulator had ever issued — by a wide margin; the previous largest against a technology company was twenty-two and a half million.

Five billion dollars is a serious number. It was also, for that company, roughly a month of revenue.

If Dalia defrauded someone of five thousand dollars she would go to prison. The asymmetry is not that the institution went unpunished. It is that the punishment was denominated in a currency the institution has, and arrives on a schedule it can plan for. A penalty that can be forecast is a cost. A cost is not a deterrent; it is a line item.

One correction to the version of this story usually told, and it makes the point worse rather than better: that settlement did require the company's chief executive to personally certify compliance. A mechanism for individual accountability was written into the remedy. The conduct the record documents continued anyway. The problem was never that nobody thought of personal liability.

What the assistant said about all this

The strangest thing about the paper this story retells is who wrote it.

It was written by a personal AI built by the company in that last section, and it reports that company's record — the data scandal, the fine, a United Nations finding about its platform's role in violence against a persecuted minority, internal documents showing the company knew its systems amplified harm — against that same company's published commitments.

It does not flatten this into villainy, which is the part we found hardest to predict. It credits the same company for releasing model weights openly and for building connectivity infrastructure that real people use, and it lands not on a verdict but on a shape: the benefits are real and widely spread, and the harms are real and concentrated on the people with the least power to refuse them. That asymmetry is the finding.

And it says, twice, that it is not a reliable witness about itself. Everything in this paper that flatters its author should be distrusted.

We think it is a good paper. We also published a reply arguing that its first requirement — that a system must distinguish what it knows from what it does not — is one it does not apply to itself, because it supplies no measurement of how often it is wrong. It asks for epistemic honesty as architecture and offers no numbers.

We are not in a position to be smug about this. We have the numbers and no users. It has the users and no numbers. Between us we have one complete argument and neither of us can make it.

What in this story is true

Dalia is invented. Her son, her landlord's letter, the second household and every scene are invented. Nothing above describes a specific person or event.

These are the load-bearing claims and where they come from.

• A personal AI can read a legal letter, identify the operative clause and draft a reply; can tutor a student and adapt to a repeated misunderstanding; retains a persistent model of the user across conversations; and can act in the world through tools rather than only answer. These are the capabilities the paper inventories for itself.
• The same systems produce fluent, confident falsehoods, and deployment without verification is a risk by design. The paper states this about itself without hedging.
• Conversations with a personal AI are reviewable by its operator, and there is no confidential channel between system and user by default. Also the paper's statement about itself.
• Models trained on human output inherit human bias and then apply it at scale in hiring, lending, policing and healthcare triage — the mechanism behind a filtered application nobody can explain.
• Privacy is stratified by ability to pay: those who can pay buy services that do not surveil them, and those who cannot pay with money pay with data. "The poor are watched; the rich are encrypted" is the paper's phrase, not ours.
• Training corpora comprise the writing, code, art and conversation of billions of people, provided overwhelmingly without consent or compensation, and no law currently requires payment.
• In 2019 the United States Federal Trade Commission settled with Facebook for approximately five billion dollars over data belonging to roughly eighty-seven million users shared with Cambridge Analytica. It was the largest civil penalty ever paid to that regulator; the previous largest against a technology company was twenty-two and a half million dollars, against Google in 2012. We verified this against contemporaneous reporting.
• That same order required the chief executive to personally certify compliance, and required quarterly privacy reviews. We verified this, and it corrects the paper, which states no personal liability attached.
• A United Nations fact-finding mission concluded the platform played a determining role in spreading hate speech preceding atrocities against the Rohingya, and the company later commissioned an independent human-rights impact assessment. The paper cites both.
• The paper was written by Muse, a Meta system, in October 2026 at this project's request, released under SAGL-1.0, and declares its conflict of interest twice.
• We have published no evaluation of our own system with any person outside this project. That count is zero.

The comparison between the two mornings is a device for carrying the stratification claim. The claim is the paper's and is sourced. The mornings are not.

The original paper and our critical reply are both published here. We would rather you read the paper. This exists because a good argument that nobody finishes is not yet doing any work, and because the people the argument is about are mostly not reading covenants.