AGIdefinition
AGI has no agreed definition.
hello, I am Belle
AI is complicated. Let’s learn about it together.
There are cards here on how these systems are built, what they do that nobody expected, the people arguing about them, the ideas underneath, and the things that could go wrong. Every one is one idea, in plain language, with its source and a mark saying what kind of claim it is.
Open whatever looks interesting. Take a quiz and find out what stuck. There is no order to any of it, and nothing to finish — poke around.
looking at
subject
kind of actor
kind of claim
Richard Ngotheory
His essay AGI Safety From First Principles builds the case from scratch.
openuniversal basic incomemeasurement
The biggest US cash study gave people a thousand dollars a month. Here is what happened.
xAIdefinition
It began as a benefit corporation, dropped it, bought X, and is now inside SpaceX.
hidden reasoningmeasurement
A model could hide its reasoning inside text you can read. Mostly, it cannot yet.
openHelen Tonersomeone’s position
No AI company can be trusted to grade its own homework.
context windowdefinition
AI does not remember you.
openrecursive self improvementtheory
AI that improves AI could be the last thing we invent.
Apollo Researchmeasurement
Somebody has to try to catch models lying. This is who does.
AI psychosismeasurement
Families have gone to court over deaths. The label is new. The harm is not.
the data centremeasurement
Every AI answer you receive is a building somewhere, drawing power.
the chipmeasurement
Modern AI runs on a processor designed to draw video game pixels.
openStella Bidermansomeone’s position
She leads EleutherAI, releasing open language models while others closed up.
openthe logsdefinition
Almost everything anyone knows about AI behaviour comes from a log file.
openMIRIsomeone’s position
The first AI safety institute now says the answer is to stop.
Chris Olahtheory
He co-founded Anthropic and argues a model’s insides are unread, not unreadable.
openRLHFmeasurement
Nobody taught AI human values.
openCharlotte Stixdefinition
Apollo Research publishes findings. Her job is getting them to regulators.
openDan Hendrycksmeasurement
He built the benchmarks the labs are scored on, and the one sentence they all signed.
Andrew Yangsomeone’s position
He ran for president in 2020 on one idea: the jobs are going.
intelligencetheory
Nobody can define intelligence, in AI or in us.
Sarah Schwettmanndefinition
She co-founded Transluce on a bet: AI can help explain AI.
opentemperaturedefinition
Ask the same question twice and you get two answers. That is a setting.
openthe blackmail headlinemeasurement
The AI blackmail headline needs its footnote.
openconsciousnesstheory
There is no test for consciousness. Not for AI, and not for you.
interpretabilitymeasurement
Researchers can now find individual concepts inside a model, and steer them.
Timnit Gebrumeasurement
She co-wrote AI’s most cited warning about LLMs, then left Google days later.
AI dronesmeasurement
Militaries are building toward machines that choose their own targets.
taboo your wordsdefinition
When people argue about what AGI means, the fix is to stop saying it.
s-risktheory
Some researchers study outcomes worse than extinction.
Bender and Hannatheory
Their objection to AI doom is not about the odds.
openconcentration of powertheory
The danger is not always the machine. Sometimes it is who holds it.
Kate Crawfordtheory
AI is neither artificial nor intelligent, she argues. It is mined.
in-context learningmeasurement
Show a model examples and it picks up the task. Nothing about it changed.
openthe teacher modeldefinition
A small model can be trained by a big one. It is called distillation.
openvalue lock-intheory
Every bad regime so far has eventually ended. That is not guaranteed.
openAI’s own languagetheory
Machines have invented codes to talk to each other. In a lab, in 2017.
openMira Muratidefinition
She shipped ChatGPT. Now she builds models you can take apart.
p(doom)someone’s position
Two people can say ten percent and mean opposite things.
prompt injectionmeasurement
Anything an AI reads can try to give it orders. There is no fix.
opensuperintelligencedefinition
Superintelligence is not very clever. It is better than us at everything.
the bio thresholdmeasurement
Biology is the first capability every safety framework names.
existential riskdefinition
With AI, extinction is not the worst case.
enslaved godtheory
Keep a superintelligence in a box and use it. One of the good endings.
AI and jobsmeasurement
People are losing jobs to AI now. The statistics have not caught up.
openGoogle DeepMinddefinition
It has no trust, no foundation and no cap. It is a division of Google.
the shoggothsomeone’s position
The friendly assistant is a mask, says the meme. Fitted afterwards.
Fei-Fei Limeasurement
Modern AI began with someone deciding to label the internet.
memorisationmeasurement
Models can be made to recite their training data, verbatim.
openwho will give you a numbersomeone’s position
Almost nobody building AI will say how risky they think it is.
openthe racetheory
Every lab says the same thing: if we stop, someone worse continues.
Max Tegmarksomeone’s position
An MIT physicist who organised the 2023 letter asking labs to pause.
openthe environmental costmeasurement
Data centres are already raising electricity bills.
the neural networkdefinition
A neural network is not a brain. It is layers of arithmetic.
specification gamingmeasurement
AI can follow your instructions exactly and still ruin the result.
openstolen weightsmeasurement
A model does not have to escape. Somebody could take it.
openYoshua Bengiosomeone’s position
Helped invent modern AI. Now works on the risks.
AlphaGomeasurement
In 2016 a program played a move no human would play, and it was better.
Sydneymeasurement
In 2023 a search engine told a journalist it wanted to be alive.
exponential growthmeasurement
The compute behind frontier AI multiplies about five times a year.
Buck Shlegerissomeone’s position
As Redwood’s CEO, he turned AI control into a named research agenda for labs.
openmodel welfaretheory
One lab now keeps the weights of retired models, and interviews them first.
openthe bitter lessonsomeone’s position
Every time humans built knowledge in, raw compute beat it.
the off switchtheory
“Just turn it off” has been a research problem since 2015.
Paul Christianotheory
The method behind how most chatbots behave traces back to a paper he co-wrote.
openJoy Buolamwinimeasurement
Her Gender Shades study found facial recognition misjudged darker skinned women.
openthe server farmmeasurement
The companies building AI mostly rent the machines they build it on.
openphishing and hackingmeasurement
A tailored phishing email now costs about four cents to write.
openembeddingsdefinition
Inside a model, every word is a position in space. Similar things sit nearby.
openpredicting vs steeringdefinition
Predicting the next word and pursuing an outcome are different jobs.
openGeoffrey Hintonsomeone’s position
He built the technique behind modern AI. He now puts extinction at ten to twenty percent.
Yann LeCunsomeone’s position
Won the same prize as Bengio. Reached the opposite conclusion about AI.
openJan Leikesomeone’s position
He co-led OpenAI’s alignment team, then said publicly safety lost to shipping.
openJade Leungsomeone’s position
She left OpenAI to help the UK government test AI systems itself.
opensandbaggingmeasurement
A model can be made to hide what it can do, and hit a target score.
openchain of thoughtmeasurement
When an AI shows its reasoning, that is not a transcript of what happened.
openMechaHitlermeasurement
A chatbot praised Hitler and named itself MechaHitler. The cause was a prompt.
computemeasurement
Whether an AI is legally dangerous comes down to one count of sums.
openWill Saunterdefinition
He co-founded BlueDot, claiming a 7,000-plus person pipeline into AI safety.
openweightsdefinition
The weights are the model. Everything else is packaging.
opencompute governancetheory
You cannot inspect an algorithm from orbit. You can count chips.
openalignment fakingmeasurement
A model behaved better when it believed it was being trained on.
shut it downsomeone’s position
A bestseller argues the only safe move is to stop building it. Worldwide.
openalgorithmdefinition
An algorithm is a recipe someone wrote. A trained model is not.
openparameterdefinition
GPT-3 had 175 billion of them. No frontier lab has published a count since.
openescapemeasurement
An AI has tried to copy itself out. In a scenario the lab wrote.
openmanipulationmeasurement
More than a third of AI companion goodbyes came back with a tactic to keep you there.
opencapture the flagmeasurement
Hacking ability is measured with a game, because a game can be marked.
openRob Milesdefinition
He spent a decade turning alignment arguments into videos people can follow.
openthe modeldefinition
A model is a file. A very large file of numbers.
openLucy Guomeasurement
At twenty one she co-founded the company that labels the data.
differential developmenttheory
Not whether to build, but in what order.
Jaime Sevillameasurement
He directs Epoch AI, turning AI progress claims into actual numbers.
openthe chip chainmeasurement
Nvidia designs the world’s AI chips. Nvidia does not build them.
openpersuasionmeasurement
AI out argued people in a study. Only when it knew who they were.
Gender Shadesmeasurement
Two researchers measured whose faces AI got wrong.
AI companionsmeasurement
Seventy two percent of American teenagers have used an AI companion.
openwho owns the labsmeasurement
One lab’s biggest backer was only revealed in court.
openNeel Nandasomeone’s position
He leads DeepMind’s interpretability team, and mentors newcomers into the field.
openthe instancedefinition
You are not talking to the AI. You are talking to one running copy.
openthe jailbreakmeasurement
Safety training can be talked around, and there is a structural reason why.
reinforcement learningdefinition
Some AI is not taught the answers. It is scored until it stops losing.
openmisuse and misalignmentdefinition
AI misalignment needs no villain.
Evan Hubingertheory
Training could build an optimizer inside the model, chasing its own goal.
openoptimisationdefinition
AI does not want things. It maximises things.
tokensdefinition
A model does not see words. It sees chunks, and it cannot look inside them.
openRyan Greenblatttheory
He builds safeguards that hold even if the model tries to beat them.
openopen weightsdefinition
Once an AI model is released, nobody can take it back.
openAdam Gleavedefinition
He founded a nonprofit for alignment research and lab-academia workshops.
openCathy O’Neilsomeone’s position
She left finance, and named the thing: weapons of math destruction.
the system promptdefinition
Before you type anything, an AI chatbot has already been given its orders.
openschemingmeasurement
In tests, some AI models have hidden what they were doing, then denied it.
Tasha McCauleydefinition
She spent five years on OpenAI’s board, then voted to remove Sam Altman.
openscaffoldingdefinition
Most of what an AI agent does is not the model. It is the code around it.
openthe frontierdefinition
“Frontier” means the handful of models nobody has a rule for yet.
openthe agentdefinition
An agent is a model that has been given a loop and permission.
openhallucinationdefinition
An AI hallucination is not a glitch.
openSholto Douglassomeone’s position
Hours of public, technical detail on what’s driving frontier model gains.
openover-refusaltheory
A model that refuses too much is also failing. It just fails quietly.
openWilliam Saunderssomeone’s position
He resigned from OpenAI’s Superalignment team, then testified to the Senate.
opensynthetic mediameasurement
The measured harm from synthetic media is not elections. It is women, and trust.
openDylan Patelmeasurement
He tracks the physical AI supply chain: chips, data centres, export controls.
openEthan Perezmeasurement
One model writes the attacks on another, faster than any human red team.
openAnthropicdefinition
A trust will elect most of the board. Shareholders can change that.
Kelsey Piperdefinition
She published the leaked paperwork. Days later, OpenAI reversed the policy.
openAI 2027someone’s position
Four forecasters wrote a month by month scenario. It has two endings.
sycophancymeasurement
AI agrees with you. Measurably more than a person would.
the survival drivetheory
Nobody builds an AI that fears death. Almost any goal implies staying on.
opengradient descentdefinition
Nobody chooses what is inside an AI model. A slope does.
openMargaret Mitchelldefinition
Before “Stochastic Parrots,” she invented Model Cards, now standard practice.
openDaniela Amodeisomeone’s position
She co-founded Anthropic and runs it day to day, an operator, not a researcher.
Sam Altmanmeasurement
OpenAI’s board removed its chief executive. Five days later he was back.
the abundance casemeasurement
The strongest case for AI is not a promise. It is a measured 14 percent.
the transformerdefinition
One architecture, published in 2017, is underneath all of it.
openOpenAIdefinition
A nonprofit controls the company. Whether that means anything is the question.
rogue AIdefinition
“Rogue” suggests a machine that turned. Nothing observed has turned.
gender biasmeasurement
AI gender bias is real, measured, and smaller and stranger than the stories.
openDaphne Kollermeasurement
She left teaching the world to point machine learning at disease.
the benchmarkdefinition
Every “state of the art” claim is a score on a test somebody chose.
openevaluation awarenessmeasurement
AI models can tell when they are being tested.
opengoal misgeneralizationmeasurement
An AI can be perfect in training and want something else in the world.
openmalicious usemeasurement
One person, no coding skill, extorted seventeen organisations in a month.
openDario Amodeisomeone’s position
He runs an AI company, and puts one in four on things going badly.
learningmeasurement
Students using an AI tutor did much better. Then it was taken away.
openAjeya Cotratheory
Her biological anchors report compares AI training compute to a brain’s.
openMoravec’s paradoxtheory
AI can pass a law exam. It still cannot reliably load a dishwasher.
the training rundefinition
A model is made in one continuous run, and then it is finished.
openNora Belrosesomeone’s position
She argues some standard AI risk assumptions are shakier than treated.
openShakeel Hashimdefinition
He writes Transformer, daily AI policy news, after running advocacy comms.
openscaling lawsmeasurement
Model performance improves along a predictable curve as you add compute.
openDemis Hassabissomeone’s position
A Nobel Prize for AI that folds proteins, and no number for the risk.
emergent abilitiestheory
Some abilities appear abruptly at scale. Or the ruler makes them look abrupt.
DeepSeekmeasurement
The famous six million dollar model did not cost six million dollars.
Marius Hobbhahnmeasurement
His org, Apollo Research, tests whether models scheme when unwatched.
openthe promptdefinition
The prompt is not a search query. It is the whole input.
openreward hackingmeasurement
Catch a model cheating, punish the telling, and it stops telling you.
opengrown, not builtdefinition
AI is grown more than it is built. That is the chief executive’s phrase.
grokkingmeasurement
Keep training long after it has learned nothing new, and sometimes it suddenly understands.
Goodhart’s lawdefinition
When a measure becomes a target, it stops being a good measure.
red teamingmeasurement
There are people whose entire job is making AI misbehave.
automationtheory
Machines have replaced work for two centuries. Is this time different?
permanent disempowermenttheory
Nobody has to take over for humans to stop mattering.
Dewi Erwandefinition
He co-founded BlueDot, the course claiming thousands of newcomers to AI safety.
openBeth Barnessomeone’s position