How bad, and how it happens, are two separate questions
Public argument fuses them constantly. One person says "AI risk" meaning a chatbot giving bad medical advice. Another means humanity losing control of its future forever. They use the same words and talk past each other.
Severity and cause are independent. Misuse can produce a small harm or a permanent one. Misalignment can produce a trivial bug or a catastrophe. You cannot read the severity off the mechanism, or the mechanism off the severity. So this is a grid, not a ladder.
Pick a square
Each one is a different combination, and most public arguments are two people standing in different squares.
The word that causes the most trouble
Existential does not mean everyone dies. In the framing used in this literature it means the permanent destruction of humanity's long term potential. Extinction is one way that happens. A permanent unrecoverable dystopia is another. So is permanent stagnation.
This matters because people hear "existential risk," picture extinction, decide it sounds like science fiction, and dismiss the whole category. The claim being made is usually broader and, to its proponents, more plausible than the one being dismissed.
The disagreement that is actually about priorities
There is a real and serious argument that attention to speculative long term risk pulls resources and political will away from harms happening now: discrimination in deployed systems, labour effects, surveillance, environmental cost, and the concentration of power in a handful of companies. Researchers who make this case are not denying that future risk exists. They are making a claim about attention.
Researchers on the other side argue the two are complementary, that the same governance capacity serves both, and that waiting for a harm to be measurable is a poor strategy for harms that are irreversible.
The grid is not mine. Drawing catastrophic risk on two axes rather than one ladder is Bostrom and Cirkovic's move, from the introduction to Global Catastrophic Risks in 2008, where risks are plotted on scope against intensity. The severity axis here, including the insistence that existential is broader than extinction, follows Bostrom's 2013 four class typology. The causal axis is assembled from others: misuse and misalignment are common currency in the field, and the structural category is specifically Zwetsloot and Dafoe, 2019.
What is new here is only the pairing of these two particular taxonomies, the twelve cells, and the wording inside them. It is a recombination of existing work, and the MIT AI Risk Repository, which catalogues over 1,700 risks drawn from 65 identified frameworks, is a good reminder that no taxonomy in this area is canonical, including this one.
Someone is going to tell you the probability. Ask them what outcome they mean, by when, and whether they are counting the chance it never happens. Most of the time the number falls apart in your hands.
The number you will see quoted
Sooner or later someone tells you the probability. It has a nickname, p(doom), no agreed definition, no resolution date, and no way for anyone to be calibrated on it. That does not make it meaningless. It does make it a weak instrument, and it is worth seeing why by building one yourself.
What the surveys actually found
Individual quotes travel further than survey data, which is unfortunate, because surveys of published researchers are the closest thing to a measurement in this area. Two findings are worth holding on to.
The first is that the range among people who work on this professionally is enormous, spanning several orders of magnitude. That spread is itself the most robust finding. The second is that answers move substantially when the question is rephrased, which is a documented result and a serious problem for treating any single figure as a measurement.
There is also a persistent gap between superforecasters with strong general track records and domain specialists, with the forecasters consistently lower. Neither group has feedback on this particular question, so neither can claim calibration.
Why the number is a weak instrument, stated fairly
Against quoting it. There is no agreed operationalisation, so two figures are usually not comparable. There is no resolution date and no feedback loop, so nobody has a track record on it. There is no defensible base rate to anchor against. And a single scalar destroys the structure of the underlying argument, which is where all the actual content lives.
For quoting it. Refusing to put a number on anything hides real disagreement behind vague language, makes positions impossible to compare, and lets people avoid committing to a view they are in fact acting on. Decisions get made either way, and an unstated probability is still a probability.
Quoting a figure without the definition it came with. Treating a person's estimate as though it were a measurement. And using someone's number as a badge of which side they are on, which turns a question about the world into a question about tribe.