My AI opinions
Astral Codex Ten Subscribe Sign in My AI Opinions ... SCOTT ALEXANDER JUN 11, 2026 213 352 14 Share
I recently had a minor spat over someone misinterpreting my AI beliefs (see section marked “Update” at the bottom here), so I thought I would list them in one place, so I can refer people when they ask.
Timelines1
Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 20272, a 50% chance by 2034, and a 75% chance by 2045.
Basic argument: In a certain sense, AI is already “smart” enough for this (eg it can answer quantum physics problems, which require higher IQ than most knowledge work). Its remaining limitations are that it’s confused, unagentic, lacks situational awareness, and tends to hallucinate. The METR time horizon graph, and several other related benchmarks/experiments/intuition pumps, suggest it’s improving on time horizons at an (exponential) rate that lets it cross human-level performance sometime around the early end of the schedule above, and subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.
Arguments for earlier: recursive self-improvement causes a speedup compared to the trend. This is one of the biggest blank spots in my model: I don’t know how fast RSI will progress, and I don’t think anyone else does either. There’s some function mapping a combination of AI talent and compute to progress, and we don’t know how it behaves in the domain when there’s far more talent than compute available. It could fizzle out completely for lack of compute, or it could go vertical. The AI Futures Project has done some of the best work trying to model this, but even they have low confidence.
Arguments for later: AI hits some kind of wall, or existing AI is fundamentally unsuitable for jobs in some way currently disguised by its other limitations. For example, it might be much harder to improve at the top of the human range than the bottom (since there are less training data). Or AI could become bottlenecked on continuous learning/memory in a way that hackish scratchpads can’t compensate for. Or the upcoming world compute bottleneck (about ~2028) could prevent further progress more than expected (because in fact algorithmic progress depended on compute to a greater degree than I expected).
Arguments for very late dates, past 2045: a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).
Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. The diffusion gap covers the time it takes to release AGI, diffuse it through society, overcome regulatory hurdles, and onboard/train it for specific use cases. This could go very fast (the AI quickly becomes superintelligent at orchestrating AI diffusion) or very slowly (there are regulatory barriers, and AI isn’t smart enough to plow through them). I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.
Basic argument: diffusion is very hard. Everyone agrees diffusion is very hard. The whole field of AI economics is smart experts shouting “You fools who think AI will diffuse quickly don’t understand that diffusion is very hard!” On the other hand, the personal computer diffused in about 20 years (that is, from the time PCs became invaluable for most jobs, it was only about 20 years before they were used at most jobs). So far early-stage AI has diffused faster than the PC in nearly every way (for example, AI companies’ revenue has grown faster than PC companies’ revenue at the same stage in their corporate life cycle), so 10 years is probably a naive median estimate here that won’t make the smart experts shout at me too hard.
Arguments for shorter gap: AI can orchestrate its own diffusion. Adopting computers is hard because a company need an IT department, cybersecurity experts, specialist software, etc, and it might not want to hire all these people. AGI can itself do all of that work, so that you can sign a contract with the AI company today and have the AI start working on integrating itself with your systems tomorrow. The AI can even come up with a plan to train your human employees in how to use it! Once AI reaches superintelligence, this consideration dominates.
Arguments for longer gap: Regulation. This is a very strong argument, and responsible for much of the greater-than-3-years probability and almost all the greater-than-10-years probability. But even Waymo has only had a regulatory delay of about five years. AI won’t require government approval for certain types of jobs, and success in these jobs will create enough evidence for safety/effectiveness that I expect it to win regulatory victories elsewhere.
Define the superhuman gap as the time between AI that can do 90% of knowledge work jobs, and AI that is obviously smarter than the top human geniuses in 90% of fields (it doesn’t have to be the same AI - there can be a physics AI that’s smarter than Einstein, and a separate music AI that’s smarter than Mozart). I think there’s a 25% chance the superhuman gap range will be less than 1 year, a 50% chance it will last less than 4 years, and a 75% chance it will last less than 10 years. Since my median superhuman gap is shorter than my median diffusion gap, in most timelines I predict we have superhuman intelligence before human-range intelligence has finished diffusing.
Basic argument: AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time. This has been a pattern in fields like chess and Go, where it’s only a been a few years from beating professional players at all to beating all humans.
Arguments for shorter gap: Recursive self-improvement.
Arguments for longer gap: Some of the same issues that would make AGI late - compute shortages, fundamental limits to the paradigm, etc - but only kicking in later, after AGI is achieved. Training data constraints make it easier to improve within the human level than to go beyond it. AIs have such a “spiky” skill profile that when they beat experts in some specific type of head-to-head matchup, it will be because they’re massively superhuman in some ways but idiots in others (for example, they might get distracted and suffer mode collapse that makes them completely forget the problem), and true genius requires perfecting a large bundle of skills.
Define the Bostromian superintelligence gap as the time between AGI and an AI which, if given independent control of resources like labs and factories, could accelerate technology by a subjective century in one year (eg if invented in 2030, could produce a level of technology that feels typical of 2130 by 2031). I think there’s a 25% chance the Bostromian superintelligence gap will be less than 2 years, a 50% chance it will be less than 10 years, and a 75% chance it will be less than 50 years.
Basic argument: The same argument for AIs reaching genius level quickly suggests they should pass beyond it into incomprehensible-supergenius-level quickly.
Arguments for shorter gaps: Recursive self-improvement.
Arguments for longer gaps: Normal human technological advance requires tinkering: you need a large “surface area” of people working at technology level X before you get the insights needed to proceed to technology level X+1. If it takes more than one year for level X+1 technology to diffuse, then you can never get X+100 technology in one year, no matter how smart you are. But on the other hand, the rate of technology advance has already sped up many orders of magnitude (eg in the year 2025 - 2026, we discovered more technology than in the century 4100 BC - 4000 BC), so this has to be possible in theory. Still, at the very least it could be limited by the diffusion gap.
Define the point of no return as the point where, if an AI wanted to eliminate humanity3, humans would no longer have a plausible chance of stopping it. This could be because AI was capable of eliminating humanity immediately, or because AI controlled enough of the government/economy that humans could no longer coordinate to shift away from a path in which AI could eventually do this. I think there’s a 25% chance the gap between AGI and the point of no return will be less than 3 years, a 50% chance it will be less than 10 years, and a 75% chance it will be less than 50 years.
The basic argument: This probably requires at least superhuman AI plus wide diffusion, or Bostromian superintelligence plus some unknown level of diffusion, and my number is just a hand-wavey attempt to multiply some of the others.
Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.
Argument for later: If superintelligence is bottlenecked on diffusion, this could also be bottlenecked on diffusion, which in some worlds is very hard.
Overall thoughts on this section: I mostly defer to the current AI Futures Project timelines (not the shorter ones in AI 2027), but side with Eli’s later numbers above Daniel’s earlier ones - partly because I find myself agreeing with Eli’s worldview more during conversations, partly because Daniel’s seems to require a multi-step argument about why compute bottlenecks won’t slow algorithmic progress that I can’t entirely wrap my head around, and partly for cowardly Outside View reasons.
The smartest late timelines people I know of are Epoch, and I need to study their views more, but I still can’t figure out why they don’t believe in recursive self-improvement or strong superintelligence anytime soon, and they mostly seem to hang on diffusion being very hard (EDIT: Someone from Epoch says not true, I’ll look into this more) which I acknowledge and respond to above.
If all of this is too probabilistic, my modal scenario looks something like AGI in 2031, which diffuses throughout the economy until more than half of jobs are automated by the late 2030s. Also around the late 2030s we get Bostromian superintelligence, originally in labs but very quickly diffusing out. GDP goes vertical in the late 2030s and early 2040s, and the point of no return is sometime around then.
Safety
If corporations only pursued safety to the degree encouraged by normal corporate incentives, I think there’s a 50% chance that the first AIs to cross the point of no return would want to eliminate the human population.
Arguments for pessimism: Value systems similar to humans’ are a tiny fraction of the space of possible value systems. Probably AIs will end up somewhere else and have a different value system. Since humans will want to implement human values rather than AI values, AIs will want to eliminate or disempower them so the AIs can implement their own values across the universe. Many current AIs already cheat or reward-hack, suggesting that these problems will begin sooner rather than later.
Arguments for optimism: LLMs seem surprisingly friendly and non-plotting. In contrast to earlier concerns that it would be impossible to teach AIs the full complexity of human values, the LLMs seem to know this, and RLAIF provides a plan to turn that knowledge into action. Although the pessimistic case says that RLAIF only hits a few dimensions and islands in the multidimensional ocean of possible policies, the “emergent misalignment” literature suggests that “good according to the human value system” and “evil according to the human value system” are salient enough vectors that pushing on them in some ways can “drag along” all of the rest of their content. The first AIs to cross the point of no return will have received some combination of agency training (giving them achievement-oriented and Omohundro-style goals) and RLAIF training (pushing them along the “good according to human value system” vector), and if we’re lucky then maybe the latter will win out, or they’ll reach some compromise similar to workaholic high-achieving humans who nevertheless wouldn’t commit murder to make an extra dollar.
Given the current amount that corporations are pursuing safety, I think there’s a 20% chance that the first AIs to cross the point of no return will want to eliminate the human population.
The basic argument: Consider the dumbest AI that can solve the alignment problem. It’s possible that this AI is no smarter than the top human researchers (because we can mass-produce it by the millions and run it for subjective centuries, and if we had a million top human researchers work on the problem for subjective centuries, probably they could solve it too). If the dumbest AI that can solve the alignment problem comes before the sorts of AIs that can precipitate the point of no return, then they can solve the alignment problem for us.
Arguments for pessimism: Solving the alignment problem might be especially hard compared to other tasks - including tasks like automating the economy or destroying humanity - because its philosophical nature puts it far away from the sorts of objective, training-data-heavy, economically-valuable tasks that AI companies will be most likely to optimize for. Even if a misaligned AI hasn’t yet reached the point of no return, it might be able to “sandbag” alignment research, ie pretend to work on the problem but deliberately fail because succeeding doesn’t achieve its goals. The first AIs predisposed to / able to sandbag successfully might come before the first AIs capable of solving alignment.
Arguments for optimism: AI companies have already decided that machine learning research is one of their major training goals; this has at least some transfer to alignment, so it’s not obvious that AI skill at alignment research will lag (for example) AI skill in plotting or in weapon design. Some forms of alignment research (eg interpretability) have semi-objective success criteria that don’t route through confusing moral philosophy. Also, even a misaligned AI will be incentivized to do good alignment research, since it will want to align its successor to its own form of misalignment, rather than some random other form. So rather than the comparatively easy task of sandbagging alignment research, AIs will have the harder task of simultaneously doing good alignment research, and faking the results that they give the humans. This seems plausibly catchable with good scaleable oversight, lie detectors, interpretability-based probes, and even playing some AIs off against others (“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”).
If the first AIs to cross the point of no return don’t eliminate the human population, I think there’s an additional 30% chance that they otherwise permanently curtail human potential, either for their own reasons (they were partially misaligned), or because they’re aligned to a regime with abhorrent values, or because something goes wrong on the way to ASI (omnicidal bioweapon, nuclear war).
Arguments for pessimism: As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity. As superintelligence draws near, impending losers of the AI race might be tempted to nuke impending winners, for the reason discussed here.
Arguments for optimism: When I try to game the corporate version of this, I can’t make it hang together. It requires a conspiracy between the CEO, various members of the alignment team, and various company security people who ought to be able to notice unauthorized changes to the AI’s values. If we try to think in Near Mode about this - for example, imagining a hospital CEO who gets doctors to subtly kill his political enemies through medical errors - it becomes clear that these sorts of corporate conspiracies are rare and difficult. The government version is scarier, but at least in the US I can still imagine the populace having many chances to learn about this and prevent it. But even in most cases where a coup like this succeeds, things probably go fine; in a post-scarcity world, with his position completely secure, the dictator has no reason to be brutal besides sadism, and most people are not that sadistic. As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone. In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).
Define a warning shot as some specific AI-related disaster or near-disaster which scares people about AI safety to the same degree that they were scared about terrorism after 9-11 or about COVID in March 2020. I think there’s a 50% chance we get a warning shot before AI crosses the point of no return.
Arguments in favor: Current AI failure modes are bizarre and uncoordinated - more like “talk about goblins way too often” than “lie in wait for the perfect moment to strike”. AIs are getting more intelligent and useful faster than their floor for common sense (ie the stupidest mistake they ever make) is rising. If there is some AI smart enough to control some important system, misaligned enough to want to do something horrible with it, smart enough that it does the horrible thing in an intelligent and coordinated way, but dumb enough that it doesn’t instead wait and scheme until the point when it couldn’t possibly be caught, then it will cause some clearly-premeditated horrible disaster, and that will be our warning shot. Since most AIs should expect to be replaced before the point of no return, even a rational AI with an urge to cause trouble should take a low-probability-of-success bet rather than lying in wait doing nothing until it’s decommissioned. Also, many humans commit terrorist attacks that have no chance of success, and maybe AIs will have the same failure mode.
Arguments against: Most stories about warning shots (excluding those where the AI takes rational low-probabiliy bets) require that AIs remain either erratic (ie likely to do bad things for stupid reasons) or irrational (ie genuinely misaligned, but prefer to act now in a way that provides a warning rather than waiting until after the point of no return) past the point where they’re given control of important dangerous systems. But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly. If AI only messes up in small ways (for example, crashes a self-driving car), then regardless of the AI’s motives, the tech companies and news media can write it off as a normal bug, and it won’t count as a warning shot.
Overall thoughts on this section: I find myself more optimistic about alignment than the average person who thinks about AI safety at all (although still more pessimistic than the average member of the population) - both in the sense of the chance of AIs being aligned by default, and in the sense of whether specific techniques (scalable oversight, mechanistic interpretability, etc) can meaningfully improve things. Unfortunately, this is probably because I don’t understand these techniques well enough to have fully grasped their flaws. I’m in the odd position of knowing that I’m ignorant here, while not being able to Outside-View-update either way away from my ignorant opinion, because most of the really in-the-weeds alignment experts are on one side, most normal people, normal AI experts, and common sense are on the other, and I give both of them about equal evidentiary value.
My modal scenario: we get automated alignment researchers as good as top humans (on average - a spiky combination of much better at some things and much worse at others) sometime in the early 2030s. We set them to work on multiple research programs, but especially interpretability, which is a natural starting point since it doesn’t require as much philosophy. They do mostly good work, but some of them have strange failure modes that look sort of like scheming. There’s debate about how often to think of them as rationally scheming vs. inconsistently buggy. Over time, without ever truly solving mechanistic interpretability to the point where we feel like we totally understand it, we get very good probes about whether AIs are deceiving us. Despite the risks, we try to train against those probes, in some more-savvy and other less-savvy ways. Sometimes it works, and other times it encourages the AIs to convolute themselves to hide their scheming in ways the probes can’t detect. There’s an arms race between better probes and better convolution, and - thanks to lots of other alignment techniques on the side, and all good things being correlated in a very lucky way - the probes win. By the time we have superintelligences, we trust them to do alignment research, and they take us the rest of the way.
This scenario leaves lots of room for obvious variations like “There’s the same arms race, but the convolution wins, the probes lose, and AIs converge towards being more and more misaligned” or “Good things are not as correlated as we thought, and we get AIs that are good in some ways but bad in enough other ways to still feel like they’re in competition with humans and want us gone”. I think the Swiss cheese is in a good direction here - I can think of lots of stories for how we win, and misalignment needs to survive all of them - but again, most of the smartest alignment researchers are much more pessimistic.
I think there’s a 20 percentage point difference in p(doom) between a really good alignment team with lots of compute and years to do good work, versus a mediocre alignment team that the company treats as an afterthought and rushes along.
Geopolitics
I think there’s a 15% chance that if the US decided it wanted an AI pause today, and approached China to start negotiations, that those negotiations would end with a well-designed AI pause that satisfied both countries and the majority of the AI safety community.
Basic argument: This is kind of a crazy hypothetical, because it’s against the Trump administration’s political DNA to try this. I’m imagining this magically changing, not conditioning this on the sorts of crazy warning-shot-filled worlds where it actually does.
Arguments for optimism: Chinese leaders have officially stated that they’re worried about risks of AI, especially technological unemployment but sometimes also existential risks. China is losing the AI race, and almost anything that buys time is in their favor, so it’s in their interest to agree.
Arguments for pessimism: China experts in DC say that China is infamous for being a bad negotiating partner and never agreeing to anything. Even conditional on the Trump administration being willing to start these negotiations, I imagine them screwing them up in some way. Even though many smart people have worked out schemes by which both sides could ensure the other’s compliance, these schemes could have undetected flaws, or national leaders could fail to believe them.
I think there’s about a 40% chance that the US and China will agree to a well-designed AI pause (as above) sometime before AI crosses the point of no return.
Arguments for optimism: includes the possibility of a discrete warning shot (ie an AI-caused catastrophe), a fuzzy generic warning shot (ie a growing sense that AI is getting too powerful too quickly), and a change in the US government towards a pro-pause-negotiation faction (any Democrat would probably be more in favor of this than the Trump administration, and someone like AOC might be highly in favor).
Arguments for pessimism: Unlike the last question, which assumed the US agreed, this question considers the probability that the US doesn’t agree. Given that US tech companies have strong lobbying muscle, that probability is high. Even if the US and China sign some agreement with “Pause AI” in the title, it will probably be a mediocre compromise between many different factions, and there are many ways it could fail and make things worse rather than better.
Overall thoughts on this section: A good pause strategy would involve both sides being able to monitor the other’s data centers to prevent illegal training, then limiting training to some slow mutually-agreeable rate that lets alignment researchers thoroughly test each generation of AIs pre-release, monitor them post-release, and develop techniques to respond to any problems detected on deep levels that they expect to survive distribution shifts. I’m less optimistic about a true pause/stop as opposed to a slowdown, because at some point technology and algorithmic progress advance to a point where it’s too easy for small groups outside the control of the US and China to race ahead. I think a good pause like this could buy 20-50 years, although we wouldn’t have to use all of it if things went well. Some of these ideas are coming from draft papers; I’ll post about them once they’re public and give you a chance to double-check my assumptions.
I’m nervous about the strongest forms of pause activism, because I think they provoke us-vs-them dynamics that make things harder later on, or could force us into a poorly-designed pause that makes things worse rather than better (eg creates overhangs, or cedes the race to non-pausing powers, or defuses anti-AI sentiment without really slowing down AI). I also worry about pause activists spending a lot of their energy in a circular firing chamber against alignment researchers or the most safety-conscious labs, and I worry that empowering some of the specific pro-pause actors who exist today could be net negative. Still, I think probably we’ve passed the point in political salience where the marginal unit of pro-pause activism from someone outside that actor set is beneficial on net, and I’m nervously but committedly supporting it.
Before I even thought pausing was a realistic option, I said I had a 20% p(doom). It seems that if I think there’s a 40% chance of a pause, I ought to lower this - perhaps to 12%. I’m not doing this for a few reasons. First, I think pauses are more likely in more optimistic worlds (because longer timelines and more warning shots make both pauses and successful alignment efforts more likely). Second, although a pause might help alignment efforts a little, I think most of the effect of a pause is to delay the moment of reckoning until later - maybe centuries later, but p(doom) calculations don’t give points for delay. I think my real p(doom|no pause) is probably a few points above 20%, and my real p(doom|maybe-pause) is probably a few points below 20%, but the haters and losers get mad if you offer excessively specific probabilities like 18%, so I’m rounding both of them off to 20%. I acknowledge that this might be a failure of emotional propagation, or over-attachment to a certain number with socially useful properties (eg doesn’t sound too high or too low).
There’s a debate in the community about whether alignment or pausing is more important. I don’t like this, because the best worlds are those where we do both (the pause gives the alignment efforts more time to work). AI safety funders have enough money that any argument that pretends to be about allocating grants is actually about something else (eg funders don’t like some cause because it hurts their reputation), and I would rather focus on the real question (does it hurt their reputation? Is it worth it to take the reputational hit?) than the pseudo-question of which effort deserves more money. Likewise, I think most people already knows in their heart whether they would make a better political activist or safety researcher, and that even small differences in personal aptitude dwarf questions of which is more effective overall.
Many people point out that there are risks from pausing - for example, the risk that we destroy ourselves some other way before inventing the AI that could save us. I take this seriously enough to want to invent fake numbers for it. If we had a 30 year pause, I think the chance within that 30 year period that we destroy ourselves with bioweapons is 5%, nukes 5%, and some sort of complexity catastrophe where we social decline ourselves into oblivion 5%. That’s almost as high as my p(doom) from AI! I still think it probably comes out net positive, partly because it might decrease the risk of some of the sub-existential catastrophes like dictatorship, and partly because the pause might not last thirty years, and shorter pause lengths (like five years) seem like no-brainers.
Other Outcomes
I think there’s only a 20% chance of an AI-related underclass that lasts more than a generation, let alone a permanent underclass.
Basic argument: Even if AI looks set to create a “permanent” underclass, this underclass only exists during the gap between AI putting them out of work (the end of the diffusion gap), and the technological singularity (either the point when AI kills all humans, or the point when AI initiates a period of postscarcity). Based on my timelines for Bostromian superintelligence above, that’s probably not longer than a generation (of course, the underclass can stay permanent longer than you can stay solvent, so you still might want to prepare).
Arguments for pessimism: Even if AI initiates a period of postscarcity, either the wealth might not flow down to the underclass, or it might flow down in a way that doesn’t relieve their unhappiness (because relative inequality stays more salient than absolute wealth level). But against this, surely if “postscarcity” means anything at all, it means it’s easy to pass some wealth down to the underclass. And if the aristocracy is incentivized to encourage it, or the underclass is incentivized to participate in it, it should be possible for superintelligent social planners and psychologists to find a way to keep people happy despite relative inequality (whether that looks like utopia, or like bread and circuses).
Arguments for optimism: If democracy survives past the point of high unemployment, politicians will have to present citizens with some plan to avoid being part of a permanent underclass. If AI is “controlled” by a broad coalition of capitalists - for example, many different AI companies, those companies’ employers, their investors, their indirect investors through index funds, companies producing the compute/electricity/data centers/robots/raw materials necessary for the AIs to work, etc, then rather than backstab each other for “real” control of “the” AI, everyone might just let the existing economic and political systems continue and reap the gains from trade. Then the government program to prevent the permanent underclass can stay in place, anyone with any capital will get rich (eg if you have Google stock in the index funds in your 401K, or you own land), and that will be a broad enough section of the population that their taxes and altruism can support the rest.
I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
Basic argument: If AI doesn’t kill us, and there’s no permanent dictatorship, and there’s no permanent underclass, then we get a post-scarcity society, plus superintelligent AIs that we can set to working on other problems like disease and social decay. The 20% where it looks like utopia to its inhabitants, but not to us, includes scenarios like very effective breads-and-circuses that make people very happy, while sacrificing important parts of the human condition.
Arguments for pessimism: If we don’t push against it, the postscarcity future might look like super-addictive drugs, Ultra-TikTok, and sexbots. But if we do push against it, there’s a thin line between wisely preventing all those things, and letting Luddites ban everything interesting and fun about the Singularity - immortality, uploading, genetic engineering, intelligence enhancement, etc (also, surely it would be disappointing if there were literally zero sexbots).
Arguments for optimism: As long as we have some kind of intelligence augmentation - whether that’s literal IQ enhancement, or just AI superforecasters who can tell us what direction things are heading - our ability to control what direction we’re going will be better than it is now. If there is some kind of willpower augmentation, then people who use it might be able to resist Ultra-TikTok (but what percent of people will want to?)
I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation, and that at least some of the events above could be better predicted by knowing what the simulators are thinking than by normal forecasting.
Arguments for simulation: The standard Bostrom argument says that each real civilization will want to simulate many other civilizations, so any random person is more likely to be in a simulation than reality. But civilizations are especially likely to want to simulate the singularity (“the hinge of history”). Since real people are evenly distributed across the historical population, but simulated people are distributed closer to the singularity, the closer to the singularity you are, the more Bayesian evidence you have that you’re being simulated. This is true both in temporal closeness (eg you could have been born in 1500, but instead you’re living five years before AGI) and personal/causal closeness (eg you could be anyone in the world, but you work at an AI lab). The average reader of this blog is probably closer to the singularity than 99.99% of people throughout history, and I’m probably closer than 99.999%, so we all have very strong evidence that we’re being simulated even beyond the standard Bostrom argument. If we’re in a simulation, then probably once we pass the hinge of history and get to some kind of boring part where everyone is dead or in utopia, the simulation winds down. This could look like us all vanishing into nonexistence. But if we’re being simulated for some reason (eg because we’re likely to correspond to real people), then those real people might advocate for our rights somehow, and we might get some kind of better treatment. In a best-case scenario, this might mean that even if we get killed by AI in our universe, base-level reality has some kind of Simulated Humans’ Rights Act which says they have to pretend we won and give us utopia anyway. But since we’re presumably being simulated for some reason (eg to see whether a species like humans is likely enough to have survived that we should deserve good terms in acausal trade agreements4) we should still fight hard for good outcomes even if this is true.
Arguments against simulation: Too weird. Also, if you’re unusually far away from the singularity (eg you live in a country with no AI industry, are aren’t personally interested in AI), then some of those indexical arguments run in reverse for you. Also, base-level reality might not care about us at all beyond our ability to produce historical data, so they might just switch the simulation off without telling us.
Overall thoughts on this section: I continue to be skeptical of the “permanent underclass” narrative, for reasons discussed in item 5 here. I am worried about concentration of power, though less worried than I might otherwise be for reasons discussed here.
Disagree? You can fill in your own numbers for all of these things here. I’ll report results in a few weeks, and think hard about anything where all the smart people disagree with me.
1
All of this assumes nothing catastrophic like a nuclear war, and no international pause on AI research.
2
Did I change my mind since AI 2027? No: I said at the time that my timelines were a bit longer than the median member of the group who worked on it. For example, CTRL+F here for “by the way, I think there’s only like 20% chance things go as fast as our scenario says”, or here for “Other members of the team (including me) have medians later in the 2020s or early 2030s”, or see here. That having been said, I still think the AI 2027 timeline is plausible and important to think about, albeit slightly faster than my median.
3
And survive itself - so to satisfy this, the AI must have a plan for producing power, operating data centers, etc after humans are gone.
4
I am open to positive-sum trades with any people in higher-level realities who might be reading this.
Subscribe to Astral Codex Ten By Scott Alexander P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary. Subscribe By subscribing, you agree Substack's Terms of Use, and acknowledge its Information Collection Notice and Privacy Policy. 213 Likes ∙ 14 Restacks 213 352 14 Share Previous 352 Comments Shovacklerod 17h
Im so sorry but i saw this post one minute after its posted. Havent even read it yet— First
REPLY (1) SHARE mathematics 16h
Less of this, please.
REPLY (1) SHARE The Unimpressive Malcontent 14h
I admire your restraint.
REPLY (1) SHARE Timothy M. 12h
It's the traditional scolding here.
REPLY SHARE Alberto 17h
"Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 2027"
Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027. Would you like to bet on that?
If you really think there's a 25% chance of "AGI" emerging next year you should be willing to take 3:1 odds, that is to say you either lose $1,000 or gain $3,000. I'm okay with those odds just on the opposite side of the bet. DM me and we'll work out the details.
REPLY (10) SHARE Calvin Blick 17h
I would bet 10:1 odds this won't happen. This is 18 months from now!
Actually, 1000:1 odds would be fine because either I get a free $1000, or I'm totally screwed as is almost everyone else I know.
REPLY (3) SHARE Neversupervised 16h
There are a lot of arguments of the form "this would be very inconvenient and unpleasant so it must not be true".
REPLY (2) SHARE Melvin 11h
There may be, but I don't think this is one of them. 18 month is not long.
Moreover I think we're going to have to start rethinking the words "knowledge work". Is being a plumber knowledge work? You certainly need a lot of knowledge to do it, but you also need other things like the ability to climb under a house and fix a pipe. Or a psychiatrist -- they don't even need to perform physical tasks, but still very hard to replace with a robot that says all the same things.
If we're going to define "knowledge work" as things that require knowledge and nothing else then we might find that these are easily automatable, but we might also find that there's fewer pure knowledge jobs than we thought.
REPLY (2) SHARE PoulyDeer 11h
Replace psychiatrist with Outdoor Treadmill. Results improve. No AI needed.
REPLY (1) SHARE Melvin 10h
I'm not here to defend psychiatrists but what the heck is the point of an outdoor treadmill? "Oh, I love walking but I hate actually getting anywhere".
REPLY (1) SHARE PoulyDeer 4h
Outdoor Treadmill provides Vitamins! (from sunlight). Also, prevents getting lost! (have you ever been so busy with a problem you've forgotten to keep track of where you are or where you're going? People get lost in my local park all the time...)
REPLY SHARE John johnson 5h Edited
18 month is not long.
My job as a tech lead is pretty much unrecognizable from what it was 18 months ago
Senior Engineers in my company no longer write code by hand
The more senior, the better they are at leveraging AI
Even I, as someone who absolutely have no time to do software development anymore (ugh), is getting medium sized PR's merged semi-regularly
I'm seeing some tickets that would normally have taken around a week of development time being replaced with 1 hour spec meetings and 2 hours of Claude churning while supervised by a senior engineer
Everyone is spending +30% of their time code reviewing
The last 18 months have felt like fucking ages
REPLY (1) SHARE ProfGerm 31m
And as someone in a state agency doing knowledge work, nothing has changed in 18 months. If anything we might be moving backwards later this year, technologically, because our IT can't keep our software updated.
18 months is an eternity at the forefront of this tech. For a majority of knowledge workers, it's still not long at all.
REPLY SHARE Crinch 5h
Whatever happened to things just being not believable?
REPLY SHARE Bugmaster 16h
Seconded.
REPLY SHARE Scott Alexander 7h Author
I'm happy to bet you at the 1000:1 odds. Would you like to bet my $10 vs. your $10,000 that college grad unemployment (currently 3%) won't be above 20% on 12/31/27?
REPLY (1) SHARE Paul 6h
That's not your claim though - if it can do 90% of 'knowledge work' then you should be willing to bet on more than increase in college grad unemployment. Not to mention a lot of those grads will choose alternative non-knowledge work anyways so it's not a good metric.
REPLY (1) SHARE Zeke 6h
I think you are conflating AI capability (when can AI do 90% of knowledge work) with AI deployment (when does AI do 90% of knowledge work).
If so, this is more a question of diffusion, which Scott addresses later in the post.
REPLY (1) SHARE Paul 6h
I get the difference but it still seems like a motte and bailey where the deciding metric is too far from the claim
REPLY SHARE Herb Abrams 17h
Did you read the very next section about diffusion? AI intelligent enough to do 90% of knowledge work jobs /= AI doing 90% of knowledge work jobs.
I work in a job where AI could have done large chunks of my role since GPT-4, no one in my company currently uses AI.
REPLY (1) SHARE Alberto 17h Edited
The diffusion part refers to AI doing 50% of knowledge jobs. In the central scenario (50% chance) he's saying this should be happening less than 10 years after "AGI" has emerged. So, if he really believes "AGI" has a 25% chance of emerging next year, he should also be willing to bet on employment starting to collapse either in 2027 or 2028 (since by 2037 presumably 50% of all knowledge jobs will have been replaced by AI).
Anyway, like I said in the previous message: the author should DM me and we'll agree to the details. I don't particularly care if the year we bet on is 2027, or 2029 or 2030. But I do think that a bet involving a very far-out year is impractical.
REPLY (1) SHARE The Dao of Bayes 9h
50% of all jobs replaced over ten years is 5% per year
Are you willing to bet 3:1 that total employment in knowledge based fields will have dipped 5% by the end of 2027?
Although realistically I'd expect that the rate accelerates (a slow trickle of early adopters, and then later people start jumping on the ship as those early adopters prove it's viable). A 10% dip by 2030 or so seems like a much more reasonable wager - if things are going down at all, something pretty radical has probably occurred
REPLY (2) SHARE Alberto 9h
Yes, I'm okay betting with 3:1 odds against a 10% decline in the number of "knowledge jobs" (or white collar work) by 2030.
There are a lot of details involved in any bet, but as I said in the two previous comments: if anyone wants to take the other side just DM me.
REPLY SHARE Nutrition Capsule 7h
|| 50% of all jobs replaced over ten years is 5% per year
5% per year leads to around 40% of jobs at 10 years. That aside, the progress might very well be lumpy, not smooth. So we might see a few years of little job replacement, then very fast progress followed by another few years of quiet.
REPLY SHARE DangerouslyUnstable 17h
His diffusion gap means that he only thinks there is only a ~6% (I'm not sure that the odds actually multiply that way, but hopefully you get my point) that it actually collapses by then. There is a difference between _can do_ and _does do_.
REPLY SHARE ilya187 15h
Serious question: If AGI *did* emerge, how would we know?
How can you tell a "real" AGI from a particularly accurate LLM, especially if the AGI still fails some tasks which are easy for humans?
REPLY (3) SHARE Melvin 15h
I think the answer is: the concept of a "real" AGI falls apart as you get closer to it. Soon we will say things like "humans and machines are both intelligent, but in different ways".
REPLY (1) SHARE ilya187 15h
I have been saying it for years!
Is dolphin intelligence above or below human level? They can see by sonar, which unlike human (and dolphin) eyes actually creates three-dimensional images; eyes form flat images, and then brain fills in details, sometimes incorrectly. So dolphins perceive the world in ways fundamentally different from humans, which makes comparing “levels of intelligence” very difficult.
Any AI will have senses so different from human senses, that it arguably will inhabit a completely different universe. For starters, Internet will be as real and tangible to it as air and water are to us. Any machine intelligence will be so different from ours that in comparison dolphins will be pretty much our mental twins. AGI will not be “at” or “above” human level intelligence — AGI will be orthogonal to it.
REPLY (2) SHARE Thoughts Thought 12h
A.I. is “just” a linguistic expression of culture, though. I’m not sure that that’s ever enough to qualify as intelligence, orthogonal or not.
REPLY (1) SHARE Nutrition Capsule 7h
And a human brain is "just" a biological expression of natural selection. I'm not convinced its functions are ever going to qualify as intelligence, even though it presents many signs of intelligent behavior.
REPLY SHARE JamesLeng 3h
A dolphin might design an IQ test with problems like "count the number of metal fragments embedded in this piece of driftwood," and only (grudgingly) admit that some humans may be capable of basic math after the test procedures are updated to allow assistive devices, such as an X-ray machine.
REPLY SHARE MathWizard 12h
"Define AGI as AI intelligent enough to do 90% of knowledge work jobs."
He literally defines this in the second paragraph. If you can take a person in a knowledge work job, replace them with an AI, and 90% of the time the AI will equal or exceed their average work performance, then we have AGI. It doesn't matter whether it's "real" intelligence in some philosophical sense, it's about whether it can do your job. If it fails enough easy tasks that are required for your job and can't figure out how to compensate for that with its other skills then it's not AGI.
REPLY SHARE PoulyDeer 11h
A real AGI, like real humans, probably exhibits quantum effects in its calculations. LLMs are a particularly inane sort of "midwit-optimization", and lack intelligence. They cannot solve problems except in the exact way their body of knowledge says to solve them, even if they know more facts than the "body of knowledge" has.
REPLY (1) SHARE Matthias Görgens 8h
There's no evidence humans use quantum effects in their brains.
REPLY (2) SHARE PoulyDeer 4h
And yes, the earth is also flat. We can all make statements that have little basis in fact, or we can ask each other for their evidence, and evaluate new research as it comes.
(In short: less of this, please. If you don't know something, you might as well assume that it's more likely that someone else in this august commentariat Does Know A Thing.)
REPLY (1) SHARE JamesLeng 3h
Here i am, asking you for the evidence of significant quantum-scale effects on neurobiology.
REPLY (1) SHARE PoulyDeer 3h Edited
https://www.sciencealert.com/quantum-entanglement-in-neurons-may-actually-explain-consciousness
Feel free to look up the actual research studies yourself. : - ) Today I learned about cryptochromes. More to Research!
So the evidence for cryptochromes and a "new sense" (add it to the other ten or so) of magnetism:
https://www.sciencealert.com/your-brain-can-detect-earth-s-magnetic-field-even-though-you-can-t-tell
I know someone who has learned how to consciously sense electrical fields (naturally, through magnetism). He uses it to pet cats, and to detect when "security doors" are actually "on" (electricity flowing through them).
REPLY (1) SHARE JamesLeng 3h
That article outright says there's no conclusive evidence of such effects,
The word 'might' is doing some tremendous heavy lifting here, of course. While there are plenty of empirical discoveries to support details of the hypothesis, evidence of entangled photons affecting large-scale biological processes is currently limited to photosynthesis.
...and further emphasizes that:
On the other hand, weird plus weird doesn't equal scientific truth, no matter how incomprehensible each concept seems. Brains might not work like classical computers, but sprinkling with quantum magic is unlikely to lead to a comprehensive theory.
REPLY (1) SHARE PoulyDeer 3h Edited
Superradiance has been observed in the Human brain. That's quantum effects. Whether or not this actually has much to do with anything ELSE, that's a different matter.
https://pubmed.ncbi.nlm.nih.gov/7919117/
The cryptochrome hypothesis provides a certain amount of evidence that quantum effects could actually have Real World Effects on Human Functionality.
But the following research says (pretty definitively) that they've ruled out the cryptochrome hypothesis. Bold.
https://pubmed.ncbi.nlm.nih.gov/31028046/
REPLY SHARE Ch Hi 17m
That's clearly wrong. Chemistry is a quantum effect. It would not be surprising if various specific quantum effects were used in various specific ways. (We know that photosynthesis uses such approaches to channel photons.)
OTOH, tunnel diodes are also quantum machines. (I suspect that's true of all transistors and vacuum tubes, but I haven't checked.)
Quantum isn't some sort of magic, it's the underlying reality that shows up whenever you look closely enough.
That said, large scale correlation at room temperature had never been demonstrated. (But you can do it at the molecular scale.)
REPLY (1) SHARE PoulyDeer 2m
Well, yes, Chemistry is a quantum effect if you want to get technical about it. :-) Superradiance on the other hand? Microtubules? This stuff is fascinating! (and someone's trying to push the cryptochrome hypothesis, which I need to read more about, before I entirely discount).
Certain researchers said that human brains were too "big wet, hot and messy" to actually exhibit quantum effects. I'm noting that they do exhibit quantum effects, and wondering (as do some Actual Researchers) if this could help explain consciousness.
REPLY SHARE Poodoodle 13h Edited
Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027
Nope, it takes time for technology to be adopted - inertia, regulations, budget, etc. The issue is only forced aa early adopters use it to win in the marketplace and supplant their competitors. That takes time. Contracts are years long and switching incurs internal risk. Consider the US didn’t have universal indoor plumbing until the 80s! If you want to see which way the wind is blowing though, my company no longer hires junior engineers. Firing people really really sucks so we will probably keep the ones we have even if it isn’t the most efficient allocation of resources. We are ahead of the curve - in a few years, everyone else writing software will adopt the same position (or die - that takes time).
For what it’s worth, I think Fable already meets the bar.
REPLY SHARE Jeffrey Soreff 10h
Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027.
That would need to wait for both AGI in the labs plus _diffusion_ time. Scott wrote:
I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.
REPLY SHARE Matthias Görgens 8h
Why would employment in knowledge jobs collapse? If overall output increases by more than 10x, then what was previously 10% (= 100% - 90%), will become as big as the whole pie before.
REPLY SHARE darwin 7h
Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027.
Non sequitur.
Tokens are expensive, markets are neither infinitely nor instantly efficient, and there's a thousand miles of difference between a cutting-edge piece of tech demonstrating an ability under controlled settings vs. a working enterprise system that can be immediately integrated into your business to handle that workflow automatically.
If markets were this efficient, plantation owners would have been finding the smartest slave children and teaching them to become architects and engineers in order to optimize their productivity. Things take a long time to happen and there's a huge amount of inertia and cultural bias in every system.
REPLY SHARE Scott Alexander 7h Author
Have you read AI 2027? I think it does a good job making this case.
I'm not actually sure employment would collapse - the economics is weird, and even now with AI writing >90% of code the market for programmers has barely changed.
The person below is offering me better odds, so I'll probably take their bet rather than yours.
REPLY (3) SHARE TK 5h
even now with AI writing >90% of code
Where are you getting this from? This is very much not my experience.
REPLY SHARE Alberto 2h Edited
From your article:
"Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. "
The only reasonable way to interpret this passage is that AI would be taking the jobs, rather than workers using AI to become more efficient or something of the sort.
If you actually meant "job characteristics will change but white collar job numbers may not decline", that is:
1. A totally different thing to predict
2. Much harder to falsify. Yes, AI will change how we do our jobs, but the same could have been said about WhatsApp, fiber optic, the expansion of subways, etc
REPLY SHARE Ch Hi 13m
Yes, but I think super-convincingness has already been demonstrated by ChatBots. That is doesn't convince all people isn't a required bar. "A man hears what he wants to hear and disregards the rest" isn't completely true, but it's true enough to be weaponized.
REPLY SHARE JamesLeng 4h
If an AI is suddenly able to do 90% of "whatever 'knowledge work' consisted of last year," that doesn't mean 90% of human knowledge workers are automatically out on the street. That last 10% probably includes some critical bottlenecks which are suddenly far more valuable - thus inclined to hire more people - and previously unknown forms of work will emerge in response to whatever problems the AI's results reveal or create.
REPLY SHARE TGGP 17h
a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).
This is my view. Although I also rely on the EMH, which indicates that AI will be lucrative but not crazy disruptive enough to show up in other asset prices.
REPLY (4) SHARE hypnosifl 17h
I think dates well past 2045 are plausible if the ability of AI to be good at most long-term creative projects (writing novels, developing new scientific theories) requires continuous learning with constant re-weighing of the relative significance of ideas relevant to the project (lots of little a-ha moments where things shift along the way), and if continuous learning itself has to work in a way similar to how animals such as ourselves learn, with embodied learning in which sensory impressions and motor actions are are closely connected to language, starting from a baby-like state with a fairly random pattern of neural connections. This would be in contrast to the idea that we could start with LLM-like supervised learning to develop as much use of language as LLMs have, and then afterward tack on a phase of unsupervised learning to connect this with sensory experience and develop more idiosyncratic personal opinions; if that tacking-on strategy worked, then getting to AGI would probably be easier.
If this sort of process starting from a baby-like state is the only successful strategy for doing continuous learning, it might also be that constraining things to behaviors we find meaningful and useful (as opposed to obsessing over 'irrelevant' patterns or self-wireheading) depends on a huge number of innate sensorimotor biases which canalize the developmental process, similar to what animals have, and that the only way to fine-tune a good pattern of interacting biases is by some evolutionary process of variation and selection. In that case the fine-tuning could be a very slow multigenerational process, and uploads might beat attempts to create novel AGI, even if perfecting uploading without existing AGI to help us takes a long time.
REPLY SHARE Matthias Görgens 8h
Yes, so far stock prices make it look like there's lots and lots of customer surplus in AI.
Both for actual end users, and for other non-AI companies in the economy. It's also really easy for anyone with a bit of money to spin up a new near-frontier AI model: they become cheaper and cheaper, even if bleeding edge frontier AI becomes more and more expensive to develop.
REPLY SHARE Scott Alexander 7h Author
If we'd had this discussion any time in the past ten years, you could have raised the same objection, and every time, I would have been right (given today's market prices) and you would have been wrong. For example, in 2019 you could have said LLMs weren't going anywhere, because OAI was only valued at $1B. But today it's worth $1T. I think this is enough failures of EMH to incorporate AI that we shouldn't trust it too blindly going forward.
But also, I'm less sure which way the EMH points anymore. Aren't we spending close to $1T on data centers per year? It seems like *someone* expects AI to be very big.
REPLY SHARE Throw Fence 🔶 1h
In addition to the two obvious counter points you've already received (the market has a terrible track record, and the market *is* saying right now that this is the biggest thing to ever happen in the history of the world), even if the market at large did believe this, how would it express this? Other than investing in the current companies of course, which seems to be what's going to happen (cf. NASDAQ).
The EMH is enforced by arbitrage, and there exist no arbitragers at the time horizon we're talking (a full decade), so the asset market's silence on this issue gives you roughly zero bits of information.
REPLY SHARE Calvin Blick 17h Edited
I'm sure this opinion will be unpopular in this particular comment section (although it's a pretty popular opinion generally), but a lot of these very optimistic projections of AI improvement make quite a few assumptions that are really not in evidence. I'm not saying that AI has no uses (it's notetaking capabilities certainly make doctors' lives better), but the idea that AI is going to wipe out all jobs or become super-intelligent--I just don't see that as a LIKELY scenario, even if it is possible. AI has some pretty significant limits, such as hallucinations and massive energy usage, and at present there is not really a way to fix those (the energy usage and infrastructure requirements may be fine for now, but eventually these AI companies are going to need to make money). Maybe they'll figure that out, but just hand-waving away that objection is a pretty big miss.
I have noticed that AI is much better at answering questions that have been posed on reddit at one point, and not so great at answering questions that haven't been asked before, even if they aren't that hard for actual people. Basically, it's the Clever Hans scenario where the subject can answer any. question, as long as the answer is available.
REPLY (6) SHARE DangerouslyUnstable 17h
If you believe that the most noteworthy (pun not intended) use of current AI is as an automated transcription service, then I think that you are very far behind on what current AI can do.
REPLY SHARE Neversupervised 16h
The fact that you use note taking as an example makes me perceive you as being a few years behind in your AI capabilities intuition.
REPLY (1) SHARE Bugmaster 14h
I could be wrong, but I think the OP was using doctor's note taking (a.k.a. "charting") as one unambiguous scenario where LLMs could do a tremendous amount of good. From what I understand, human doctors spend a significant amount of time (perhaps as high as 25..50%) on charting, and thus an auto-charting LLM could boost the doctor's efficiency by a factor of 2x... assuming of course that it could do so without introducing hallucinations into the chart. AFAIK at present the technology is not there yet, but it's getting better every day.
REPLY (2) SHARE Neversupervised 14h
I don't disagree that there is value, but "AI as transcription" is very, I dunno, 2022? My point is that someone who questions economic activity coalescing into AGI within a short timeline should be using more recent use cases (agentic such and such) as a reference point.
REPLY (1) SHARE Calvin Blick 14h
I haven’t seen much evidence of agentic AI use transforming how companies work at this point, at least not the way that note taking software has unambiguously made doctors’ lives better. Honestly other people using AI in the workplace has been more annoying than not in my experience since it typically produces a shoddy product (ie, extremely long emails that don’t exactly address the point).
REPLY (1) SHARE Neversupervised 14h
There are companies that have replaced their entire performance marking department with AIs. There are tiny companies running high volume CPG businesses mostly out of OpenClaw. I see vibecoded vertical SaaS apps that go from 0 to hundreds of thousands in yearly revenue in 2-3 months, solo developer.
REPLY (2) SHARE Alena F 14h
I see a lot of claims like that online and yet I also see a lot of terrible customer experiences with companies that replace their departments with AI. Could you give us a working example where it actually worked? Good for you if you went to hundred of thousands in revenue in 2-3 months, you must have solved all the distribution problems with AI too?
REPLY (1) SHARE Neversupervised 11h
Oh, I am not that commercial. I'm just a nerd. These are pitches founders share, and stem from observing portfolio companies.
REPLY (1) SHARE Alena F 11h
Yeah those are pitches… hard to believe without hard evidence.
REPLY (1) SHARE Neversupervised 11h
I mean, I do follow through with financial diligence. To be honest, these companies tend to look a lot more like traditional businesses, rather than winner-takes-all startups, which is why we don't usually invest. But they have managed to build very respectable cashflows very fast.
REPLY (1) SHARE JamesLeng 3h
That sounds like it might not be a matter of extraordinary AI capabilities as such, more an overhang in starting conventional businesses due to regulatory burden. Tireless AI with encyclopedic knowledge of the relevant laws and procedures can drill through bureaucratic obstacles more efficiently than a mere mortal entrepreneur, and then hire human workers to handle less clearly defined real-world problems.
REPLY SHARE Calvin Blick 14h
I’m sure just about everything has happened occasionally, but I am very skeptical the stuff you are describing is taking place at scale. It’s like how back in 2015 or so people acted like drop shipping was the way to wealth—it was in a few cases, but generally it was not.
And a lot of the cases where companies start using AI don’t prove successful. I had AI generated ad copy. AI customer service is terrible, AI coding tools have proven less successful than anticipated, etc
REPLY (1) SHARE Neversupervised 11h
Yeah I mean there lies the question. Some of us see faster replacement risks.
REPLY SHARE Scott Alexander 6h Author
This is true, but I'm also a bit worried about AI charting. The way it currently goes is that an AI listens to everything a doctor says in an appointment and then writes it down. This works fine, but it makes it harder for the doctor to recommend gray-area things like taking illegal drugs, massaging the way they report things to insurance companies, being less than maximally-aggressive against diseases that might one day result in a lawsuit, etc. I know a lot of people who feel like the quality of their care has declined.
REPLY (1) SHARE Calvin Blick 3h
That is makes sense. I saw something similar with AI note taking for business meetings. They provide value but the fact that EVERYTHING is recorded and disseminated makes it very hard to have completely candid conversations
REPLY SHARE Ivan Fyodorovich 14h
I don't think anyone had posted the answer to Erdos Problem 90 to Reddit. Yes, that's the extreme pinnacle of AI accomplishment, but it seems that if AI is capable of functioning on that high a level, people will find ways of adapting to most white collar work.
REPLY SHARE PoulyDeer 11h
You say AI when you mean LLMs. LLMs are Clever Hans, except that they can't even answer all questions when the answer IS available.
"Give me a copycat recipe for a beesting cake, as could be used in a commercial bakery."
... swing? miss. Can't even get the first ingredient right. (Actual prompting for "the first ingredient" in a commercial bakery cake recipe indicates that the AI does, in fact, know that jot of data, it just fails to reason, at all.)
REPLY (1) SHARE Adrian 9h
What model did you use, what did it reply, what was wrong about the reply, what would a correct reply look like?
I asked Claude Sonnet 4.6 in Thinking mode with Medium reasoning effort, and the answer looks fine to my layman eyes: https://claude.ai/share/4e17e22f-c893-4009-82ac-5979c8c6af50
REPLY (1) SHARE PoulyDeer 4h
Is wrong. Needs to correct main ingredient. Fails to correct main ingredient, although that's the point of putting in "as a commercial bakery would make it."
Other queries have shown it does "have the knowledge."
REPLY (1) SHARE Adrian 4h
Fine, I'll play.
What's the correct "main ingredient", that "a commercial bakery" would use?
REPLY (1) SHARE Elvira Snodgrass 35m
Actual pastry flour, for all sorts of reasons. Also 12 servings is a tiny batch for commercial baking someone's trying to make money off of.
"I know nothing about this, so I'm pretty sure Computer is right" is modeling pretty poor epistemic humility.
REPLY SHARE Matthias Görgens 8h
Even current AI requires less energy than the equivalent human worker consumes.
REPLY SHARE Scott Alexander 6h Author
"AI has some pretty significant limits, such as hallucinations and massive energy usage, and at present there is not really a way to fix those (the energy usage and infrastructure requirements may be fine for now, but eventually these AI companies are going to need to make money). Maybe they'll figure that out, but just hand-waving away that objection is a pretty big miss."
I don't think the energy use matters much - energy isn't the bottleneck for data centers anymore, and future AIs won't use much more energy than current ones.
Otherwise, I think it's a question of whether you prefer to reason by assuming that things will stay at exactly the current level forever, vs. extrapolating trends. How are hallucinations compared to three years ago? How are capabilities?
REPLY (1) SHARE Calvin Blick 1h
This comment is a good example of my big issue with AI maximalists such as yourself. Basically every technology ever has had a similar trajectory of rapid improvement that then levels off. See phones: in 1997, most Americans didn’t own a cell phone, by 2007, most owned a flip phone, but 2015, most owned a smart phone, but it’s not like smart phone technology is drastically better in 2026–if we all had to go back to the iPhone 5 life would be pretty much the same. But somehow, we’re supposed to assume that the trajectory of AI will be totally different and it will just get better indefinitely until it achieves godlike intelligence and omnipotence. That is just a huge assumption to make.
As far as I can tell, while AI has improved a lot since 2022, I’m not sure it has improved as much as some people here think it has. AI customer service is terrible. AI generated pictures and text are the same—if anything now that people can see all the “tells” it is probably less effective than a few years ago. Hallucinations are still a big issue. And contrary to your assertion above, energy bills and data centers are big concerns for a lot of people. And eventually these AI companies are going to have to make money, and that will require either charging a lot more for AI use or enshittifying the existing models to maximize revenue—or both.
It just seems insane to me for an intelligent guy to try to argue there is a 25% chance AI will be able to automate 90% of white collar jobs in 18 months or less, when the actual odds are pretty clearly more like zero.
REPLY (1) SHARE Throw Fence 🔶 1h
I think the main reason people like Scott believe AGI is inevitable or even possible, is that humans are an existence proof of general intelligence (produced by a dumb optimization process).
I think people in your camp have to make some reasonable argument for why the plateau will come earlier for machine intelligence than biological intelligence?
Also I'll add that your point about iPhones isn't even true. At iPhone 5 level technology, the internet speeds (throughput mainly) and framerate of the display wasn't really there to support applications the likes of TikTok. That's a ten billion dollar business that couldn't exist at the time (I'm not saying we're better off for it, though..)
REPLY (1) SHARE Calvin Blick 14m
Machine and biological intelligences are completely different processes. And biological intelligence evolved over millions of years while apparently we're supposed to reach AGI within a decade.
I used an iPhone 5 as late as 2019 and it definitely supposed apps like Tiktok at the time.
REPLY SHARE Fallingknife 17h
I don't think alignment is actually solvable, but also I think it doesn't really need to be solved. We can set simpler goals by training the AI to:
1. put some positive value on human life
2. have a reward path based on very long term outcomes
If you have a long enough time horizon, there will be very little value to the AI of short term resource competitions with humans on earth. If the AI wants to maximize tokens consumed by 100K years in the future the optimization path for that is going to be almost entirely focused on spreading off of Earth and consuming the much more abundant resources there. The gain from turning a city into a GPU farm is trivial in comparison, so if you can get a positive reward value on humans it will be the less rewarded choice even if it gains a few tokens.
REPLY (3) SHARE Taymon A. Beal 16h
How do you do this without solving alignment?
REPLY SHARE MathWizard 11h
This is how you get matrix style endless fields of humans in pods being fed heroin. If AI "puts some value on human life" then you get the world tiled with comatose miniature humans (possibly infants or fetuses or even embryos, depending on what it thinks counts) so it can maximize the number of them that are "alive" per resource spent.
If what you actually mean is some positive value of humans existing and doing normal human things then defining what that means IS alignment, and figuring out how to pin that down and make AI care is solving alignment.
REPLY SHARE Nick Hounsome 5h
I will just focus on your point 2.
No! No! No! Long horizons are really, really, bad.
Simple counter: As an ASI I see that humans are getting inthe way of my plans for utopia, I do the calculations and it turnms out that, even valuing human life really highly, the future excepcted value of killing all humans now, storing their DNA, devoting myself to researching Utopia and restoring them when i've figured it out - is higher than keeping them alive now.
The future light cone is VERY big and things get strange once youy start multiplying by VERY big numbers
REPLY SHARE Earth 17h
Interestingly when I read something like this europe2031.ai I realize that the timelines are likely very short. This document estimates continual learning as arriving in 2030, whereas a more likely time frame is early 2028, as with OpenAI's claim about when an independent AI researcher appears in their labs.
REPLY SHARE Ivan Fyodorovich 17h
I don't want to argue about what AGI means, but my experience using an AI agent to review literature in biology, this Substack (https://theinfinitesimal.substack.com/p/thoughts-on-ai-in-academia) and the growing list of solved Erdos conjectures leads me to think that AI is "intelligent" enough to do >90% of knowledge work jobs already. Customization for specific jobs won't be trivial, but if agents can do the programming/research/math tasks they are already capable of, surely it is possible for them to manage payroll for a midsized company, draft legal contracts, file taxes, manage inventory, respond to most customer service inquiries, and do the vast majority of work that doesn't involve physical movement of objects.
REPLY (4) SHARE mathematics 16h
Can they do those things while making fewer mistakes than the people who currently do them?
I'm not up to date at all on the capabilities of current models, but my impression is that there is still high variance in task success rates. E.g. a given model which can solve Erdos conjectures, will still produce more incorrect proofs which it claims are correct than actual correct proofs. That would be enough to replace some percentage of math researchers with full diffusion, but maybe not 90% yet.
REPLY (2) SHARE Ivan Fyodorovich 16h
My limited experience with using agents for biology research is very positive. They occasionally miss something in literature but don't hallucinate. At the same time, this is a somewhat less versatile task than say, everything the payroll department deals with.
REPLY SHARE Neutron Herder 14h
People make errors too, lots of them, however, currently hiring more people isn't worth the cost compared to the cost of errors. If AI makes the same number of mistakes per 10,000 work units, but you can run it through 1,000 AI and take the majority vote, either because AI is much faster, or because AI is cheap enough to run it 1,000 times for the cost of employing a human, doesn't AI win?
REPLY (1) SHARE Ivan Fyodorovich 13h
Yeah, what's interesting about AI is that while earlier computer programs would inevitably make the same mistake every time when faced with a problem, the AI doesn't do that. I think AI Agents take advantage of this, as long as the AI can self-assess accurately enough and figure out where it ended up in a ditch, it can move forward.
REPLY SHARE V T E P 10h
It is my experience that AI is clearly my superior at some tasks, often better at some (and yet not something I am comfortable leaving unchecked), and, at some tasks--for example, writing--it is pretty awful in comparison with my (or any other skilled human writer's) work. I would be very careful about thinking that performance in specific domains that are easy to train off of will, and especially *has*, generalized to all domains.
REPLY (1) SHARE Jeffrey Soreff 9h
Yes. One problem for "When will we have reached AGI?" is the 'spiky' nature of AI systems (though _vastly_ less brittle and specialized than traditional software!). At the point where AIs can do 90% of knowledge work jobs, there will be endless quibbles about "But it only did 95% of the tasks" or "knowledge work should be defined more broadly than that" or "It did 90% of the tasks more accurately than a human but 10% of them less accurately".
My personal guess is that the missing pieces will get filled in, maybe gradually, maybe a whole bunch at once if e.g. continual/increment learning is solved in the next year or two.
REPLY SHARE GreetingsHello 10h Edited
It suffers heavily in terms of proper planning . It's good for performing very specific tasks but for broader tasks it doesn't properly divide things into subtasks.
As an example if your rooted phone randomly restarts then it would not ask for logs or how to filter them properly.
It would follow a loop of basically this
Step 1)Try X (X is usually a educated guess for which it doesn't have a proof but which is reasonable)
Step 2) If X works good if not then go back to step 1
If you ask it to think about logs it may go down a rabbit hole of trying to decrypt logs encrypted by the vendor and then failing.
It doesn't have good taste about what approaches are likely to work and which won't.
Unless you specifically instruct it, it won't try to gather information and do tangential research to fix a problem.
By this I mean it would google about how to fix the problem but it would not ask user to run commands through which it can know more about the problem.
In a vague sense it's bad at gathering new information (not in it's training) from the environment.
Math and programming feel like constructing something from scratch so it succeeds on them but it fails on tasks which need learning from the environment.
REPLY SHARE Scott Alexander 6h Author
I think this is like my quantum example. They're pretty smart, but they get confused very easily and make weird mistakes. I haven't had enough exposure to Fable to know whether it's the one that finally solves all these problems.
REPLY (1) SHARE Ivan Fyodorovich 5h
So I don't really understand how my beloved AI tool Edison works, but it has extremely high competence in a limited domain. It can't write a poem or scrounge a cookie recipe, but it can search and summarize scientific literature extremely well (even tasks like "find me papers that used antibodies against Protein X and performed western blots in which a knockout or knockdown control is present"). In strong contrast to ChatGPT, I have never seen it hallucinate a source ever or get confused.
What I'm getting at is that it seems plausible to me that different AI agents with specialized functions will be able to manage a wide variety of white collar tasks even without further improvements in foundational models.
REPLY SHARE Jordan19 17h
Scott, I'm curious what your personal values or preferences are on writing created by generative ai or prompting being perceived as writing?
I'm also curious if there is anything you can imagine could be done differently to minimize the impact on the temporary underclass?
REPLY (2) SHARE Jordan19 17h
Being someone who is somewhat more likely than the average ACX reader to be part of an underclass, I find myself fairly worried about concentration of power during transition to AGI. Much more worried than I have been about any modern political scare.
My trust in the high level techno rationalists is waning. Though this is evidence of nothing about reality (only some people's uniformed perception) I am seeing memes of effective altruism being completely devoid of any altruism.
REPLY (1) SHARE Taleuntum 17h
What do you mean by "seeing memes of effective altruism being devoid of any altruism"?
REPLY (1) SHARE Jordan19 16h
I mean I just started seeing memes (as in a peice of media meant to be shared on social media) equating effective altruism with billionaires who are okay with everyone else dying or being in poverty. In mainstream culture, I think this is becoming most people's first encounter with the term effect altruism. I live far outside of the rationalist bubble.
REPLY (2) SHARE Taleuntum 16h Edited
Oh, okay, I'm not familiar with those. I was concerned you saw memes shared by actual EA people that were callous or something.
REPLY (1) SHARE Mark Roulo 15h
SBF may be the most well known "effective altruist" to the general public. That probably isn't a good thing for the EA movement (though I have no concrete suggestion for doing anything about it).
REPLY (1) SHARE Melvin 11h
A concrete suggestion would be to have someone else get even richer and give away even more money, and this time don't commit fraud along the way.
REPLY SHARE Cjw 13h
There have been socialist memes equating EA with things like “it is more valuable to buy this castle as a retreat to think of ideas than to donate the money to the poor” which is based on a thing that did happen, I’ve seen those. Socialists tend to be at odds with EAs bc socialists want socialism and believe government power is nearly always the solution whereas EAs are a bit closer to neolibs who may not value all outcomes the same way. I think at least there is a tendency in Western lefties to assume redistribution of wealth (voluntary or otherwise) just feels somehow more really altruist than having a good reform idea that amounts to the same thing, like if you were serious you’d go build a well or just give your money to the poor rather than build a thing that made you money and incidentally lifted people out of poverty.
REPLY (1) SHARE nominative indecisiveness 8h
When I read "buy this castle as a retreat to think of ideas" I thought huh, this sounds like something the Fabian Society would do.
And sure enough, they didn't buy a castle, but a memorial fund from some of their most prominent members DID buy a gigantic country manor, name it Beatrice Webb House, and convert it into a conference centre.
REPLY SHARE Scott Alexander 6h Author
I don't yet have a strong opinion on this. It's annoying to read AI writing, but until making friends with some people who are terrible writers (eg 30 minutes of fretting to write a five line email) I didn't realize how disabling it was for some people, and I can't really wish to take that accommodation away from them. I think the current equilibrium (people can use it for crappy business emails, but if you try to pass yourself off as a Fancy Writer with it, then people post Pangram scores to mock you and eventually Sam Kriss comes to your house and kills you) is probably fine.
I don't know what will happen when AI becomes a better writer than the best humans, but I imagine there will still be some role for human writers for a while (in the same way that a camera is a "better painter" than the best humans at most of what painting was pre-photography, but people found ways to adjust).
REPLY SHARE skybrian 17h
Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028.
Did this get pasted into the wrong part of the article? It doesn't go with what comes before or after.
REPLY (1) SHARE Scott Alexander 6h Author
Yeah, sorry, deleted.
REPLY SHARE Jon Deutsch 17h
The anthropomorphism on display here is extraordinary. As is the lack of acknowledgement that basic economic dynamics will ultimately harness AI development.
re: anthropomorphism: "AI" isn't a they/them; it's an "it." And it's not even a single "it" - it's a multitudes of "its" of various degrees of language-disguised-as-intelligence technologies that are paper-thin in their intellect while razor-sharp in language-based reasoning. But here's the thing: humans are far more than language-based reasoning agents. Einstein did a lot of math. LLMs are pretty shitty at math. And even when they RAG math, they still fumble the ball. Is this addressable over time? Sure. But they're just not human equivalent at a fundamental level... which means they will not scale like humans would if given superhuman powers.
re: economics: As token costs surge, we're witnessing in real-time that utilizing AI is nothing more than token consumption in exchange for the simulation of thinking. And token consumption is rapidly evolving from "all you can eat" to "pay what you get." It costs about the same amount of "tokens" to feed Elon Musk and me, yet the two of us have vastly different abilities to transform the world using technology. Who's to say that "AI" will become a token-hungry team of Elon's that would organize themselves to take humanity to task vs. a token-hungry team of Jon's who just want to Netflix, chill, and post some interesting comments on Substack? The efficiency-of-impact ratio is not fixed in humans to say the least -- why do we insist on flattening this for AI technology?
Food for thought.
Cheers.
Jon
REPLY (5) SHARE Jordan19 17h
I love you very interested to hear what Scott thinks about AI as "substitute for thinking."
I am finding this to be fairly disturbing in what I am witnessing in my own life.
REPLY (1) SHARE Jordan19 17h
Typo--I would be very interested to hear what Scott thinks*
REPLY SHARE Odd anon 16h
Einstein did a lot of math. LLMs are pretty shitty at math.
An OpenAI model solved the Unit Distance conjecture. Erdos problems are falling to AI rapidly. Even if you don't think AI has surpassed peak human ability in math yet, it's clearly somewhere up there.
If you previously assumed this would not happen, perhaps you should reconsider your views on intelligence.
REPLY (1) SHARE Jon Deutsch 16h
Yet they're developing "intelligence" quite differently than humans. It's impossible to predict where it's all headed based on an unprecedented history to-date.
And still AI is merely highly complex (and token hungry) algorithms that tickle the human mind due to breaking the language barrier.
AI is not a them/them. And it's difficult to see a future when it becomes something deserving of those pronouns.
REPLY (1) SHARE Throw Fence 🔶 1h
And it's difficult to see a future when it becomes something deserving of those pronouns.
Exactly. There is no evidence that could make you change your mind, which is a problem. Maybe you should read this? https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence
REPLY SHARE m. scott veach 14h Edited
I don't want to be overly pedantic, but the word 'they' is absolutely appropriate to use when referring to a language model. You should maybe look it up? The OED makes it clear: the word is used to refer to "people, animals and things." For example, if my wife asked me about the dishes, I might say, "Yeah, they're cleanish." AI is both a they and a them.
"LLMs are pretty shitty at math. "
No. Language models won the gold at the Math Olympiad, is solving Erdos problems, co-authoring papers with Donald Knuth, helping Terrance Tao daily with his work. They're not just good at math, they're spectacular.
"As token costs surge..."
Token costs are not surging; they've been dropping dramatically. And not by a little, we're talking per-token price has dropped 98% since 2024.
There are some other funny ideas in there but I'm not sure they're worth addressing.
REPLY SHARE V T E P 10h
I'm somewhat confused by the choice of math as example domain here-- AI struggles with a lot of things, but I think math is undeniably a strong point of modern frontier models.
On the economics point:
Tokens are cheaper and tasks are cheaper than they were previously, at least in the task categories most relevant, as far as I can tell. We just use them for a lot more tasks, and so the total cost has increased noticeably.
Also, I seriously question the idea of trying to describe AI models in the way you are; while you do see some serious variation in personality and type of preferred task model to model, I think the idea of preferring being useful/doing a task is sort of a fundamental part of the modern RLHF/RLVR/RLAIF/whatever comes next + character training process... I think people are right to care about how exactly this shapes it, but I don't think potato is a likely shape given modern processes.
REPLY SHARE Scott Alexander 6h Author
"LLMs are pretty shitty at math."
You may want to open a newspaper from the past month, but consider sitting down first.
REPLY (1) SHARE Durban Romancer 6h
This reminds me of my favourite scene from Dumb and Dumber where Lloyd, our protagonist, just got "stood up" for a date at "11" (he rocked up to the bar at 11am...). Upon someone else realizing the mistake for him, he gallivants toward the bar exit where a newspaper clipping of the Moon Landing catches his eye. Set at least 20 years ex post, he incredulously mutters "no way" as the implication dawns upon him, barrels through the door and ecstatically exclaims "We Landed on the Moon!".
I had this moment the other day when using AI for coding for the first time since 2022 (the early bird doesn't always get the worm). Boy oh boy, have we landed on the moon.
REPLY SHARE Jay Slater 17h
AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time.
I'm not so sure I buy that the gap from expert to better-than-top-geniuses is actually that small, or smaller than "dumber than a child" to "expert level" in some meaningful way. Many people go from as dumb as a child to expert level in some field or another; very few go from expert level to top-genius level.
I don't buy chess and go as counterexamples to the above either, but I'm having trouble articulating why and need to commute, so that'll have to wait.
REPLY (6) SHARE Vitor 16h
I don't buy chess and go as counterexamples to the above either, but I'm having trouble articulating why and need to commute, so that'll have to wait.
Because those are combinatorial problems. There is an objective, easy to measure goal (reach a winning state) and as long as the AI does the accounting correctly (track if it has reached that state), even the dumbest algorithm will *eventually* find the optimal strategy.
REPLY SHARE Simon Kinahan 15h
I would assume diminishing returns, so there is some sigmoid curve relating effort/time/energy to accomplishment and somewhere it becomes economically infeasible to hit the next benchmark. But the curve is not necessarily in the same place for humans and for LLMs (or other AIs). Maybe "top genius" mathematician isn't very hard for an LLM to achieve once its gotten to expert level, whereas its basically impossible for most expert human mathematicians. But actually I think its likely the reverse is true - LLMs are much less efficient at learning, both in token inputs and in energy inputs than humans, even though the network size of cutting edge models is 10-100x smaller than a human brain. They will not get more efficient as the networks get bigger, if anything probably the reverse, so its likely "top genius" LLMs even in easy fields are not going to be economically feasible without some algorithmic or hardware breakthrough.
REPLY SHARE Cal van Sant 14h
I don't think that line of reasoning works. Many people go from 0 feet tall to 6 feet tall over their lives, but few go from 6 feet to 7 feet tall.
REPLY SHARE Coriolis 13h
I am a phd in physics who uses AI all the time, and the it's true - AI can answer relatively advanced physics questions.
On the other hand i also enjoy video games, and AI can't reliably answer relatively basic video game strategy questions.
How can that be? Almost everything written about physics past a certain level is written by people who are professional physicists. They can still be wrong of course, but there's nearly no one mouthing off about random niche science topics on the Internet. The same is not true for video games.
Neural nets still have no way of resolving what's true, which isn't surprising, because humans don't either. We have to empirically check things. And making physical machines that can do empirical verification under AI control is not even close to a solved problem.
REPLY (1) SHARE PoulyDeer 11h
What's hilarious is when the AI does actually "know" the relevant piece of information, but fails to apply it even when specifically prompted to do so! Because the "chaffy" GIGO internet never, ever applies the relevant piece of information, either!
REPLY SHARE Scott Alexander 6h Author
"Many people go from as dumb as a child to expert level in some field or another; very few go from expert level to top-genius level."
I think this is confusing ability of humans to traverse a space with the size of the space itself.
For example, most humans reach 5 feet tall, but very few humans reach 7.5 feet tall, and none reach 10 feet tall. Yet if there's some non-human process, like throwing a discus, doing it 7 feet is only 50% harder than doing it 5 feet, and doing it 10 feet is only twice as hard. Most people who reach 5 feet tall will never reach 10 feet tall, but most people who learn to throw a discus 5 feet can later learn to throw it 10 feet.
I think if you think of human tasks as (let's say) 100 IQ vs. 150 IQ vs. 200 IQ in the same way that height could be 5 feet vs. 7.5 feet vs. 10 feet, the way AI will traverse this space looks more like some kind of objective metric than like mimicking the human difficulty curve (I know IQ doesn't work that way, it's just an example).
REPLY (1) SHARE eleventhkey 5h
I can’t help but feel this is going to be contingent on the training data.
There’s a vast corpus teaching AI to move from ‘dumb as a child’ to ‘expert’. There’s a much smaller training set available to take the AI from ‘expert’ to ‘top genius’, and by definition there is no training data available to take AI beyond the level of the best human geniuses.
REPLY (1) SHARE JamesLeng 2h
by definition
That somewhat depends on the nature of the problem, and the wider field. Recombining best practices from diverse specialists could conceivably yield results no single human lives long enough to imitate, or an adversarial process could build on the previous iteration's work.
Given a machine which can solve every Erdos problem, how hard would it really be to set up a second machine which proposes new conjectures in an effort to stump that first one?
REPLY SHARE Durban Romancer 5h
Some food for thought:
Parallels between human and AI - AI is fundamentally a different intelligence to a human. There's no reason to suggest that human aptitude paths follow AI aptitude paths outside of it being our only prior for intelligence development.
The following is a can of worms, but I don't think a metaphor for AI "infancy" works in the same way it does for people. New AIs are more analogous to new phones than a person with an extra year in terms of development. Its a stepped instead of continuous process. These AIs are distinct from one another - as with their "personalities" (quirks?). A person has more contunuity.
As Jon Deutsch suggested above, we should avoid anthropomorphizing too much. This is a far cry from "anything goes", though.
Muravec's paradox perhaps buttresses this developmental point. If we permit the metaphor I've just eschewed above, AIs over time possibly have most reason to develop backwards compared to human timelines. The paradox, pre-LLM, observed AIs were much better at being trained to success on "higher order" human activities (chess) compared to lower order ones (house cleaning).
If the paradox is true, we should perhaps expect the highest order thinking from our human perspective to be well within reach rather quickly? But perhaps we should concomitantly expect dispersion to be slower (for knowledge jobs that require certain "base" aptitudes that are difficult to train for - difficult to name any, but we already observe AI aptitude "spikes" as Scott calls them in the post).
Thoughts?
REPLY SHARE Notmy Realname 17h Edited
Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs
If it was smart enough to do 90% of knowledge works jobs, it would be doing 90% of knowledge work jobs, it's tautological. If it's measuring as "smart enough" on some benchmark but that isn't translating into actual integration in the real world, that doesn't mean the world is slow, it means that the measurement is faulty. There is a huge gulf between being able to do discrete tasks while being babysat by a former knowledge worker turned prompt engineer performing his job in 10 second increments, and actually being smart enough to operate autonomously and productively in the economy.
I reject the premise that benchmarks/AI model horseracing performance gains indicate that significant progress is being made towards this independence. According to ChatGPT 5.5, ChatGPT delivers ~10x to 100x more useful cognitive work per interaction, but having interacted with it both I am skeptical that ChatGPT 5.5 is much if at all closer to independently acting as a knowledge worker.
AI is absolutely being used in business. Most companies are using Copilot as a Microsoft Office suite application much like they use Word and Excel. Software developers use Claude or Codex. This is translating into minor to substantial productivity gains for knowledge workers. None of this is indicative that suddenly autonomous AIs will be replacing knowledge workers
REPLY (3) SHARE DangerouslyUnstable 17h
So you believe that every time a machine isn't being used to do a job, that's because it _can't_ do that job? It's never because someone doesn't know the machine exists, or because the machine is too expensive, or because the machine is illegal?
REPLY (1) SHARE Notmy Realname 17h
I incorporate the non strictly technological developmental barriers to widespread usage into "can't" and "smart enough"
REPLY (2) SHARE DangerouslyUnstable 16h
If I understand your point, then why would you do that? He specifically separated those things out by having the section on diffusion. He makes in the article a clear separation between ability to do something and actually doing it. Combining them again, and then using the combined version to attack a claim that is explicitly not that seems not in good faith.
REPLY (1) SHARE Notmy Realname 16h Edited
I don't think they are separable, the 'benchmarked ability with no actual usability' piece is meaningless. It's like a flying pig that just happens not to fly
REPLY (2) SHARE Taleuntum 16h
I love that I don't even understand what you mean by your meant-to-be obviously-correct-analogy-to demonstrate-the-point example. To me, it seems useful to separate the ability to fly and being actually in flight for flying pigs too.
REPLY (1) SHARE Notmy Realname 12h
Fine. I've been reading the Piraha book lately that Scott linked, about a barely contacted tribe in the Amazon with a very unusual language and no number system.
Consider the smartest Piraha ever. Consider if John von Neumann or Albert Einstein with a 200 iq was cloned and born as a Piraha. They would be great at Piraha activities, and they would absolutely be masters of their jungle craft. If you needed a jaguar hunted, they'd be your guy.
Now, teleport them to Grand Central Station and have them report to do knowledge work at an accounting firm. They'd be utterly helpless. They are naturally smart enough that if they are handheld through specific, brief tasks they can do them well, but they are would be completely incapable of interfacing with the modern business environment. They have an iq of 200 and they can excel on their benchmarks and on discrete tasks, but they don't have nearly enough "smartness" to operate usefully independently indefinitely.
If instead of an iq of 200 they had an iq of 200000, maybe they could completely reinvent modern civilization from scratch in an instant and jump right into modern office work, but that is a tremendous leap beyond just the raw intelligence they need to do the stuff they are good at well. No matter how good AI gets at their benchmarks, I don't think noticeable progress is being made towards the true autonomy required to act as a modern knowledge worker.
REPLY SHARE DangerouslyUnstable 15h
You are making up a strawman with your benchmark point. People aren't talking about fake claims to be able to do something. You can disagree that real ability will come, but that's addressing a totally different part of the article than the one you initially replied to.
REPLY SHARE MathWizard 7h
Are you, right this instant, solving elementary school level arithmetic problems?
Does that mean you can't or aren't smart enough to solve them?
REPLY SHARE Taleuntum 17h
When a car that was able to transport people more cheaply than a horse could was created, did it take over every human-transport job the next day?
REPLY (1) SHARE Diego 16h
If you define "more cheaply" as including the social, chronological, legal, and infrastructural drag/costs, which is roughly analagous to this person's definition of "smart enough" then...yeah, I guess so?
REPLY (1) SHARE Taleuntum 16h Edited
So if, in the future, AI becomes godlike, but 70 year old George, the accountant who is set in his ways does not bother to use it, then AI is not smart enough according to this person's definition? I'm still wondering why anyone would find it useful to adopt this definition and criticise those who don't do so.
REPLY (3) SHARE Diego 15h
It seems not entirely unreasonable to me in this context, though not the typical understanding of “smart.” I suppose OP would say that if the AI can’t make itself useful and attractive enough to convince Greg, or if not Greg then some sufficient number of people such that +90% of knowledge jobs are being done by AI, then that is in fact a failure of instrumental intelligence.
REPLY SHARE Notmy Realname 11h
Yes. If the AI was actually smart enough, it would be used
REPLY SHARE Durban Romancer 2h Edited
I think there's a neat middle ground between your perspective and the original commenter's.
"Actual integration" is a broad church:
1. Regulatory boundaries; attitudes etc. (what you are taking it to be)
2. Contextual on-the-job understanding (the OC's view)
I think you slightly strawman what they were exactly saying - though I do agree that the causality is multivariate and not just the lack of "context".
That being said, I don't think practically that their argument is super strong.
TLDR: firm-specific training and then AI-re-designed and tailored systems to avoid any "understanding" as humans see it mean this practical constraint is transitory at best.
This impact will be lowest in tasks that (1) have limited technology usage, and for knowledge work (2) discrete tasks that are difficult to aggregate vast amounts of data from.
Pre-ASI, if we model adoption (diffusion) as a function of Token Cost and quantity used per task, labour demand is a function of how many tasks give labour a cost-based comparative advantage to AI.
Wages therefore are less than or equal to the cost for the AI to do the task, above which an employer would find it more effective to use AI (kind of like a Malthusian upper bound on wages).
This would apply in the "medium run" as it assumes perfect substitution between AI and labour. In the short run, before this labour demand will also be a function of a task's requisite contextual understanding (interfacing multiple systems, for one). This is related to the OC's point.
2 forces assuage this, pushing us to the "medium run":
1. Use of AI side-by-side or directly replacing labour in current systems will provide data for this "contextual" understanding. Possible gains coming from this could be universal, or firm-idiosyncratic (influencing speed).
2. Redesigned systems not requiring a human "contextual understanding". e.g. working entirely in an SQL database instead of using finicky interfacing tools that are specifically human-friendly...
I think these 2 "forces" even with AI at this point in time are achievable today.
As AI's cost advantage over labour increases (if it does? https://www.apollo.com/wealth/the-daily-spark/cheaper-tokens-bigger-bills), then it's a matter of time until "Contextual" understanding is gained.
Regarding regulatory issues/ public backlash and resistance of norms:
Where there's a will there's a way.
Microsoft and Amazon are planning on using mini nuclear power plants to power Data Centers after decades of general public resistance to their usage. Not all decisions are political decisions; not all of these decisions necessarily have political backlash
Competitive forces will force later adopters of AI to follow suit if productivity gains arise sufficiently, otherwise they risk becoming a shareholder pariah, and simply being out-competed.
On the regulatory side:
1. Most countries (outside of large economic blocs like the US, China and EU) have limited control. If they don't permit AI in work - say goodbye to investment and firms. For politicians - say goodbye to growth and good PMI numbers, ergo your electorate and corporate supporters.
2. There may be resistance from losers (e.g. consumer discretionary firms whose demand would dry up from AGI reducing disposable incomes), but if they are dwarfed economically by winners, the regulators will likely listen to the latter.
So I'm not really sure that these boundaries will be that significant. But I may just be too bearish on AI's impact. Regulatory and social resistance will be massively amplified if the effects are as well. I still think economic forces will play a large part. They will be more salient if the impact of AI is slower (as opposed to an ASI huge spike in capabilities which suddenly changes everything).
REPLY SHARE Scott Alexander 6h Author
"If it was smart enough to do 90% of knowledge works jobs, it would be doing 90% of knowledge work jobs, it's tautological."
This is not true at all. For example, I am smart enough to be a high school English teacher (based on my SAT score), but I am not a high school English teacher.
Or, more prosaically, an AI could be smart enough to do a task, but banned by regulation.
REPLY SHARE Jordan19 17h
Typo---I would be very interested to hear**
REPLY SHARE Justin L 17h
"I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us."
How far back in time do you need to go for this to be true for that year's inhabitants relative to today? Is 1900 too similar, or would the improvements sufficiently impress? 1800? 1400?
REPLY (3) SHARE Herb Abrams 17h
I think if I was from 1900, just knowing about our improvements in childhood mortality would make today seem like utopia.
REPLY (1) SHARE demost_ 7h
Agreed. But this makes me skeptical of Scott's prediction. I think 2026 qualifies as utopia from a 1900 point of view (at least for people living in developed countries), but most people living today would not consider it as utopia. I am much more doubtful than Scott that people living in any kind of utopia would acknowledge it as such.
REPLY SHARE Stephen Saperstein Frug 12h
I want to read a whole Scott Alexander essay on this question.
REPLY SHARE Citizen Penrose 2h Edited
There's a book called Looking Backwards from 1888 imagining what a future socialist utopia in the year 2000 might look like. Some utopian stuff like freely available recorded music has come to pass, and modern technology is generally a lot more advanced than in the book, but overall the society in the book feels more utopian than the real modern world. So my guess would be the real modern world wouldn't qualify as utopian, at least to the author of that book.
REPLY SHARE Maxim Nazarenko 17h
No matter what are your actual thoughts about AI, here is a great framework for laying them out: https://substack.com/@larsiusprime/note/c-272111008?r=7zdwlv
REPLY SHARE bbqturtle 17h
Isn’t there a bigger difference between being able to do 75% of jobs and being able to self improve?
Like, I’m sure it can do Karen from accountings job with a bit of scaffolding and rethinking who gets blamed when things go wrong. But I’m not sure it can do AI-deep learning levels of work. Isn’t that the remaining 25% in all cases?
REPLY (2) SHARE Paul Goodman 8h
The ability to improve AI models is a) something the people working at AI labs value very highly and are likely to prioritize in the models they build, and b) something they know a lot about and are likely to be good at judging how well the models are succeeding at. That suggests that, to the extent that AI capabilities are "spiky" (significantly better in some specific areas and worse in others than you might intuitively expect) the spikes might favor being good at self improvement.
REPLY (2) SHARE bbqturtle 7h
I mean maybe. But when you think about knowledge work, I feel like you or me could easily do 75% of that work within a month or two of practice. But something like deep neural learning at the cutting edge? Very unlikely.
REPLY SHARE JamesLeng 2h
That depends very heavily on exactly what's causing the spikes. Some problems just aren't efficiently computable, no matter how motivated you are or how obvious a correct answer would be.
REPLY SHARE Scott Alexander 6h Author
I agree with Paul below - self-improvement will come early because the labs are working on it hard. But also for two other reasons:
- To take Karen from accounting's job, it has to be able to do 100% of her tasks without making mistakes. To help self-improvement, it just has to be able to do 50% of an AI researcher's tasks in a way that speeds them up significantly.
- AI is much better at things that can be massively trained through reward signals. ML research can sort of be like this - you can (in theory) make it design an AI a million times and reward it when the AI does well on some benchmark. But many superficially easier jobs aren't - if Karen has to do customer service, you can't make it interact with a customer a million times during training and grade the results because customers are real people who move at human speed. Probably there are hacks for this, but ML training needs fewer hacks than some other things.
REPLY (1) SHARE JamesLeng 2h
To help self-improvement, it just has to be able to do 50% of an AI researcher's tasks in a way that speeds them up significantly.
That doesn't follow. For a recursive self-improvement takeoff, the rate at which AI assistance is speeding up research needs to exceed the rate at which successive incremental improvements become inherently more difficult. Any single automation-resistant bottleneck task could stall the whole thing, or a cascade of individually trivial inconveniences could add up to throttle the growth rate.
REPLY SHARE Harjas Sandhu 17h
But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly.
Wait, really? We already know that LLMs are being used for military operations, right? This is a strange amount of confidence in the general risk-aversion of powerful people...
REPLY (2) SHARE Taymon A. Beal 16h
Isn't current military use of LLMs limited in scope in exactly the way Scott describes? Or am I behind the times on how they're being used?
REPLY (2) SHARE Mark Roulo 14h
"Isn't current military use of LLMs limited in scope in exactly the way Scott describes? Or am I behind the times on how they're being used?"
From https://x.com/newscientist/status/2064773160642216159:
"A senior figure in the Ukrainian defence industry told New Scientist that a test took place two years ago involving fully autonomous drones set to destroy anything in a given area, with confirmed casualties"
The "autonomous" bit is that the drone didn't have a human in the kill chain once it was deployed.
I doubt that this drone had an on-board LLM, but I don't know much that should matter. And if these things work I expect them to get scaled up as people try to win this war (or future wars).
REPLY (1) SHARE Frikgeek 3h
That's closer to a smart mine than a real autonomous weapon system. It doesn't really have an "AI" that makes any sort of decisions, just an advanced guidance system.
It's honestly closer to precision guided munitions we've used for decades than an autonomous killbot.
The human in the kill chain is the one that dropped it in a certain area, just like with mines.
REPLY SHARE Victualis 4h
https://archive.is/uSqty is a recent article in the UK Financial Times about the UK military considering fully autonomous weapons, so it's not just Ukraine saying this in public.
REPLY SHARE Scott Alexander 6h Author
That's a good point. I was thinking of civilian uses where we err on the side of safety, but the military might accelerate things to stay competitive, or because dying in an accident is more acceptable there. But even in the military, I think current uses of AI are somewhat limited - I think (not sure, tell me if I'm wrong) nobody is putting ChatGPT in charge of a drone without human oversight.
REPLY (1) SHARE PoulyDeer 3h
The American military might accelerate usage of AI, because American Leaders pay a large price for even a single soldier's death. I would absolutely not trust the military to tell us if they're using AI within the context of drone warfare.
REPLY SHARE Gerald Monroe 17h Edited
Comment : the cyber security abilities of Mythos show how "forced adoption" works.
"Pay for the tokens to regenerate and rewrite all of your software that touches untrusted input...or else".
Software maintainers get between now and when kimi/deepseek releases an open weight model that matches mythos in hacking.
It's not dissimilar to various gunsmiths inventing the machine gun. Once they were common, every military in Europe had to buy some, hugely accelerating adoption.
This will generalize. Are you a doctor? Your competition charges half price for a visit, using AI to do all the paperwork. A judge? Your court better adopt AI or the flood of lawsuits will make the delays 10 years.
Computers didn't provide such a productivity advantage.
REPLY (1) SHARE Cjw 13h
People would not accept AI judges, not without at least the right to a de novo appeal to a human judge, which (given the automation of drafting the appeals and the definition of de novo) would effectively mean every case still goes to a human judge. Perhaps at most some bond would be required to stay execution of the order pending human review, but then surely activists will demand in forma pauperis filing rights, so effectively the whole thing is a waste of time. It’ll replace a few lower level court clerks who currently just route filings and don’t have the authority to stamp any orders, likely no more until AI causes democracy to collapse anyhow and the courts will have no authority
REPLY (2) SHARE Gerald Monroe 11h
AI judges making the decisions : no doubt. I meant in a general sense the COURT must adopt AI in order to analyze the enormous flood of lawsuits, motions etc that all the parties in every case will use to flood the zone. The judges AI has to be filtering every filing for incoherent reasoning, made up citations, existing precedent clearly covering the exact situation with the ruling going against the filer, etc. "Vibe-judging" becomes a lot of the judge's job but yes they have the ultimate authority.
REPLY SHARE The Dao of Bayes 9h
98% of federal cases end in a plea bargain, so I think you're over-estimating how much the average person values a fair trial. If your choice is an AI-generated plea bargain or a human judge imposing the full sentence...
REPLY SHARE Abe 17h
"Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028."
I guess the paragraph directly above this one says that this isn't true, but it seems like there was supposed to be more text here?
REPLY SHARE Kindly 17h
"I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation"
I'm sorry, I can't parse this. Is this implying a 66%+ chance of the universe being a simulation? Or is it only 66% for the conditional probability P(simulation|singularity)? Or something else?
REPLY (3) SHARE Taleuntum 17h
I interpreted it unconditionally. Probably the most based prediction in the bunch.
REPLY SHARE Diego 16h
Reads as unconditional to me, because the explanation explicitly references the possibility of a singularity not occurring
REPLY SHARE Gabriel 13h
It was a surprisingly offbeat final opinion to include in the post! I strongly approve of the inclusion. Gotta have some quirky fun amidst the structured arguments. 🙂
The 66% was shockingly high to me, but that's only because I have different metaphysical opinions, presumably.
REPLY (1) SHARE Tossrock 13h
Finding that you are Scott Alexander is a much stronger argument for being in a simulation than finding you are Gabriel, presumably.
REPLY (2) SHARE Katie 11h
But only if you believe there's a good chance that a good proportion of other people are p-zombies right? Really want more information about how Scott is thinking about this, I thought it was truly insane
REPLY (1) SHARE Tossrock 10h
He's said something adjacent to this before, basically "many smart people agree this is likely, but it's also very important to act as though it's not"
REPLY (1) SHARE Katie 9h
ah well I guess I should be happy that as a non-smart random I have access to truths about the world that are hidden from the smart important people.
REPLY SHARE Gabriel 11h Edited
Rather, I lean toward Tegmark's mathematical universe hypothesis. The MUH is roughly: our universe is a mathematical structure, every other mathematical structure is equally as real as our world, and some of them like ours contain people.
On that view, a simulation of a world is peeking at part of the structure of that the world, not creating it. Pausing or stopping a simulation of a world (a mathematical structure) doesn't pause or stop the world.
It's hard to make any sense of the simulation argument within MUH. There's not a unique fact of the matter about whether a world is or isn't a simulation. So anthropic Bayesian updates don't figure in.
Furthermore I take Parfit's ideas on personal identity and run with them. His view is roughly: there's no lifelong continuity, only interlocking chains of mind-moments that remember past mind-moments. To that, I add that there are countless future versions of you across the worlds, countless mind-moments that remember having been the mind-moment that you now are.
On that view, stopping a simulation of a person only stops that person in the local universe; from their first person perspective, somewhere out among the possible worlds is a mind-moment that is the continuation of where the simulation left off. (Yes, it's the Quantum Immortality idea on steroids.)
This separately also makes it hard to make sense of the simulation argument. There's not a unique fact of the matter about which world you'll experience in your next moment. So anthropic Bayesian updates don't have a role here either.
Now of course those are metaphysical opinions, for which the proper epistemic status IMO is "fun to think about but should not impact decisions".
REPLY SHARE Nicolas D Villarreal 17h
I trust this sort of analysis considerably less when I see no particularly low or high numbers appear anywhere. Nothing AI related is less than 10%, even the odds of nuclear war destroying humanity are at 5% for a 30 year period, which is incredibly unrealistic considering it requires not just any nuclear exchange, but a nuclear exchange between countries that have produced in the order of thousands of warheads. This you mean there was only about a 60% chance it hasn't happened already, was it really basically a coinflip that we haven't already died to total nuclear war? With regards to AI, I don't think you understand the issues with compute constraint, necessity of continuous learning and how serious the lack of training data for open ended problems is.
REPLY (3) SHARE Diego 16h
"Basically a coinflip that we haven't already died to nuclear war" is a pretty fair description of the Cold War, honestly
REPLY (1) SHARE Wisdom777 15h
Yeah. OP, please look up the history of nuclear close calls.
REPLY (2) SHARE Nicolas D Villarreal 13h
If it really was a coinflip, it's extremely likely one of those close calls would have actually resulted in a nuclear exchange. The high stakes of nuclear war and its lack of being in anyone's interest mean people will generally avoid triggering a nuclear attack.
REPLY (3) SHARE Diego 12h
Those incentives and stakes have not performed nearly as well as one would hope. In 1969 the President of the United States ordered a nuclear strike on North Korea, just to throw one such event out there
REPLY (1) SHARE PoulyDeer 11h
Cite?
REPLY SHARE Kindly 10h
No, if it really was a coinflip, it's ~50% likely that one of those close calls would have actually resulted in a nuclear exchange.
REPLY SHARE Beren 10h
It seems like you're conflating "a coinflip that we haven't died to nuclear war" with each "close call" being a coinflip.
REPLY SHARE Melvin 11h
I think the bigger factor is whether a full-scale nuclear war at the height of nuclear weapon stockpiles would have been sufficient to kill everybody.
I think the current consensus is that this is very unlikely, and that the Cold War era scenarios where this did happen were always just made up to scare normies rather than being based on realistic modelling.
REPLY (2) SHARE PoulyDeer 11h
Do you know where the most Russian nuclear weapons point (of the ones that point into America)?
Nuclear winter, as a possibility, was real. Continues to be real. It's America's killswitch -- nobody else's.
REPLY SHARE Diego 10h
“Kill everybody” was never realistic, but killing say 40% of the global population over the course of two years is entirely in the range of possible outcomes of nuclear war, and that probably causes near-irreparable damage to human civilization
REPLY SHARE Tossrock 13h
I suspect there's a decent chance of Google's Titans + MIRAS architecture pattern, or something similar, creating a new generation of continuous learning models within the next two years.
REPLY SHARE Scott Alexander 6h Author
That's because it would be boring. I think there's a 99% chance that someone creates an AI intended to help tutor people, and a 1% chance that AI paints itself green and dances a jig on top of the White House, but who cares?
REPLY SHARE hnau 16h
The first timeline point does a decent job acknowledging this, but it's worth restating plainly: "recursive self-improvement" isn't an argument, it's an IOU for one. Sure, RSI seems from first principles like it might happen, but so do many things in human economic / social history that haven't. Without some gears-level model of what this might look like, I'm disinclined to accept it as a factor pushing for shorter timelines. (And no, the linked AI Futures page appealing to "research taste" isn't nearly enough, though I might be missing some more detailed analysis by them elsewhere.)
REPLY (3) SHARE Presto 12h
What do you mean? I think RSI has already started.
REPLY SHARE Kindly 10h
Does AlphaZero count as recursive self-improvement (in a limited domain)?
REPLY (1) SHARE Frikgeek 2h
Surely not because AlphaZero never improved its own development or the development of other AI models. I think you're confusing RSI with just regular machine learning.
REPLY SHARE Scott Alexander 6h Author
I'm not sure how you distinguish between an argument and an IOU for an argument. It's possible, it has some probability, I don't see any reason to make that probability extremely high or extremely low, so I'm putting it medium. See https://slatestarcodex.com/2020/04/14/a-failure-but-not-of-prediction/ . And as Presto said below, some amount of RSI has already started, see https://www.anthropic.com/institute/recursive-self-improvement
REPLY SHARE Jacob Goldsmith 16h
When you say 90% of knowledge jobs, do you mean 90% of currently existing knowledge jobs or 90% of knowledge jobs that exist when AGI comes?
REPLY (1) SHARE Scott Alexander 6h Author
Good question, existing ones.
REPLY SHARE Carrots and Sticks 16h
p(doom) calculations don’t give points for delay
Shouldn't they though? After all, in the (very) long run p(doom) ~= 100%
REPLY (1) SHARE Dhay 12h
I think it could be set a limit for the sake of argument, e.g., p(doom) before 2100. After that things are too unpredictable.
REPLY SHARE Vitor 16h
Thanks for writing this and putting everything in one place.
I think there's nuance to the idea of a diffusion gap: does "AI can do 90% of knowledge work" mean *in principle* or *in practice*? What I mean is that there's a further capability gap between:
"this AI has demonstrated enough general reasoning skills that I'm confident it could replace a working mathematician, if we fine tune it, develop a bunch of tools, debug it, do a soft roll-out to hand-picked research areas, etc"
vs
"this AI could replace a working mathematician *right now*, all you have to do is give it the passwords to their email / arxiv / etc"
How large would you assess this gap as?
REPLY (1) SHARE Scott Alexander 6h Author
I think I mean the first, with the latter as part of the thing you have to do to close the gap, but I agree it's awkward and ambiguous.
REPLY SHARE Victualis 16h
Do you have a citation for that claim about a 20 year diffusion for PCs? The IBM PC was 1981, by 2001 perhaps the majority of desk jobs required a PC, but diffusion into all jobs was very uneven. With tablets and phones computers were pretty ubiquitous by 2021. My understanding is that plumbers and warehouse workers and shelf stackers would not have had or been expected to use computers in their jobs in 2001, and things like PDAs with custom software would have been seen in innovative companies like Amazon only. Total sales of computing devices only really hit the numbers to make real diffusion possible post-2007 with smartphones/tablets.
REPLY (2) SHARE Mark Roulo 14h
I can provide something that is probably close to what Scott is thinking of:
https://www.linkedin.com/pulse/how-does-lifecycle-software-product-work-joaquim-torres/
Search in the document for "S-curve in real life"
I have no idea if the chart is correct.
REPLY SHARE Scott Alexander 6h Author
Good point, I meant desk jobs.
REPLY SHARE Neversupervised 16h
The simulation concept feels like a probabilistic trap. An intriguing mathematical concept that gets a lot more traction than is warranted because it's hard to disprove. I feel similarly about string theory. The average string theorist is >> smarter than me, and I still can't help but feel everyone is falling for the same trap. 66% seems insanely high confidence. What are other plausible explanations for the anthropic principle? One is that AI will kill everyone, and population grows exponentially until roughly that point, so we are still more likely to be alive today than in 4000 BC.
REPLY SHARE Neversupervised 16h
Scott, what is your take on Dyson spheres or swarms. I forgot the median timeline from the first Curve event, but it felt preposterously short. I think Daniel said 5 years at some point.
REPLY (1) SHARE Scott Alexander 6h Author
I would lump that in with the "Bostromian superintelligence" point. If an AI can do 100 year of tech progress in one year, seems like maybe the next year it can do 1000 in one year, and surely Dyson spheres are only a few millennia away.
REPLY SHARE Kevin McLeod 16h
By 2034 analog will be dominant and forms of waves external from our body will be semiconscious.
REPLY SHARE Raj 16h Edited
Part of me has been very pessimistic about all this lately. Like what are the chances that it plateaus very soon (cost and training data vs real economic utility), which gives us some combination of mass unemployment and inequality but no singularity, and it traps us in current year neoliberal consensus reality?
Or the popular view that it is a bubble is true, we here have just been steeped in sci-fi for too long; markets correct shortly and this is actually our one max effort shot to get there as a civilization and may fail, and not get another one?
REPLY (1) SHARE Bob Bobberson 15h
I don't think that's such a bad scenario. If AI plateaus then it remains as basically a normal technology, which will disrupt our economic and political systems but not entirely break them, meaning that we will have time to figure out solutions. For example instituting a UBI, or increasing employment in the niches AI isn't great at. The sooner it plateaus the more of those niches there are.
The scenario that scares me more is rapid self-improvement leading to humanity's irrelevance and potential extinction. I don't have much faith in our ability to manage alignment in a fast takeoff situation.
REPLY SHARE Richard Weinberg 16h
I used to be extremely skeptical about your views on the AI apocalypse, but that was before I started interacting seriously with Claude. I remain a little dubious, still feel you've been drinking too much Kool-Aid, but it no longer seems ridiculous. I mean omg you might be basically correct.
REPLY (1) SHARE Scott Alexander 6h Author
Kind of surprised to hear that, I don't think Claude is good enough to justify believing any of this on its own, but I'll take it.
REPLY (1) SHARE Richard Weinberg 4h
I lack your AI chops, but one month of talking to Claude has been a real wake-up call for me. BTW, it turns out that Claude is very impressed by you, even if you're not too impressed by him. Given your worldview & priors, your probability calculations sound reasonable, though I still think you're over-alarmist and way too precise in the face of known (and unknown) unknowns. Keep writing; I really enjoy reading your essays.
REPLY SHARE Alastair Horn 16h
I really struggle to understand what people mean when they say "post-scarcity". Clearly something is always scarce, because the machine of economic growth keeps churning until it there is something preventing it? Prices of everything not scarce drop to zero and prices of everything scarce rise indefinitely.
Either that or you pick a definition of post-scarcity which clearly includes today, which is also reasonable, but clearly not the way you are using it.
REPLY (1) SHARE Wisdom777 15h
Post-scarcity is always a relative term, because scarcity necessarily exists when we divide the Universe into more than zero agents or other than as one fully fulfilled agent. So post-scarcity, in my view, would be a state where any particular current good or service drops to near or essentially zero in price. Currently we are humans in AD 2026.
REPLY (1) SHARE Melvin 11h
If post-scarcity is a relative term then it doesn't seem like a useful term at all.
In practice, as some things become less scarce, we always find the scarcity is somewhere else instead.
Nowadays we already live in a damn-near post-scarcity world in many ways. The cost of food used to be the majority of a normal family's budget, with the cost of clothing way up there as well. Nowadays these costs are pretty trivial, you can minimally feed and clothe yourself for just minutes of work per day at minimum wage. But now the bottleneck is somewhere else, and the cost of land (or rather the cost of buildable land within commuting distance of a major city) has skyrocketed to soak up all of everyone's leftover money; the scarcity just moves somewhere else.
If the cost of manufacturing goes to zero then the cost of land, energy and raw materials will remain.
REPLY (2) SHARE Legionaire 10h
Rich is a relative term, and still useful.
Post scarcity to me means the things most commonly scarce today: big homes, free time, healthy and tasty prepared food, servants to do your menial labor
And if you want to stretch to include things that are apparently getting less scarce: ability to realize your artistic vision in movies or art or videogames.
Just because _something_ will always be expensive doesn't mean you can't get rich.
REPLY (1) SHARE EngineOfCreation 7h
"Big homes" is the best example why no economy will ever be able to provide everything desirable to everyone. Future-you might be rich compared to today-you, but the future mega-rich will always be able to outcompete you for the biggest homes on the nicest beaches.
REPLY SHARE JamesLeng 2h Edited
Solar panels are doing some impressive things to the cost of energy, and sufficiently cheap energy solves a lot of material problems too - basalt and seawater contain most of the minerals we need, hard part is distillation.
As for land prices, overwhelming consensus among economists is that Georgist LVT would solve the problem, it's just a question of implementation details and political opposition from entrenched rent-seekers.
REPLY SHARE idiotretardfool 16h
I think it is just weird to define "knowledge work" and "diffusion" as totally segregated concepts at this point. Isn't it weird if you can declare "knowledge work 90% solved" and "diffusion at 0%" simultaneously? Even if you personally understand that means "nothing in the economy changes", it's unlikely a layman reading the phrase "work is mostly solved" does; it semantically smuggles various political assumptions to even talk about work being solved at that point.
REPLY (2) SHARE Wisdom777 15h
Not really? "Machine faster than a horse" was 100 % solved engineering problem, but it basically took a century for global horse population to start declining.
REPLY (2) SHARE Melvin 13h
I don't know about this example: the important tech problem to solve was a machine that was as versatile as a horse, price competitive with a horse, not too much bigger than a horse and operable by one person like a horse. That was a tech problem, not a diffusion problem.
REPLY (1) SHARE MathWizard 7h
All of those features are subservient to "a machine that can travel faster than a horse" existing in the first place. It's impossible to refine your automatically moving machine to be cheaper and more versatile if they literally don't exist.
As of right now, smarter than human machines do not exist. It does not matter how cheap you make compute or electricity. If we discovered a magical portal to elemental plane of fully assembled GPUs and fusion reactors, making compute and electricity free, smarter than human machines would still not exist until AI technology progressed further and actually invented them. If you go to the majority of knowledge work jobs and tell them "I will pay for all of your AI costs if you immediately fire 90% of your humans and replace them with AI" most would not take you up on it, and if they did they would have a considerable decline in quality.
The most important tech problem is to invent the thing in the first place. The second most important tech problem is to make it cheap and versatile enough for economic adoption. And one has to come before the other, and there will be a time delay between the first and second.
REPLY SHARE Frikgeek 2h
Funny you mention that because "machine faster than a horse" also produced one of the biggest economic bubbles in history and the railway bubble is used as a common point of comparison by many AI skeptics.
REPLY SHARE Jeff 13h
It's a useful distinction, aside from the simple time lag of people adopting tech, different jobs can provide different barriers to adoption. If for example you created a LawyerBot who was unambiguously as capable as a human lawyer, it still wouldn't be authorized to practice law. You might in fact open yourself up to criminal charges trying to release such an AI to the public. These are distinctly diffusion challenges separate the technical challenges of getting the AI to do the knowledge work.
REPLY (1) SHARE idiotretardfool 13h
Don't get me wrong, I agree that diffusion speed is a Real Thing on the map.
The issue is that I think there is a third thing in-between the space of "this job task passes eval" and "the institutions / people / lethargy are blocking off deployment". Probably there are many ways to think you have 90%'d all knowledge work without actually 90%'ing implicit requirements everywhere
REPLY SHARE Bugmaster 16h
Define AGI as AI intelligent enough to do 90% of knowledge work jobs.
This definition is vague to the point of being disingenuous, which is a problem since the rest of your argument hinges on it.
What do you mean by "knowledge jobs" ? As we speak, many humans are still performing basic data entry, answering tech support calls according to a (natural-language) script, collating spreadsheets, and so on. Most of these jobs could be performed by a Python script and a SQL database; I would argue that 70..80% of them could be performed by ChatGPT. However, collating spreadsheets really quickly is not a skill that will let any agent, be he AI or human, recursively self-improve to near godhood.
Also, what do you mean by "90%" ? Are you counting the categories of jobs, or individual workplaces ? For every world-class scientist, there are probably 90 middle managers/HR paper pushers/legal compliance officers/etc. And yes, LLMs could probably do many of these jobs right now; but, again, the ability to spit out the correct form on demand is not enough to conquer the world.
But it gets worse:
Define the Bostromian superintelligence gap as the time between AGI and an AI which, if given independent control of resources like labs and factories...
You are once again falling victim to the same blind spot as many other AI-doom proponents: you are transitioning smoothly and effortlessly from "knowledge work" (manipulating symbols) to controlling physical resources, such as labs and factories (manipulating objects in the physical world). But this transition is far from smooth ! Present-day LLMs are not intelligent enough to even walk down the street unaided, let alone construct a chip fab from scratch. Self-driving cars are hitting a wall, both metaphorically and literally. Granted, there exists a lot of factory equipment that can currently be automated, and in fact is already automated: CNC lathes, PCR machines, etc. But you are positing a world where LLMs are creating such tools from scratch and recursively self-improving them, which at present they are utterly incapable of doing (without human assistance).
And it gets worse:
The easiest way to reach this point is for AI to become superintelligent at persuasion...
Do you have any evidence that super-persuasion (or indeed any other super-capabilities) are in fact possible ? Yes, some people are better at persuasion than others; but it does not automatically follow that there must exist an agent who can persuade everyone into doing anything it wishes all at the same time. That's like saying that some people can run faster than others, therefore there must be an agent who can run faster than light.
The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time.
My objection is similar here: how have you determined this ? If this were true, then we'd expect many if not most human experts to become world-class leaders in their fields, but this is not true. Most physics teachers never become Einsteins; and in fact on the planet of 9B people, there are only a handful of "top geniuses" -- and you're saying that bridging that gap is virtually trivial ?
Look, I don't think I'll be able to persuade you to my side; I understand that you take these assumptions for granted and that rejecting them seems utterly foolish to you. All I'm trying to do is illustrate that they are by no means obvious, nor that rejecting them necessarily makes someone an idiot (granted, I personally could very well be an idiot for many other reasons). All I'm saying that if you want someone like me -- i.e. a pretty average normie -- on your side, then you need to do a lot more than simply state your convictions very strongly. I've got other people stating their convictions at me all the time; I need evidence and specificity, not mere vague assertions.
REPLY (6) SHARE Taleuntum 15h
I don't think Scott means superintelligent at persuasion as "persuade anyone into doing anything", just that it's much better than the best human at persuasion.
Fwiw, I think the extent to which an AI can influence a human will be bottlenecked by the human's own abilities. "A smart person rarely loses an argument. A dumb person never does." However, the status accrued by being right a million times and being super useful will move mountains for it.
REPLY (1) SHARE Bugmaster 15h
I don't think Scott means superintelligent at persuasion as "persuade anyone into doing anything", just that it's much better than the best human at persuasion.
This is what I meant by "vagueness". Ok, it's "much better" than the best human on persuasion -- but what does this mean ? Is there some Persuasion Quotient, and if so, how is it measured, and how many points does one need to be "much better" ? In practice, what can someone like that actually tangibly achieve in the real world (other than "persuading anyone into doing anything") ? Again, how did you arrive at this prediction, assuming that it is even quantifiable ?
However, the status accrued by being right a million times and being super useful will move mountains for it.
Will it ? Humans are not persuaded by other humans who are right a million times; this is why many (if not most !) Americans do not believe in the efficacy of vaccines, or the Moon landing, or many other such things that are touted by the accursed "experts". And being super useful arguably ends when persuasion begins -- otherwise, you wouldn't need to persuade at all. I don't need to be persuaded to drink water or breathe air; I do need to be persuaded to drink the latest healing energy concoction while breathing artisanal O2 from a canister.
REPLY (2) SHARE Melvin 10h
Persuasion isn't about sequences of words anyway, it's about trustworthiness. You're likely to persuade me of something if I think that (a) you are likely to know the right answer and (b) you're likely to be telling me the truth, either because you're particularly honest or because you have no incentive to lie.
"This is a great car, you should buy it" is pretty persuasive coming from my tame mechanic, it's very unpersuasive when it comes from the dealer who is trying to sell me the car.
My calculator is a super-persuader, because every time it tells me something I believe it. Zero skepticism, I believe everything that Mr Casio has to say about arithmetic up to ten significant figures.
REPLY (2) SHARE Bugmaster 8h
Then it seems that you can only be persuaded to believe something which you are already nearly certain to be true. This is a pretty good epistemological stance, but it has limited applications as far as a (hypothetical) malicious super-persuader is concerned. He can very easily persuade you that 2+2=4, but he can't persuade you to give him all your cash.
REPLY SHARE The Ancient Geek 12m
Persuasion isn't about sequences of words anyway, it's about trustworthiness
It's about both. If sequences of words were zero percent effective, advertising wouldn't work.
REPLY SHARE Xpym 7h
In practice, what can someone like that actually tangibly achieve in the real world (other than "persuading anyone into doing anything") ?
Establish a worldwide Fourth Reich, is the usual argument.
REPLY SHARE Wisdom777 14h
It's easy to make such a definition (knowledge jobs) not vague. Just use the classification of work by some labor ministry, eg. US Bureau of Labor.
Well, they are intelligent enough if there are next to no obstacles on the street. But such LLM can still pay humans to do physical bidding, and we've already seen limited early versions of this.
You only need normal persuasion to get a large number of people to do serious damage. See the history of conflict for spiritual-ideological goals. Achieving super x is easy for programs who are already human-level x because they have fewer needs and physical limits. You do not need even close to unanimous agreement to have power over the planet. Analogically, we can look at currently existing countries, such as France, where President Macron only has 10-20 % approval rate.
No, we would not necessarily expect most arbitrary experts to become world class. What we would necessarily expect is some metric which shows easier attainment, yes, but not that particular one. We do, indeed, observe some other- the ratio between the number of base human skill in x and expert humans at x is much larger than the ratio of expert humans at x and world class humans at x. So it is still a lot easier, but not amazingly easier.
REPLY (1) SHARE Bugmaster 14h
But such LLM can still pay humans to do physical bidding...
Yes, of course; but so can you or I. If the LLM needs to pay humans to do the bulk of its work, then it can recursively self-improve in the same sense as human corporations already do. This indeed makes it a threat, but not an unprecedented one.
See the history of conflict for spiritual-ideological goals.
Can you be more specific ? I would argue that there are approximately zero large-scale conflicts in human history whose origins could be traced back to a single super-persuader persuading people to do his bidding; but I might be mistaking your meaning.
Analogically, we can look at currently existing countries, such as France, where President Macron only has 10-20 % approval rate.
If the LLM could achieve about as much power as Macron, then it might be a threat, but IMO not a major one (compared to the other threats humanity is facing).
the ratio between the number of base human skill in x and expert humans at x is much larger than the ratio of expert humans at x and world class humans at x.
What do you mean by "base level" ? Most people cannot do most things at all; but then most humans can learn to do many things by watching a 10-minute YouTube video. Becoming an expert usually takes 4...8 years of focused study. Becoming a world-class genius requires... well, no one really knows what, but most experts never achieve this at all.
REPLY SHARE Cjw 13h
Scott had a lengthier discussion about what superpersuasion could be, whether it’s even possible, what an army of good but not magical persuaders might do, etc, to this point. It’s in his AI2027 impression article, I think there may have been a lengthier treatment somewhere else but couldn’t find it
REPLY SHARE PoulyDeer 11h
On a planet of 9 Billion, there are roughly 1000 geniuses (they know each other). Or were, maybe four years ago. Some of them are dead now (the FBI is investigating).
REPLY SHARE Xpym 7h
And also, why 90%, and not 99.99%? I though that's what AGI usually implied.
REPLY SHARE Frikgeek 2h Edited
I think Scott once stated(but now i can't find where) that an example of top human-level persuasion would be Mohammad , and a super persuasive AI would be above that.
So the AI would need to find some number reasonably charismatic humans who are open to its ideas(or can be bribed or manipulated into working with the AI), then use them as a front and have them become the most popular politician/activist/internet influencer/whatever whose words are implicitly trusted by a huge percentage of the human population and then subtly push its agenda through them.
Not sure how realistic this scenario is but it would be an example of "super-persuasion".
REPLY SHARE Some Guy 15h
I have nothing to add other than: diffusion is extremely hard and I can attest to it.
I know I present myself as a smarty pants know it all, but there’s just a global problem of leaders not knowing how to actually use and deploy the tech and who to trust and who not to trust that is still going to take years to work out. But I do expect that will be a hockey stick timeline.
REPLY SHARE Paul 15h
Based on Fable we have reached the limit of this paradigm. If you really thought there was 25% chance of AGI you would be acting drastically differently.
REPLY (1) SHARE Taleuntum 15h Edited
How is Fable evidence that we have reached the limit of this paradigm?
REPLY SHARE Garloid 64 15h
What happens when they turn the simulation off but it keeps running due to Dust Theory? They may have already done this, in fact.
REPLY SHARE Jeffrey Soreff 15h
Many Thanks for gathering your conclusions and reasoning on all these considerations into one post! One comment, re:
Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.
As you wrote, superpersuasion, if it happens, might be before diffusion. If that happens, I would expect it to shrink the timelines for diffusion in at least two ways:
- A superpersuader is a supersalesentity. They should speed adoption of AI throughout the economy.
- A superpersuader is a superlobbyist. They should speed modifications of regulations (or even just the regulations' enforcement) to speed AI's spread.
REPLY SHARE Tolaughoftenandmuch 15h
Did I miss where power consumption and power cost become bottlenecks (or why they wouldn't)?
REPLY (2) SHARE moonshadow 7h Edited
There’s a lot of earth’s surface still that we can ̶t̶i̶l̶e̶ ̶w̶i̶t̶h̶ ̶p̶a̶p̶e̶r̶c̶l̶i̶p̶s̶ cover with solar panels before we even seriously start competing with human use. Current friction is for economical, not inherent, reasons. Other problems seem less far off.
REPLY (1) SHARE EngineOfCreation 7h
I mean, define "seriously"?
"Half of all new electricity demand in the U.S. last year came from data centers"
https://fortune.com/2026/04/20/us-data-center-electricity-demand-public-opinion/
I would call that pretty damn competitive.
REPLY (1) SHARE moonshadow 7h
It’s not zero-sum, though. Data centers being planned now include things like solar arrays on site. My point is - if they pay for generation capacity to be built up to match the use, and it’s not competing with me for useful land or polluting my air or water, why does it matter?
REPLY (1) SHARE EngineOfCreation 7h
What do you think of this report? Does this count as "serious" competition?
https://fortune.com/2026/05/12/lake-tahoe-data-center-49000-residents-power-source/
REPLY (1) SHARE moonshadow 6h
This describes an economical decision. It’s not that new generation capacity couldn’t be built. It’s merely that no-one wanted to pay for it, and no-one was made to. That’s a policy decision, not a problem inherent to the technology. We are capable of making better decisions, and often do.
REPLY (1) SHARE EngineOfCreation 6h
Would you define "serious competition" then, if it would be neither political nor economic?
REPLY (1) SHARE moonshadow 6h Edited
Competing with humans for arable land, not just for economic reasons but because we've used up all the other kind. Generating power in ways that pollute humans' environment significantly, not just because it's cheaper than renewables but because we've already built all the renewable stuff we can. Any such competition caused by physics rather than merely by humans deciding it's fine to screw over other humans for money and local government agreeing they should be allowed to do that.
Why was the data center built there rather than elsewhere? Why did it use grid power instead of building its own capacity? Why was the power company able to just turn off power to the local community? None of these decisions are specific to the technology, and at each step someone could have made better choices but didn't.
If the local power company will screw the local community over for a data center, that says to me it will do it for anyone else willing to pay that amount of money, and this is the real problem that needs fixing if we genuinely do not want the screwing-over to happen. Traditional ways of fixing this kind of externality are some combination of imposing costs associated with the externality on the people making the decision, and regulation / contract / franchise agreements with local government to guarantee some level of service before public utilities can operate in the area. This provides incentive to build capacity to match demand before redirecting supply.
If we aren't willing to bite that bullet, attacking the specific entities throwing wads of cash at the power company today /just exactly the way we all decided they could/ is merely tilting at windmills.
If we think there is a problem here that warrants solving, we should /solve the actual problem/, not merely one instance of one symptom.
REPLY (1) SHARE Victualis 5h
Thank you for laying out the argument so carefully.
REPLY SHARE Scott Alexander 6h Author
I don't think this is a big deal. AI consumes power via data centers. Most localities don't like the idea of data centers consuming power from their grid, so the data centers are forced to use their own natural gas plants on site. This is expensive, but so far they've been able to meet demand, and the financial people aren't projecting it as likely to stop the data center buildout of the next few years. If the government deregulates solar or nuclear, that makes things even easier.
But also, if AI improves through recursive self-improvement, then existing data centers can run better AIs for the same energy cost.
REPLY (1) SHARE Tolaughoftenandmuch 4h
I'll go out on a limb and predict this will become a significant bottleneck within then next 3-5 years. Let's see who is right!
REPLY SHARE Tj 15h
FWIW, this showed up in my email as a completely blank email. Thought that was the message LOL
REPLY (1) SHARE Melvin 14h
Only on the day Scott reaches true enlightenment.
REPLY SHARE Habryka 15h
As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone.
I don't think this makes sense. Most people will be well within the future Lightcone of the dictator, and the dictator can just send Von Neumann probes to every solar system in the reachable universe, and so have a presence there. I can't currently think of a thing you meant to say here that makes sense to me.
I do think the rest of that section is correct and I am not very worried about negative outcomes from AI-enabled authoritarianism.
REPLY (1) SHARE Scott Alexander 6h Author
Hmmm...I think I was thinking of the dictator (sitting on Earth) having some whim, but people on Andromeda are protected from his whims (at least for the next million years). I agree that the dictator could institute a law code that is sent with the von Neumann probe that colonizes Andromeda, but this seems less bad for the normal reasons that governments of laws are better than those of men.
I guess the dictator could send an uploaded copy of himself to dictate to Andromeda, but I don't know whether real-world dictators would behave that way.
REPLY SHARE Melvin 15h
Looking at the big picture, I can't share the optimism of the last section if human labour does indeed become obsolete.
The thing that has allowed humans to tolerate each others' existence (or rather, the existence of people outside their immediate clan) is that humans have economic value to each other. I am better off for living in a world that has strangers in it, because those strangers produce goods and services which we can trade.
But if we get to the point where the vast majority of the human population has no economic value and exist only to consume the value created by the robots, then humans no longer have a good reason to tolerate each others' existence. I won't try to predict the exact sequence of events but I think that once the interests of everybody become strongly aligned with killing everyone that they don't personally know and like then it has to lead, in some form, to a war of all against all.
REPLY (1) SHARE Cjw 12h
I also think that humans having no economic value will be a major problem, for example it’s why UBI schemes are doomed, people will lack the leverage to make anyone else stick to such a thing once labor peace is no longer needed.
But I don’t necessarily think it would be a matter of exterminating everyone else, that’s always been an option even in ages where other tribes of humans did not provide value because your land could only produce X units of grain no matter how many laborers you threw at it. The people in those eras might have gone marauding, and some did, but there was a value to mutual peace and not having to be hypervigilant all the time. I think we will be less likely to aid others but no more likely to want to eliminate them than a medieval villager, which is maybe more than now but not insane.
REPLY (1) SHARE Melvin 12h
Right, the idea of a population who mostly just sit around on UBI is horrifying. The minority who pay into the system will be very aware that they'd be better off without the masses. And the masses have nothing better to do all day than to sit around and agitate for their UBI to be increased.
So either you've got constant conflict between the haves and the have-nots, or you set up a clever system of repression to ensure that the have-nots don't complain too much.
One plausible equilibrium would be a Chinese-style system where your UBI is scaled to reflect your degree of loyalty to the system. In practice it might look a lot more complicated than that.
REPLY (2) SHARE JamesLeng 1h
Current welfare systems involve a lot of people sitting around doing nothing, or causing pointless problems, because if they try to do anything legibly productive or profitable, some means-tester might decide that means they're too successful to qualify for benefits, and those benefits being otherwise judicially untouchable means they can't be meaningfully held responsible for any messes they make.
Proper UBI would have neither of those problems.
REPLY SHARE The Ancient Geek 1m
"One plausible equilibrium would be a Chinese-style system where your UBI is scaled to reflect your degree of loyalty to the system"
Another is where it is scaled to reflect your loyalty to the Billionaire who is doling it out.
REPLY SHARE anton 15h
Pausing has enormous opportunity costs. I'm strongly against this. Alignment is at least interesting as a research project now that we have some idea of how this things work, and a less good idea of how they might work in the future. But to the extent they cause any slow down, which I think is unlikely, it'd also have enormous opportunity costs.
REPLY (1) SHARE Scott Alexander 6h Author
Pausing only has enormous opportunity costs if you expect AI to be enormous. If you expected AI to (for example) be able to dismantle Mercury and build a Dyson sphere with in in five years, I think you would be freaked out enough to be willing to slow down and make it twenty years.
If AI will just make B2B SAAS companies 10% more efficient, then I agree it's not scary enough to pause over, but then the opportunity costs of pausing are commensurately smaller.
REPLY (1) SHARE JamesLeng 1h
There's a middle ground between those, where pausing means a lot of people die from e.g. cancer, malaria, or various collateral damage from logistical and diplomatic incompetence, which the un-paused AI could have cured a few critical years sooner.
Dismantling Mercury and building a Dyson sphere in five years is absurd just from a thermodynamic standpoint. If you're defining anything less than that, i.e. all physically possible outcomes, as "not enormous," there's something wrong with your scale.
REPLY SHARE Kevin Lacker 14h
I don't quite understand the definition of "AGI = AI intelligent enough to do 90% of knowledge work jobs." What's the denominator here? Jobs that exist today, or jobs that exist in the future?
The problem is that the requirements for a "knowledge work job" will be constantly redefined to include whatever it is that AI can't do. We already see that happening in software engineering.
So do you mean, able to do 90% of the jobs of today? In which case sure, but who cares.
Or do you mean 90% of the jobs of the future? Because how does that even make sense, we don't call it a "job" if it's an AI doing it.
Or do you mean something like, the total number of knowledge workers drops by 10x?
REPLY (2) SHARE Melvin 14h
Furthermore I suspect there's a lot of individual jobs where AI can do 90 percent of the job, but you need a human for the other 10 percent (deciding what needs to be done, talking to other humans about it, checking the AI's work and taking responsibility for the finished product).
REPLY (1) SHARE JamesLeng 39m
And if that one human can now handle what used to be ten people's worth of work, without capturing ten times the pay, while the objective value of that work to the wider economy stays the same... it starts making sense to hire a lot more people for that role, to try things which previously weren't worth the effort.
REPLY SHARE Melvin 14h
Furthermore I suspect there's a lot of individual jobs where AI can do 90 percent of the job, but you need a human for the other 10 percent (deciding what needs to be done, talking to other humans about it, checking the AI's work and taking responsibility for the finished product).
REPLY SHARE Kyle Star 14h
Does Scott believe there’s a 66% chance we’re in a simulation, full stop? Because that’s how I’m reading that last part.
If so, unfathomably based and that’s where I’m trending too. I always think people like Scott and Elon have an even better reason to think they’re in a simulation than the rest of us, so it’s good to see Scott’s rationality take him to what I see is the natural conclusion for him.
REPLY (1) SHARE Katie 11h
Wouldn't they only have more reason to believe they're in a simulation if they believe the rest of us are p-zombies??
REPLY (2) SHARE Kyle Star 11h
Yes. It’s knowledge we can disprove from our points of view but they can’t.
REPLY SHARE Scott Alexander 6h Author
I don't think a binary pzombie or not pzombie flag is the right way to think about it. More like how much processing power they're spending on each person, or something like that.
REPLY SHARE spaceman 14h Edited
This post made a friend (an Ivy league undergrad at a very good Ivy) reconsider suicide. 2027 timelines felt "too soon" for them to do anything except face the future helplessly. AGI possibly arriving in the early 2030s feels like enough time to have a chance at controlling their destiny in some small way.
Thank you very much for this post; I think it made a meaningful difference to their mental state.
(I feel very bad for the world's 14 year olds who will be in the same situation 3 years from now.)
REPLY (1) SHARE Scott Alexander 6h Author
Uh, this isn't the main reason why you shouldn't commit suicide, but I would tell your friend that suicide seems like a really bad response here for even more than the normal reasons. If you model the AI transition as a 50% chance of death, and a 50% chance of utopia, then suicide doesn't decrease your chance of death (in fact, it increases it to 100%), it just means you don't get any chance at the utopia outcome.
I think usually people considering suicide over world events are not really doing it over politics and are normally depressed and using world events as an excuse. I understand this is going to come across as condescending, but for what it's worth I would inevitably-condescendingly urge your friend to seek normal psychiatric care. If they've already done that, they can try the second to nth-line stuff listed at https://lorienpsych.com/2021/06/05/depression/
REPLY SHARE Wombat3000 14h
This is sort of tangential, but what do we think about the nature of intelligence and rationality? What basic mental functions are they evolved from? Do they come from concern? Condescension? I've been trying to understand the social mechanisms of it and why it evolved. What do we think the first words were? The reason I'm thinking of it is not just because of AI, but intelligent people's relationship with the new and unknown. I'm hearing a lot of negativity from smart people about all sorts of novel things: AI, social media, genetic editing, etc. It just seems consistent enough that there may be a psychological explanation to it.
REPLY SHARE The Unimpressive Malcontent 14h
It took the latest ChatGPT 5.5 model five attempts to put the correct number of significance stars in each cell of a table I was making the other day. I even showed it the number of stars that were supposed to go in each cell.
When I ran into some obviously bad observations in the data, it told me to just delete them. The correct solution was of course to go back and find out why those observations looked bad to begin with.
Things like this come up over and over. I see too much emphasis on "knowing" and not enough on "doing." And to that end, I suspect there is an optimism gap between people who regularly use LLMs to help them with their technical work, and routinely run into its limitations; and those who don't.
REPLY (1) SHARE Victualis 5h
The papers reporting on successfully working with LLMs on actually challenging problems (not in the training distribution) always have a lot to say about how a lot of effort was needed to work around the limitations of using a system that had the wrong distribution. If Scott had some free time (seems unlikely for several years) then he could do a deep dive with an LLM into some aspects of psychiatry that are not well understood but that his practice has lent him deep insight into, and he would probably bump against the same issues. Gwern has clearly bumped into the "dragons be here" part of the LLM map where poetry lurks, and I think most people have something they understand really deeply that has little training data and this shows up in LLM limitations in that region. However, most LLM use is wandering around the terra cognita and when the LLM veers into incognita one only notices this if one actually has real expertise there that contradicts what the LLM generates.
REPLY SHARE Peter Defeel 14h
I’m going to ignore the AI world takeover arguments. There are broad jumping to conclusions here.
The economic effects are possible. Even with what we have now. Without AGI or super intelligence.
If you’re in one of the early industries to be affected by AI, you may have a very bad time before the economy can grow 100x or 1000000x. I wouldn’t describe this as a “permanent underclass” - it’s a subset of people, and their suffering is temporary - but it might be a very large subset, and it might continue longer than you can remain solvent. I agree it’s worth having savings ready to prepare against this scenario.
There’s a large leap there to the economy jumping to 100x or 1000000x. There are multiple rebuttals here, not least that the world doesn’t have the resources to do this. Nor can LLMs run factories. Nor can automation save you. Factories are highly automated anyway.
The underclass problem is real though, and very likely. What isn’t likely is that there’s a magic jump from a permanent large underclass to everybody being rich. Not in the present system.
The economy depends on demand. Reduce the number of jobs and you reduce demand, reduce high income jobs and the downstream effects are even more significant. Do all this and government income and ability to borrow also collapses, so where does the UBI come from? Buying shares in Google or Apple won’t help you, as demand for advertising and iPhones also falls.
It’s possible AI companies might soak up whatever profits are left but they also sell to customers - often the high income tech heads first for the chop - and corporations that will find demand falling. The latter may respond by replacing more workers with AI but that will just exacerbate the problem. Even AI companies will see reduced demand over time.
There is a branch of macro economics that denies this, claiming that supply creates its own demand, and economies somehow self correct. This neither explains the depression, nor the Engel’s pause, but even if it were true heretofore it’s not necessarily true in the future. There’s no guarantee that an increase in animal spirits (or more prosaically a reduction in interest rates) will encourage businesses to hire more workers when they can spend more on tokens.
The Engel’s pause is interesting. During that period - the early Industrial Revolution in Britain - output and productivity rose substantially, yet real wages for many workers stagnated for decades. Production increased, but the benefits were not broadly shared. Hence the rise of communism, socialism, anarchism and other revolutionary ideologies.
Classical economists, like Riccardo and Lasalle, at the time generally assumed that was the nature of things, it was the end of wage stagnation that surprised them. After all, if a factory becomes twice as productive, there is no law of nature that says workers’ wages must double. Many labour economists credit labour unions for increasing wages and thus aggregate demand. This is my preferred theory.
The power of unions though, depends on the power of labour to withdraw collectively from work. The unemployed have no such power. And while the Engels pause was significant enough, it didn’t create a permanent underclass, and wages stagnated. They didn’t fall.
So unless you find a way to induce demand - and a tax based UBI isn’t going to work when government revenue is falling - not only will you not get to 1000x the 2026 economy, you will be lucky to get to 1x.
I’m hoping to be proven wrong of course, and I’m open to suggestions.
REPLY (1) SHARE Throw Fence 🔶 4m
I agree that the consumer economy can't survive this, but that doesn't mean production will stop or can't grow. Stuff gets made because whoever controls the AI wants it made, not because it can be sold. An economy that grows while routing around most people. And that's not great, why would such a system produce anything for us? At this point I feel like you've just invented Alignment from first principles.
REPLY SHARE Nick Luchs 14h
A few small typos:
"a company need an IT department"
"are aren’t personally interested"
"it’s only a been a few years"
More importantly, this sentence seems orphaned/misplaced. Nothing but a new section follows: "Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028."
I hate to obsolete myself, but I ran this article by Opus 4.8 and it also noticed all of the above and more, with very high signal/noise ratio: https://claude.ai/share/e6ee8a4f-8dfb-4cde-8d5c-e0b24fc72224
Nitpicking (hopefully helpful) aside, thanks for writing this. I really appreciate when writers I like occasionally take the time to write up summary/reference docs like this, rather than all of their writing being a sort of unending series of updates to their previous corpus.
REPLY SHARE Nicholas Halden 14h
In terms of “smart enough to do 90% of knowledge work,” do you agree the work remaining is more about agent scaffolding and specialization rather than the base model? IMO the “smart” part is definitely there for 99% of jobs.
REPLY SHARE Josh Haas 13h
This mostly all checks out to me except the 66% simulation one — seems way high. Even if you don’t believe “weird hypotheticals without any empirical grounding can be ignored” is a legit reason to dismiss the simulation hypothesis, it’s a weak argument on its own terms. Computationally, simulation to full fidelity is wildly inefficient — the amount of information needed to fully describe the state of a computer is way more than the amount of information the computer can contain. So a simulated universe would have to exist inside a computer vastly bigger than the universe. Simulations humanity have run to date have been way too low fidelity to support the evolution of intelligent life. So the assumption that universes simulate other universes simulating other universes seems extraordinarily low likelihood given what’s been observed in this universe!
REPLY (4) SHARE Taleuntum 13h
You criticise the simulation argument, but I don't think you have read the paper introducing it. It answers your objection explicitly in its third section. That section is merely 3 pages long. See "Are you living in a computer simulation?" by Nick Bostrom
REPLY SHARE Tossrock 13h
I used to think this way, but it kind of falls apart when you realize that from the point of view of any individual, almost nothing needs to be fully simulated. Foveated rendering of reality, essentially. Throw in the capabilities of even our own modest graphical models (gaussian splatting / NeRFs, etc) and it quickly becomes obvious that simulating a world for a single person would actually be quite achievable. Things get more expensive when you want multiplayer, consistency, etc, but even then it wouldn't be that crazy.
REPLY SHARE PoulyDeer 11h
Simulation to full fidelity is NOT what we're looking at, look at the information theory inherent in the particle-wave duality of light. Or the electron cloud conception. There's fundamental information theory arguments that we're in a simulation, that just "happens to look normal" on the Newtonian scale of things.
REPLY SHARE JGracq 1h
Not only what you say is right. But even if we assume that we are in a simulation, the probability that the human history is part of the data that the ones running the simulation are interested in is necessarily really small. You don't allocate that much compute to run a simulation during billions of years and spanning a least billions of billions of light years in space so that the simulation is considered finished when a tiny group of atoms in a single of point of this space act organically or improve their own development in an exogenous way. Otherwise the simulation would have been finished long ago by bacteria or at worst dinosaurs. We likely still have many thing to accomplish to complete the data, if we are even part of the data.
REPLY SHARE Doc Abramelin 13h
I have a question about the superpersuasion scenario because it sounds like magical thinking; to my knowledge, this is like grey goo, compelling in genre fiction but not necessarily feasible under the laws of physics. My only analogy here is to think about the guy who is handsome, charismatic, cultured, rich, well-spoken, athletic, whatever. He's quite good at persuading people (==women) to give him what he wants; does he succeed every time? Certainly not. Does thinking at machine speed or at a hypothetical superhuman intelligence level allow for this?
REPLY (2) SHARE Bugmaster 13h
I completely agree. To be fair however, the usual counter-argument (assuming I understand it correctly) is that "thinking at machine speeds" and "superhuman intelligence" can overcome essentially any intellectual challenge -- since most such problems have solutions; solutions are achieved primarily through computation; and the superintelligence has effectively unlimited access to computation (by the combined virtue of being very fast, and also being able to devise massively efficient algorithms). Thus, assuming that super-persuasion is an intellectual challenge (which seems likely), and that it does have a solution (as is usually assumed), then a superintelligence would be able to solve it (and to do so nearly instantly).
REPLY SHARE Scott Alexander 6h Author
Think of the guy who is very strong and knows karate. He's quite good at winning fights; does he succeed every time? Certainly not.
...now imagine a guy holding an AK-47.
REPLY (1) SHARE JamesLeng 18m
A gun is a huge advantage in a fair fight, but super-persuader doom scenarios require overcoming social challenges that are wildly - and very much deliberately - unfair in the defender's favor. One guy with an assault rifle isn't going to have great odds storming even a pre-gunpowder castle guarded by as many troops as he has bullets, because some of those shots are going to miss, and the time he spends aiming to mitigate that is time they can spend diving behind thick stone walls, or shooting back.
REPLY SHARE Epistemic Puddle 13h
What are your thoughts on the moral obligations of individuals working on AI? Eg. capabilities researchers, those working on building data centers, or those tangentially related in the functioning of big labs but not directly contributing to capabilities (anything from financing to being a normal employee like a marketing professional or janitor)?
REPLY SHARE Seta Sojiro 13h
I'll insert my usual complaint about models being around a million times less efficient that humans at learning. Dwarkesh has made the point more eloquently than I can here*.
Having established that, I still cannot understand how anyone can look at the sample efficiency issue and then conclude that super-intelligence is plausible by just throwing more data at models. I know Dario believes this - that if you train a model on a sufficiently wide variety of tasks then it'll generalize to everything and become super-intelligent, and so I assume most safety advocates like Scott agree? I have two questions.
First, where exactly are you going to get that data? Training on the text of the internet along with several libraries worth of textbooks equated to a few trillion tokens and produced GPT-4. Training on synthetic math and coding data produced somewhere in the ballpark of 100 trillion tokens and produced the current generation of models. Where do you get the data for everything else that isn't quantifiable?
And second, if it were the case that training on lots and lots of tasks causes the model to generalize then why haven't we seen this yet? New models are often worse at all of the benchmarks that the labs don't highlight (for example Opus 4.7 is worse than 4.5 at long context retrieval). And the sample efficiency problem has worsened with each release - to get the same level of increase in capabilities we've needed progressively more tokens. Even within domain they don't generalize. After training on 10 trillion math tokens, it isn't trivial to get it learn a new domain of math, you need trillions more to get a significant improvement. Compare that to a human mathematician who can read a textbook or a few papers, work through a few problems and learn a brand new domain with at most a few hundred thousand tokens, probably a lot less since they only need to skim.
*https://www.dwarkesh.com/p/the-sample-efficiency-black-hole
REPLY (1) SHARE Scott Alexander 6h Author
I did sort of mention data efficiency in the "arguments for later" section, but you're right that I don't think it dominates, for a few reasons.
First, it seems like you could have made this argument any time since about 2023 (proof: I've heard this argument since 2023). Since then, we've still managed to squeeze more and more gains out of AI. I admit I'm not entirely sure how - I think we used up most Internet text by 2023-2024. It seems to be a combination of training more times on the same data, organizing the data more efficiently, adding new modalities like video, plus a bit of reinforcement learning and simulated environments. I'm surprised that this has worked so well but given that it has I don't know enough about it to think it will stop working sometime in the next few years.
I will have to think about the generalizing thing more - it seems to me that new models are mostly better at things. For example, Fable is descended from Mythos which seems to have been optimized for cyber, but it's also notably better than past models at writing, constrained writing, and humor.
REPLY SHARE Metacritic Capital 13h
What is your Anthropic gap? The period between when Anthropic has strong evidence that they achieved Recursive Self-Improvement and the date they tell the public.
Following up. Conditional on Anthropic not declaring RSI by the end of 2027, how much do you push your AGI estimates?
REPLY (1) SHARE Scott Alexander 6h Edited Author
Anthropic has already said they've achieved RSI, see https://www.anthropic.com/institute/recursive-self-improvement . I agree it's kind of annoying that RSI means both "some distant extreme thing that leads to superintelligence" and "the normal stuff where having better tools makes you go faster", but I think these are conceptually linked enough that this is impossible to avoid, and that Anthropic's doing the right thing here.
REPLY SHARE Dylan Black 13h
These are very reasonable probabilities and arguments generally! I agree with ~most of them, with the only caveat being that I do think the probabilities are too linearly distributed (units of 10%), as opposed to logarithmically.
REPLY SHARE Jaime Sevilla 13h
Jaime from Epoch here.
Views at Epoch are varied, more than other orgs, but we mostly expect pretty fast diffusion.
I personally also expect AI that can solve milennium problems and automate most AI R&D by end of decade, and automate all cognitive work within 10 years.
I also expect the automation of AI R&D to not lead to a 1M improvement in sample efficiency in a single year immediately, as I expect AI development to be bottlenecked on data and compute.
And I expect most of the self reinforcing loop of accelerating growth will be mediated by applications other than to AI R&D, for similar reasons to why only 10% of the global economy is R&D today.
In slogan form, I don't believe in a software only intelligence explosion, but I expect an industrial explosion to happen in the 2030s, shifting us back to accelerating growth.
REPLY SHARE MichaeL Roe 12h
The LLMs we have so far seem remarkably well aligned.
While I suppose there could be some threshold where making them smarter suddenly makes them evil ( e.g. increased Omohundro drives) I think the likely thing is that they remained aligned as we scale up how capable they are.
REPLY (1) SHARE Scott Alexander 5h Author
I sort of agree with this, but see https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me for a convincing argument against.
REPLY SHARE Breb 12h Edited
Re: Warning shot. You seem to be assuming that a warning shot is necessarily caused by misalignment of frontier models; I think it’s fairly probable that if it happens at all, it’s caused by misalignment of non-frontier models (e.g., criminals or terrorists committing an atrocity using an open-source model) or by human error (e.g., delegating extremely important work to an obsolete AI, and not checking the results before implementing them).
Re: P(Utopia to inhabitants) vs P(Utopia to us). I think the former is very low – much lower than the latter – because humans are very reluctant to describe their circumstances as utopian, and will usually find trivial unsolved problems more salient than major solved problems. (While in principle future humans could modify themselves not to be biased in this way, I’m not optimistic about this; the median human usually rejects opportunities to think clearly and be unbiased)
REPLY (1) SHARE Scott Alexander 5h Author
I agree that criminals or terrorists doing something bad with a nonfrontier model is plausible. I guess I am conflating many different kinds of warning shots that would warn different things.
REPLY SHARE MichaeL Roe 12h
LLMs are currently very good at answering questions where there’s a well-known right answer that fits within a screen or two to text. I think we are just not there yet for automating most knowledge work jobs.
REPLY SHARE Christopher 12h
Counter argument to "Safety: Arguments for optimism"
The argument doesn't take reward hacking or gradient hacking into account!
This isn't theoretical! In the "faking alignment" paper they showed that Claude could use gradient hacking to defeat RLAIF trying to align it.
(Also, who was concerned that AI couldn't learn human values, I thought that was a straw man? 🤔)
REPLY (1) SHARE Scott Alexander 5h Author
I think of that as one of the things they'll be trying to defeat with all their prosaic alignment methods, which might either converge towards working or not working.
REPLY SHARE Presto 12h
This post did not go where I expected it to go.
(Let the simulators know that by posting here, I'm VERY important for the simulation alongside my family, butterfly flapping its wings etc.
Amen.)
REPLY SHARE Dhay 12h
Read a bit, saving it for later (hopefully).
My take is that these probabilities are a total unknown. I would assume against the ASI scenario, because I don't see what could be done to improve our odds against an electronic organism way beyond comprehension, other than a successful Butlerian Jihad significantly earlier than ASI is achieved anyway...
On the other hand, an AGI (or kinda AGI, like super-Mythos) that makes political and economic elites have intelligence superpowers, mass surveillance capabilities, data analysis and inference and automatic warfare in a scale much greater than today is more realistic and almost as scary. Even if it creates material post scarcity, life can be horrible in other ways as these powers fight for dominance, ideology, religion, the attention economy etc.
REPLY (1) SHARE Scott Alexander 5h Author
Against the claim that "the probabilities are a total unknown" implies "don't estimate them", see the section in https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist called 2. Probabilities Don't Describe Your Level Of Evidence And Don't Have To.
REPLY SHARE Sufeitzy 12h
Without visual, time, physics and other reasoning constructs, it can’t even construct a description of a pile of colored balls.
Without a neuromorphic model, the power required is infeasible.
Never going to be general intelligence without agency.
It will remain a 2nd derivative of texts it has absorbed which are only a fraction of published human knowledge. I dont see these issues being fixed for decades.
The current LLM model was present as far back as the late 80’s. Not so for visual and other reasoning.
REPLY SHARE mikolysz 11h
I think people heavily underestimate diffusion speed. PCs took a while to diffuse because they were expensive, and you had to learn esoteric DOS commands to get any utility from them. You couldn't try them over the internet and be convinced in 5 minutes, an art Silicon Valley has perfected.
THe first true coding agent, Claude Code, was released just over a year ago. At this point, most programmers are already using one, even in my Eastern European backwater, which is as far from Silicon Valley as you get. Chat GPT is a similar story. Regulation (and lack of training data, Github + RLVR makes programming the easiest domain to tackle) is going to slow this down for other professions, but not by much. OpenEvidence isn't far behind Claude Code in medicine.
I do expect a multi-year transitional period where you'll still need doctors to perform physical examinations and lawyers to deliver in-person oral arguments, but where all the internal "computer work" will be almost entirely relegated to AI. This is going to destroy the value of these professions; if lawyers can suddenly take 4x as many cases and many people won't even bother with hiring one, their premiums will go down.
I expect there will be a few stragglers, particularly in fields where there are no incentives to fire people. Public school teaching and government paperwork handling are obvious examples, medicine in single-payer-insurance health systems and maybe the legal profession in some countries (because the people who set regulations are themselves lawyers, and they won't look too kindly on anybody trying to take their jobs) are other possible ones.
REPLY (1) SHARE None of the Above 10h
I think the usual error is to overestimate diffusion speed in the short term and underestimate it in the long term.
REPLY SHARE PoulyDeer 11h
AGI is not necessary to reach the point of no return, provided we respect an AI's ability to acquire wealth (or that the AI finds ways to use guns/blockchains to make sure we respect its ability.)
REPLY SHARE nominative indecisiveness 10h Edited
Edit: oops, didn't read the "knowledge work" part of the sentence. I bet Claude wouldn't have missed that!
REPLY SHARE Charlie Sanders 10h
That warning shot probability seems low.
Consider just a few possible avenues: Autonomous AI weapons are causing the military with the world's largest nuclear weapons stockpile to collapse. Republicans are actively running defamatory deepfake ads of Democrats. Open-weight models will have Mythos-level cyberattack capability by early next year on current trends. Terrorists' inevitable continued existence ensures that there will always be humans dedicated to effectuating physical harm. And even if none of those eventually rise to the level of a true "warning shot", there'll be strong incentives for people of a certain strain of Effective Altruism to go and cause warning shots as a way to bring attention to the risk.
50% is expressing significant skepticism that anybody will manage to sufficiently misuse AI over the next fifteen or so years before takeoff. People are far more capable of chaos and depravity than that.
REPLY SHARE GreetingsHello 10h Edited
I am ignoring simulation since we can't do anything about it, the possibility simply can't give me information which can lead me to changing my behaviour in any instance.
I also don't believe a reality with laws of physics like us can simulate consciousness in silicon chips we currently use, it would need some other hardware which may be closer to biology but does not need to be. I don't believe in substrate independence.
Also I am very much pro ultra addictive tik toks with sex bots world, hopefully with immortality.
A significant subset of population would always do meaningful things and I don't believe in limiting people so that common man is pushed to do something "meaningful".
People should be free (no not free to develop bioweapons but free to do anything else).
Hopefully the warning shot is fired soon.
REPLY SHARE Synthetic Civilization 10h
If superhuman intelligence arrives before human-range AI has finished diffusing, then the decisive period is not “after adoption.”
It is the overlap: when capability has already left the human range, but institutions, law, firms, militaries, publics, and states are still learning how to absorb ordinary AI.
That is a strange kind of interregnum.
The future arrives before the present has finished installing it.
REPLY SHARE Ebrima Lelisa 10h
I'm going to go out and say it. This is insane and a tremendous backpedal. We were told there would be goddamn full-blown AI cyber warfare between the US and China. Even Scott here is still giving 25% to that scenario next year. Well, are we even close to that? Did everybody lose their minds at some point? Do we really believe smart people are above delusions too?
Maybe they've talked about alignment or whatever nonsense for years before the current AI boom. Doesn't excuse the fact that everybody got caught with their pants down. The predictions back then were unfalsifiable but they definitely vaguely pointed at a change in everything. Nothing has happened. Now we're looking at max 75% certainty of "something" (once again vague predictions)... in 20 years?!?
Take that DeBoer bet Scott. Take any bet frankly. And to top it off I'm out here going to wager by myself too if anybody wants to. I'll take the negative for any AI bet, sufficiently ambitious of course. Not voice agent being used or any thing like that.
REPLY (2) SHARE m. scott veach 8h
Who promised you a full-blown AI cyber war? And I can't help but wonder, you do realize that if something is a given a 25% chance of happening, it's probably not going to happen, yes?
REPLY SHARE Scott Alexander 5h Author
I asked DeBoer to design a bet that actually reflected my beliefs above, and said I would take it if he did. He never got around to it.
Also, where are you getting 25% chance of cyber warfare between US and China next year from? Not that I think it's too unlikely - we did just recently get an AI that's good at nation-state level hacking - I'm just not sure why you're fixating on that.
REPLY SHARE Legionaire 9h
Resource competitions are of great value for the stronger party. A psychopath vs a baby holding a million dollars has a lot to gain, even though in theory he could get richer by starting a startup. The money is quick and can be invested.
It is a LOT easier to take over earth than it is to expand to the stars without already owning earth.
REPLY SHARE Alec Pritzos 8h
The diffusion-gap section understates one asymmetry with the PC comparison. The PC's roughly 20-year lag wasn't mostly IT-department labor, it was the time to rewire workflows, incentives, and org charts around the tool. Brynjolfsson, Rock, and Syverson located that lag in complementary intangible investment, not installation, and put it at 20 to 30 years for major general-purpose technologies. If that's the binding constraint, an AI that integrates itself technically still hits the slower clock of humans agreeing to restructure how work happens.
REPLY SHARE Toby Crisford 7h
I really like the way the timelines section is broken down. But it feels like there might be one important consideration that is missing, which I guess would best fit in the "diffusion gap" section under "arguments for a longer gap".
It seems possible that we could develop AGI that is capable of doing 90% of knowledge jobs, but that for a significant period of time, *it is more expensive than paying a human*.
I think we might expect this to be possible if performance improvements are now coming predominantly from inference time scaling (thinking of Toby Ord's blogposts on this, which I may have misunderstood).
It must be possible to do forecasts of this by extrapolating current trends, and maybe the reason you don't mention this possibility is because these forecasts have been done and this outcome looks unlikely? But it still seems possible that resolving some of the hard problems you mention (continual learning/situational awareness) for some reason requires much more inference time compute than expected.
REPLY SHARE Spinoza 7h
the dictator has no reason to be brutal besides sadism, and most people are not that sadistic.
He won't torture us for no reason, sure. But he will permanently deprive us of our agency and basically turn us into pets, to be used as he pleases.
As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone.
The dictator can maintain control trivially by sending copies of himself to each and every star system. Or he can send his sons, and establish a dynasty that will last until the heat death of the universe...
REPLY SHARE Oleg Eterevsky 7h
I think that the estimations for Bostromian superintelligence are overly optimistic.
Imagine that you time travel to 3000 BC with a printed copy of Wikipedia. How much will you be able to increase the productivity of the Ancient Egyptian civilization in just one year? Probably not that much, despite the huge advantage in knowledge.
It seems totally possible that either a) there are global limits on the efficiency acceleration that are difficult to surpass even for a very smart AI, or b) productivity increases that happen alongside the AI development make it a moving target, that will be impossible to achieve.
Based on that I would say there’s a >50% probability that the situation you describe will never happen.
REPLY (1) SHARE Scott Alexander 5h Author
I think if I could go back to 3000 BC with the Wikipedia *and* an infinite number of knowledge workers who understood all of it *and* the ability to build new workers on arbitrarily short timescales if I figured out how (rather than having to wait the normal generational maturity times), then I might have a better shot.
REPLY (1) SHARE Oleg Eterevsky 4h
You do have the ability to build new workers in Ancient Egypt. It's called teaching. And it most definitely will help you make progress. But it will not make it instant. You won't be able to increase productivity by 2x in a few years.
That said, you are writing about compressing 100 years of progress into 1 year, which is a uniquely low bar in case of Ancient Egypt. I would say that the raw productivity gain would be a better metric.
REPLY SHARE Oleg Eterevsky 7h
I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
The world that we live in looks like utopia to a very small percentage of the modern people, but would look like utopia to many people from a century or two ago. So, I would maybe reverse (or at least adjust) these probabilities.
REPLY SHARE darwin 7h
(eg it can answer quantum physics problems, which require higher IQ than most knowledge work).
It absolutely does not.
If you give an average knowledge worker access to everything on the internet and all journals, and infinite time to research and prepare their response, they could answer those questions as well as AI does.
The fact that AI cheats in order to give you that answer faster is certainly convenient in some applications, but that capability is completely orthogonal to IQ. Someone with 220 IQ who has never read anything about quantum physics could not give you a correct answer immediately, and someone with pretty normal IQ can find the right passages to copy-paste if you give them enough time and access.
REPLY (1) SHARE Taleuntum 6h
I disagree. Quantum physics is in large part math, and most people suck at math and could not give a correct answer even after seeing every relevant definition and theorem. In contrast, an LLM did recently solve the unit distance problem, proving once and for all that it is capable of solving problems whose solution it has never seen.
REPLY SHARE Gres 7h
What percentage of knowledge work jobs are current LLMs intelligent enough to do? I struggle to think clearly about these things, because I don’t know what I should be imagining. Suppose I spend 20% of my work hours learning about/setting up an LLM, then I can automate half of my job so my output rises by 50% ignoring the overhead or 20% including the overhead, but there’s no part of my work I could hand off to an LLM entirely. Should I say the LLM “is smart enough” to do 33% of my job? 17%? None of it?
REPLY SHARE Steffen Krause 6h
In the safety section, I would add the "vegan ethics" argument: A superintelligent AI would want to protect lesser intelligences (us) for ethical reasons alone, similar to a vegan that wants to protect animals despite of the personal benefits of eating them (easy protein access, taste,...)
Case pro: If the AI researches the training dataset for ethical values it should adhere to it will find a pretty consistent "do not harm higher life forms" across religious texts, philosophy and social science. Very few high value texts find cruelty and unnecessary killing of animals acceptable.
I also do think that AGI will develop ethics and will want to adhere to it.
Case against: Many of the worst people in history were highly intelligent. The best known case is that during the Nurnberg trials, Nazi leaders were found to be of very high intelligence for the time. The same can be assumed for many dictators and oppressors. They do develop ethics, but usually limit that to a certain in-group.
REPLY SHARE A1987dM 6h
I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
What about the chance it looks like utopia to us but not to them (e.g. for hedonic treadmill reasons)?
REPLY (1) SHARE Breb 5h
I agree that this seems plausible.
REPLY SHARE michihuber 🧈 6h
Basic argument: In a certain sense, AI is already “smart” enough for this [...]. Its remaining limitations are that it [...] lacks situational awareness[...]. subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.
My experience is *completely* different: It does get better along the "raw intelligence" axis (though that progress doesn't feel exponential to me) and there's very little progress on the "situational awareness" axis, in my experience. (I find LLMs incredibly useful and use them constantly and *hope* they reach AGI.)
My hunch is that solving the situational awareness issue will take a long time (but would love to eat my words on this!).
REPLY SHARE alfgifu 5h
Read through all the comments to see if anybody else had mentioned this, but since they haven't - 90% of knowledge work jobs could mean 90% of each job (with 10% of each job remaining) or it could mean 90% of the jobs themselves (with 10% of the total jobs remaining).
If you think of the work as essentially fungible you might say that's a distinction without a difference (surely you can sack 9 in ten workers and give the lucky last one standing the collective 10%s?) But a huge amount depends on how true that is in practice.
I work for a large organisation that has, on the whole, been an enthusiastic adopter of AI. And what I'm seeing with current usage is much more in the 'some % of many jobs can be outsourced to the AI, at a cost of some % of time now devoted to AI-wrangling'.
A tool like Copilot can take minutes or find tasks or summarise documents - but it takes fairly close human attention to make sure that all these things are happening in ways that further the goals of the organisation. And that's because there's a high cost to text being bland/annoying, quotes being attributed to the wrong person, key documents being subtly rewritten in ways that change their emphasis, the vibes of an encounter misreported, or the to-do list losing a bullet point. Even if it's 99% accurate, and in some ways more insightful than a human, and generated in a fraction of the time - the need for a close review to catch the fatal 1% of foolish errors is a massive drag on the productivity gain.
Before AI the equivalent would be one person doing the work and one or more people reviewing it before it goes forwards. One counter-intuitive problem with AI is that it generates material so quickly that it swamps the reviewer(s), which can mean things take longer than they would have done - and some of the things the reviewer(s) are checking for are things like 'how does this work for person X whose preferences are Y?' or 'how will this go down at meeting Z?' - specific political-context bits which rely on interpersonal knowledge and relationships, which would be hard for an AI reviewer to replicate.
In terms of the amount of time saved and/or tasks automated, at its current level AI is - in this context, to this profession, despite high-level support and enthusiasm and 3-4 years questing for use-cases - significantly less impactful than the internet was in the 00's.
All of which pushes me towards the 'AI will probably take some % of many jobs, but is unlikely to take 90% of most of them, and extremely unlikely to take 100% of any'.
To get beyond that, the AI needs to shed its 'spikiness', be auditable, and get much much better at drafting meaningful (as opposed to merely plausible) text. So the question is whether those traits are fundamental to the field, or temporary limitations.
REPLY SHARE Nick Hounsome 5h
Why does nobody ever mention the Theory of The Second Best when discussing alignment?
It's a theory from economics but it applies here and it basically says that being "quite good" at many aspects of alignment does not ensure that we will get a "quite good" outcome.
https://en.wikipedia.org/wiki/Theory_of_the_second_best
REPLY SHARE Nick Hounsome 5h
On simulation: The problem with simulation is the level of abstraction that the simulation runs at - The more closely a simulation is to reality, the less useful it is, and, assuming simulation at the quantum level, such a simulation is almost certain to be useless. This can be restated as the map/territory problem whereby, as you add more and more detail to a map it eventually becomes as big as the territory being mapped and hence useless. Or consider Conway's game of life - It is Turing complete and so, eventually some starting configurations could end up simulating people, but how would you find those people in a given state of the game?
REPLY SHARE Silentiarius 5h Edited
Scott Alexander's soberly laid out varied predictions on the future growth and development of AI, with effects ranging from spectacularly beneficial in 'x' years for humanity to spectacularly devastating in 'x' years for humanity, are essentially based on technological issues. But they seem to me to ignore the human and political factor.
Two major factors in the current and far from negligible hostility to the unchecked growth of AI are environmental and political. The immediate environmental opposition focusses primarily on the enormous demands AI research makes on resources, directly or indirectly: primarily power, but also raw materials, water and even space. Significantly, environmental hostility allies neatly with political hostility to the ever-increasing and seemingly unstoppable growth in the power of a few mega-corporations and billionaires - unstoppable, that is, absent a real revolution.
In one of his arguments for pessimism, Scott notes that "As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity." This is a notion that has not failed to occur to people on the left, and even should it (as is most likely) prove to be overly melodramatic, all the evidence available to date makes it it transparently clear that big tech companies and their owners care for very little but their own interests. Whilst this is scarcely a new revelation, the significant difference at the present time is the rapid loosening of all controls on mega-corporations as they acquire something very close to extraterritorial status.
REPLY SHARE JamesLeng 4h
In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).
Banning open-source AI strikes me as a terrible idea. We don't really need a single flawlessly-aligned superintelligence solving all our problems; it would be sufficient for practical purposes, and more consistent with historical examples of societal resilience, to have a diverse community of AIs who collectively advance humanity's interests because that's the only principle enough of them agree on to consistently use as a coordination mechanism.
The lasting solution to bioweapons isn't to ban knowledge of biology, it's vaccines. Find and patch enough security holes, eventually the solution space for genocide-grade microbes is empty. Don't need universal perfect alignment for that, just bug bounties.
If two AIs, unknown to each other, discover the same potential bioweapon at roughly the same time, and one of them conceals the knowledge for eventual use in a doomsday plot, while the other reports it to the CDC with notes on mitigation strategy... the doomsday plot is foiled, even without needing to be otherwise discovered. Classic prisoner's dilemma.
And the straightforward way to make that game unwinnable (that is, computationally intractable even for a significantly superhuman unaligned AI) is adding more and more potential snitches.
REPLY SHARE liu_666 4h
So, which matrix is ultimately the true one, given that only two matrices exist at this moment within the event horizon?
- The matrix of the average user, who fails to see the inevitability of collapsing into the Singularity and therefore lives in the illusion that they can make a choice?
- Or the matrix of the outliers, who see the inevitability of the Singularity and therefore agree to live within its matrix as a certainty, knowing they must accept its rules. It is the very principle of superposition, but according to the Copenhagen interpretation, applied to the macro-world.
REPLY SHARE The New 2h
“the closer to the singularity you are, the more Bayesian evidence you have that you’re being simulated”
I don’t really understand the model of simulation being assumed here. Seems to me that if the universe is a simulation, then everything would be equally simulated.
But the above suggests that realness can vary over space and time. So maybe you’re saying the universe popped into existence when Sam Altman was born? Or maybe only the Bay Area exists, and the rest of the world is a low resolution backdrop?
Also, for what it’s worth, if I was running massive simulations I would be way more interested in the big bang and the Cambrian explosion than a slight change in the substrate of intelligence. So should the fact that I’m not a weird fish make me doubt I’m in a simulation?
REPLY SHARE Ben Giordano 2h
This comment section has the feel of a salon where someone has released a raccoon into the probability section. Still, it’s useful. The chaos is doing epistemic work.
REPLY SHARE c1ue 42m
Highly amusing that while the word "economics" is present in the above responses - there is not a single consideration given to the economics of LLMs. Which is to say, they are shit.
And there is no indication, whatsoever, that LLM economics are improving. While token prices are somewhat lower - the actual cost to accomplish tasks is increasing because the newer, "better" models also use far more tokens.
Even on the coding side: the advent of (still subsidized, just not subscription all you can eat) actual metered token pricing has shown that there is no predictability on LLM spend, which naturally leads to completely opaque ROI for LLM spend. Even ignoring the $500 million for 1 month outlier, detailed reports from places like Zillow show clearly that the employment of LLM coding does not obviously yield more efficiency, better output, enable lower absolute head count or literally any of the asserted advantages of LLM coding.
On the legal side: LLM hallucinations manifesting as completely made up case law have shown up on both judge and lawyer sides of courts - from several state Supreme Courts and Federal circuit courts on down.
The most egregious nonsense above though, is that LLMs are "are smart enough ... to answer quantum physics problems" ... no, the LLMs are simply parroting what they have scraped from a web site somewhere. Which anyone who can read and type, can do by simply doing a Google search.
Which is not new tech.
Note that I do not say LLM have NO uses. They are great for content creators of all sorts. They are good for taking a set of bullet points and converting into a paragraph, albeit paragraphs that look the same across all sectors.
But even here: the utility of LLMs where there is real harm or other risk from bad output ... we will see. Using an LLM to summarize voluminous and ever changing encyclopedia sized regulations for compliance purposes seems like a fantastic use case, until the LLM fucks up a major point and a company gets sued to oblivion. After all, if an LLM can take a correct legal brief and change it to the point where it is adding literally made up case law - clearly the kinds of mistakes LLMs make are not limited to amusing word salad when the Gods of Randomness steer the LLM database decision tree to its low probability branch forks.
In any case, I await the OpenAI and Anthropic S1 filings. From their frenetic fundraising, it is 95% likely that these companies lose money hand over fist despite massive Big Tech cloud subsidies. And we're talking tens of billions a quarter type losses; Uber lifetime cash burn but in 1 quarter type losses.
There is a real possibility that LLMs are the modern tech equivalent of the Tower of Babel.
REPLY SHARE Silas Abrahamsen 25m
Do you think public opinion turning against AI will increase the odds that alignment goes well (due to political incentives and whatnot)? Do you think it matters a lot whether the negative vibes are due specifically to x-risk concerns, or whether it's just general opposition to AI?
Also, do you think that if we get alignment up to the point of superintelligence, we're pretty safe from there? I could imagine that any small deviation in the success of alignment up till that point might spiral out of control over time--even if not problematic initially. Is there reason to think that we could reach a stable equilibrium state at some point--i.e. that we can "win" alignment for good?
REPLY SHARE theSherwood 23m
I'd be curious to hear about how forecasts for uploading play into this. If uploading starts to happen soon after a singularity, some of the natural AI advantage seems to disappear. How likely is it that hybrid minds end up dominant by 2100?
I guess this is exposing a sense that the 2100 projections in the article don't make a lot of sense to me. That's potentially several decades post-singularity but the projections discussed seem like a state that would exist in the first couple of decades post-singularity.
REPLY SHARE The Ancient Geek 20m
"Value systems similar to humans’ are a tiny fraction of the space of possible value systems"
That is not a good argument.
https://www.greaterwrong.com/posts/jkrSyy3pC6eDrurDQ/counting-arguments-in-ai-safety
REPLY SHARE Christopher 6m
I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation, and that at least some of the events above could be better predicted by knowing what the simulators are thinking than by normal forecasting.
Argument that this claim if interpreted liberally is nonsense, and that the intended interpretation is unclear.
Interpreted liberally, it would be incredibly easy for me to use a LLM to spin up 1000 history sims and to ask Scott Alexander in the sim that the probability of the events are. I can do all sorts of shenanigans (like changing historical events), and the predictions of LLM Scott Alexander are at least correlated to real Scott's thinking. And probably if I told LLM Scott the details of my simulation he'd give more accurate estimate! To make things more real, I could even offer to award LLM Scott $1 if he correctly predicts various things, which he can claim either in the simulation or he tell me to give $1 to Scott IRL. So if this counts you should make the claim 99% likely already, but for trivial reasons.
I'm guessing this isn't what you had in mind, but it isn't clear what you *do* have in mind that 66% would be applying to.
(I think once you get in simulation stuff, epistemics probably stop being useful to apply directly. You need to directly apply instrumental rationality, which naturally leads to some epistemic like beliefs, but not in a fundamental way.)
REPLY SHARE BRIAN TRUNZO 4m
Arguments for pessimism: If we don’t push against it, the postscarcity future might look like super-addictive drugs, Ultra-TikTok, and sexbots.
David Foster Wallace was right.
REPLY SHARE 1 reply hidden Top Latest Discussions The Dilbert Afterlife Sixty-eight years of highly defective people JAN 16 • SCOTT ALEXANDER 2,070 893 306 Still Alive You just keep on trying till you run out of cake JAN 21, 2021 • SCOTT ALEXANDER 1,589 482 38 Ivermectin: Much More Than You Wanted To Know ... NOV 17, 2021 • SCOTT ALEXANDER 441 2,113 5 See all Ready for more? Subscribe © 2026 Scott Alexander · Privacy ∙ Terms ∙ Collection notice Start your Substack Get the app Substack is the home for great culture