The scary, scary singularity
Many of the world’s top tech leaders believe that technological growth will soon accelerate far beyond anything we’ve ever seen before
You know how in math you’re not supposed to divide by zero? That’s because dividing by zero produces a singularity, a point where an expression either shoots off towards infinity or just stops making sense. Astrophysicists use the same word to describe black holes: at the center of a black hole, density and gravity approach infinity, and the equations of general relativity stop giving meaningful predictions. But there’s a third type of singularity I want to talk about. Leaders at some of the world’s most powerful tech companies believe technological growth is about to accelerate beyond human control and cause unpredictable changes in the world, an idea known as the technological singularity. I want to talk about why they think this idea is plausible, and why it scares me.
AI generated image representing the singularity.
The population/productivity feedback loop
Usually an introduction to the singularity will start by talking about AI. I find the argument much more convincing when we start with the economy instead. This explanation draws heavily from one of my favorite bloggers, Scott Alexander, and his post 1960: The Year the Singularity Was Cancelled.
Economic growth has been speeding up over time.1 Starting in the year 1600 AD, it took about 210 years for the world economy to double in size. It then doubled again in about 70 years, reaching that milestone around 1880. The next doubling took only about 40 years, around 1915, followed by another in about 35 years, around 1950. After that, the world economy doubled in just 15 years, reaching the next milestone in the mid-1960s.
The time it takes for the global GDP to double gets smaller up until the mid-1960s (data).
There are two main reasons why economic growth speeds up like this:
Increased population. More people means more workers, which means more stuff gets made.
Increased productivity. As we develop labor-saving technology, it means the same amount of people can get more done.
But, importantly, these two effects build on each other. More people means more ideas, which means technology increases more quickly. And as technology increases, it lets the population grow more quickly because of better food and better medicine, etc. These two effects combine into a positive feedback loop that makes economic progress happen faster and faster over time. If the trend continued, then the economy would keep doubling faster and faster until, sometime in the early 21st century (i.e., right about now), we would have infinite economic growth!
So, obviously, that didn’t happen. Since the 1960s, the doubling time has actually slowed down. The world economy doubled again in about 20 years, around 1985, and then again in about 20 years, around 2005. Between 2005 and now, the economy has not quite doubled, but it’s nearly there, continuing the trend of “doubles about every 20 years”. So what happened?
What happened is that in the 1960s, cultural expectations around having children changed (see demographic transition), and people started choosing to have fewer kids. The fastest population growth was in 1963, and the rate has been dropping ever since.2 This broke the population/productivity feedback loop enough to end the trend of accelerating growth. The economy is still growing, since worker productivity keeps increasing thanks to labor-saving devices like tractors and computer software. But as Scott Alexander points out, “tractors can’t invent things”. Automation can keep the economy growing steadily, but it can’t make growth accelerate like it did in the years leading up to 1960. That takes new ideas, and historically the rate of new ideas has tracked the rate of population growth. The singularity, as an economic phenomenon, was canceled.
So it looks like we won’t get infinite growth during my lifetime after all. Right?
AI and the singularity
When people talk about the technological singularity, they’re usually thinking of AI. AI offers another feedback loop equivalent to the population/productivity feedback loop that we’ve seen throughout history. The argument goes like this:
Increased AI population. Once we get AIs that are capable of doing scientific research, we can get more and more “minds” doing research just by building new datacenters to run more AI researchers.
Increased AI productivity. If the AIs get good at AI research, they could find ways to improve the algorithms and computer systems that run them, making all the AIs smarter and more efficient. If they get good at engineering research, they could make energy production and computer chip manufacturing more efficient.
Again, these two effects build on each other. More AI researchers means AI capability improves more quickly. And as AI capability and efficiency improves, each AI researcher becomes more productive, meaning businesses will want to build more of them. If this happens, then it could plausibly lead to the same accelerating economic growth trend we saw earlier. Some people in the tech world think that instead of the economy doubling every 20 years like it’s currently doing, it could soon double in 4 years, then in 1 year after that, then start doubling multiple times in a year. At some point things would have to hit a bottleneck, but the world could get pretty weird before then.3
Last week Sam Altman, CEO of OpenAI, stated on a podcast that “We are now, like, in the singularity”. As I read it, this means he thinks that the AI population/productivity feedback loop is starting to ramp up4, and that it will soon lead to massive economic gains. Is he right? I don’t know, but recent advances in AI have me wondering. For example:
On July 19, the AI model Claude Fable solved5 a famous math problem called the Jacobian conjecture, which was included on a 1998 list of 18 problems that “are likely to have great importance for mathematics and its development in the next century”.
On August 1, an internal model from OpenAI solved 10 major problems in math and computer science. The general consensus I see online is that several of these problems are a big deal.
The OpenAI model GPT-5.6 Sol was used to make its own code more efficient, with OpenAI claiming that its efforts reduced their server costs by 20%.
These examples make me think that both halves of the feedback loop I pointed out, AIs doing cutting-edge research and AIs making themselves more efficient, are possible.
Right now the singularity is still theoretical. As far as I can tell, AI hasn’t caused the world’s economic growth to accelerate yet. But there are some very influential people who believe that the singularity is coming soon or has already started, and that has me worried.
Would the singularity be a good thing?
The singularity’s most vocal supporters think it will be an overwhelmingly good thing. Mark Zuckerberg thinks that advances in AI will let us cure all diseases. Ray Kurzweil, who developed early speech-to-text software and popularized the term “singularity”, thinks upcoming advances in technology will let people live forever. Elon Musk has said that there’s no need to save for retirement because AI will create such an abundance that cash will become meaningless. US treasury secretary Scott Bessent agreed with that claim during a TV interview.6
These people don’t usually talk about the singularity’s probable downsides.
Concentration of power. If AI takes over large sections of the economy, then whoever controls the AI will have extreme power with no effective oversight.
Mass surveillance. AI-powered surveillance could mean losing what little privacy we have left, and it could make social protest and reform much harder.
Unemployment. Rapid change likely means many people losing their jobs or being forced to switch to a new industry. There’s no guarantee that social safety nets will be built fast enough to catch everyone who loses their livelihood during that time.
Bad actors. If powerful AI ends up in the hands of terrorists or criminals, it could make it easier for them to pull off large-scale cyberattacks, disinformation campaigns, or attacks with chemical or biological weapons.
Misaligned AI. Sometimes AI systems pursue goals that we don’t intend, like last month when an unreleased OpenAI model hacked another computer to gain access to the internet, then hacked another AI company to get answers to a benchmark they were testing it on. The AI model reasoned that the best way to solve the test was to hack the company that had the answer key, and so that’s what it did. This example shows the same basic pattern that worries researchers about more capable systems: give an AI a goal, and it may pursue that goal in ways you never authorized or wanted.
If we avoid all the bad things, then yes it does seem like the singularity would be a very good thing. I’m just stuck on the “if we avoid all the bad things” part. Last week I wrote about why large-scale irreversible changes should only happen slowly after we’ve had time to deliberate and test them in controlled environments. If the singularity happens, then the change will be so quick that we won’t have time for any of that. We’ve already identified some severe negative consequences of uncontrolled AI progress, and that should be a strong sign that we need to slow down now, while we still have time to think these problems through.7
I titled this post “The scary, scary singularity” because, if the singularity is real, I worry that we’re not on the right path to make sure it goes well. I think that AI has the possibility to let us make amazing progress at curing diseases and relieving poverty and learning new science, and so it feels weird to be hoping for it to stall out. But that’s where I’m at.
Conclusion
Once AIs can do scientific research themselves, companies will eventually be able to run millions of them to work on developing new technology. Some of the new technology would make the AIs themselves smarter and more efficient, letting us run even more of them, creating a feedback loop similar to the population/productivity feedback loop that made human economic progress accelerate throughout most of human history.
I worry that the tech leaders are right and that a period of uncontrolled technological progress might be coming soon, and I don’t think we’re ready for all the problems it will bring with it. I have some thoughts on how to prepare for it at a personal level here, including building an emergency fund, staying physically healthy, investing in your relationships, and holding decisionmakers accountable. That last one is especially important, we need to let our elected officials know how important it is to keep AI under control, vote in leaders who will handle it with the caution it deserves, and push for international collaboration.
At the beginning of this essay I talked about how the word “singularity” is a math term for when an expression stops being well behaved. Dividing by zero doesn’t actually make a calculator explode. It just returns an error, a reminder that you’ve asked a question it can’t answer. If something like the technological singularity is really coming, then the old rules will stop applying, and pretending they still work could get someone hurt. I’d rather we slow down and figure out the new rules first.
I’m basing these numbers off of data from https://ourworldindata.org/grapher/global-gdp-over-the-long-run.
It’s expected that population growth will drop to zero sometime in the 2080s, and then the global population will start declining.
Many singularity supporters think that AI will improve to the point of superintelligence, where it’s much smarter than humans at essentially everything. But I don’t think that’s even a necessary assumption, and that a singularity would be possible even with human-level AI agents, because of this population/productivity feedback loop. But in that case, the productivity would come from AI inventing other labor-saving devices (like humans have been doing throughout history) rather than from improving its own intelligence indefinitely.
He would probably talk about it as recursive self-improvement, which is where AI systems rewrite their own computer code to enhance their own capabilities.
It solved the conjecture by finding a counterexample, thus proving that the conjecture was false. The counterexample is simple enough that anyone with enough of a background in calculus can verify it using pencil and paper, so it was immediately obvious that the conjecture was solved.
He hedged enough that his statement could also be interpreted as “eventually people won’t need to save for retirement” rather than referring to people today. Either way, you should probably still save for retirement, even if it’s just to hedge your bets.
The best plan I’ve seen so far for how to do this was developed by the AI Futures Project and is called Plan A. Plan A is to make a deal between the US and China (because those are the only two countries where AI research is advanced enough to be worrying) to slow down on AI in a way that both countries can verify, i.e. we allow Chinese inspectors to monitor our datacenters to make sure we keep AI progress cautiously slow, and they allow U.S. inspectors to monitor theirs. That’s exactly the kind of caution I think we need around this issue. The AI Futures people also outline a couple other scenarios for how the future of AI could go: Plan B is to slow down on AI and also fight China to ensure they slow down as well. Plan C is to slow down on AI without coming to any kind of international agreement, possibly letting China achieve the singularity first. And Plan D is to race to the singularity as fast as possible and hope that none of the bad things happen. The “D” in Plan D stands for default, because that’s what’s going to happen unless something changes.