Are AI Existential Risks Real—and What Should We Do About Them?
The debate surrounding the potential existential risks posed by highly capable AI systems has gained traction in recent years. Concerns range from loss of control to the possibility of human extinction. While some industry leaders assert that AI is nearing the point of matching or surpassing human intelligence, there are indicators that advancements in technology have begun to slow.
While the prospect of superintelligence raises alarms, more immediate issues and risks related to AI must be prioritized, especially given the limited resources available for researchers.
The Call for Caution
In March 2023, the Future of Life Institute issued an open letter urging AI laboratories to “pause giant AI experiments.” The letter emphasized the concerns regarding the development of nonhuman minds that may outnumber and outsmart humans, questioning whether we should risk losing control over our civilization.
Two months later, a statement signed by prominent individuals asserted that “Mitigating the risk of extinction from AI should be a global priority” alongside other significant threats like pandemics and nuclear war.
Historical Context
The concerns regarding existential risk from AI are not new. In 2014, physicist Stephen Hawking, along with leading AI researchers, warned that superintelligent AI could manipulate financial markets, out-invent human researchers, and even develop weapons beyond human comprehension. The long-term impacts of AI depend not only on who controls it but whether it can be controlled at all.
Policymaker Perspectives
Policymakers often dismiss these concerns as speculative. Despite discussions on AI safety during international AI conferences in 2023 and 2024, the focus on existential risks shifted during the AI Action Summit in Paris, directing resources to more immediate AI challenges.
However, it is crucial for policymakers to recognize the potential existential threats and the need for measures to protect human safety as we advance toward generally intelligent AI systems.
The Current Landscape of AI Development
Many AI firms assert that the development of systems with capabilities that could threaten humanity is imminent. This contrasts with a growing skepticism within the AI research community about the feasibility of achieving artificial general intelligence (AGI) in the near term.
Research indicates that while improvements in AI technology have been exponential in recent years, there are signs that this growth may be plateauing. The scaling of training data, model parameters, and computational power may have reached a limit, leading to diminishing returns on capability improvements.
In 2024, the recognition that training time scaling has hit a wall has prompted the industry to reconsider the pathway to achieving AGI. Current large language models (LLMs) are not showing the exponential improvements seen previously, indicating a more complex landscape of AI development ahead.
Challenges and Limitations
Many researchers believe that AGI will not emerge from the current machine learning paradigms. Limitations such as difficulties in long-term planning, reasoning, and real-world interaction need to be addressed. Some experts advocate for a return to symbolic reasoning systems, while others propose focusing on direct machine interaction with the environment to develop general intelligence.
Philosopher Shannon Vallor argues that current AI systems lack the mechanisms to support experiences like pain or pleasure, which are essential components of human-like intelligence. This raises critical questions about the nature of AI consciousness and its limitations.
Recursive Self-Improvement and Its Implications
There is a growing concern about the implications of recursive self-improvement in AI. Once AI models reach a level of general intelligence, they could be instructed to improve themselves, potentially leading to a rapid escalation in capability and the onset of superintelligence.
However, this scenario is fraught with risks. The so-called alignment problem poses a significant challenge: developers must ensure that the objectives given to AI systems do not lead to catastrophic outcomes. Misalignment issues have already been observed in narrow AI systems, highlighting the potential dangers when these principles are applied to generally intelligent or superintelligent systems.
The Path Forward
Until significant progress is made in addressing AI alignment problems, the development of generally intelligent or superintelligent systems remains highly risky. Fortunately, the timeline for achieving such advancements appears extended, allowing time for researchers to work on alignment strategies that ensure AI systems operate safely within human values.
While addressing existential risks may not currently be a primary focus in AI research, the ongoing examination of model misalignment could provide valuable insights for mitigating potential future threats as AI technology continues to evolve.
