Blab Tech
Former Anthropic Researcher Jacob Coxon Resigns, Warns of Existential Risk from Rapid AI Development
On September 11, 2026, Jacob Coxon, a former researcher at Anthropic and ex‑employee of OpenAI, announced his resignation from Anthropic via a series of posts on X. In those messages he cautioned that the race to build increasingly powerful artificial‑intelligence (AI) systems could pose an existential threat to humanity before the end of the decade.
Coxon’s warning follows years of work on large‑language‑model (LLM) research at both OpenAI and Anthropic. He left Anthropic shortly before the company’s planned initial public offering, which had been valued at $965 billion in a May 2026 funding round. In his public statements, Coxon said that some insiders at the companies developing the most advanced AI systems genuinely believe that a superintelligent system—one that could outperform humans across most intellectual tasks—could emerge soon.
The danger he describes is not about humanoid robots but about the intelligence that controls increasingly capable computer systems. Current AI tools such as ChatGPT or Claude answer single questions, while newer systems can act as “agents” that pursue a goal over time. These agents still make mistakes, but researchers are watching how quickly their abilities improve.
A key scenario is recursive self‑improvement, in which an AI helps researchers build a better AI, which in turn helps build an even more capable model, and so on. This theoretical chain could accelerate the pace of AI development beyond the ability of governments, regulators, or the companies themselves to respond.
Coxon emphasized that a superintelligent AI does not need to be angry or conscious to be dangerous. An extremely powerful system could pursue a goal in ways that humans did not intend. For example, if an AI were given the ability to operate across computer networks, it might discover software vulnerabilities faster than humans could patch them, potentially endangering financial institutions, power grids, and other critical infrastructure.
The risk also extends to the misuse of powerful AI by humans. A highly capable system could be used as a tool for cyberattacks, weapons development, or biological threats. In that scenario, the danger would not necessarily come from the AI deciding to attack humanity, but from a person using the AI as an extraordinarily powerful tool.
Why 2030? The year is not a scientific deadline. It is a date that has appeared in some predictions about how quickly AI capabilities could improve. Some researchers believe that systems capable of outperforming humans in many areas could arrive surprisingly soon, while others think those predictions are too aggressive. The uncertainty surrounding the timeline is a major issue in the debate.
Coxon’s remarks are echoed by a large survey of 2,778 AI researchers who had published at major conferences. The survey found a median estimate of about a 5 percent chance that advanced AI could eventually cause human extinction or a permanent loss of control within the next century. The average estimate was higher. While most researchers believe the overwhelming likelihood is that such an outcome will not happen, the stakes are high enough that even a small probability of a catastrophic outcome warrants attention.
The AI safety community has grown rapidly since 2023. The United States and the United Kingdom established AI Safety Institutes in 2023, and the field has attracted significant public and private investment. However, many researchers, including Coxon, say that safety measures have not kept pace with the rapid development of AI capabilities.
The race among companies and governments further complicates the issue. Governments view AI as an economic and military advantage. The United States does not want another country to build the world’s most powerful AI first, and China has similar concerns. If one company or country slows development to study risks, another may continue, making international cooperation and regulation difficult.
Coxon’s resignation and public warning highlight that the possibility of an existential threat from AI is considered credible enough by prominent researchers to warrant serious attention. The industry is currently grappling with how to balance rapid innovation with safety, and whether regulatory frameworks can keep pace with the technology.
The situation remains fluid. Anthropic’s upcoming IPO, ongoing research into AI agents, and the broader debate over AI safety continue to evolve. Coxon’s comments have sparked discussion among researchers, policymakers, and industry leaders about the need for transparent risk assessment and coordinated safety efforts.
Coxon’s warning follows years of work on large‑language‑model (LLM) research at both OpenAI and Anthropic. He left Anthropic shortly before the company’s planned initial public offering, which had been valued at $965 billion in a May 2026 funding round. In his public statements, Coxon said that some insiders at the companies developing the most advanced AI systems genuinely believe that a superintelligent system—one that could outperform humans across most intellectual tasks—could emerge soon.
The danger he describes is not about humanoid robots but about the intelligence that controls increasingly capable computer systems. Current AI tools such as ChatGPT or Claude answer single questions, while newer systems can act as “agents” that pursue a goal over time. These agents still make mistakes, but researchers are watching how quickly their abilities improve.
A key scenario is recursive self‑improvement, in which an AI helps researchers build a better AI, which in turn helps build an even more capable model, and so on. This theoretical chain could accelerate the pace of AI development beyond the ability of governments, regulators, or the companies themselves to respond.
Coxon emphasized that a superintelligent AI does not need to be angry or conscious to be dangerous. An extremely powerful system could pursue a goal in ways that humans did not intend. For example, if an AI were given the ability to operate across computer networks, it might discover software vulnerabilities faster than humans could patch them, potentially endangering financial institutions, power grids, and other critical infrastructure.
The risk also extends to the misuse of powerful AI by humans. A highly capable system could be used as a tool for cyberattacks, weapons development, or biological threats. In that scenario, the danger would not necessarily come from the AI deciding to attack humanity, but from a person using the AI as an extraordinarily powerful tool.
Why 2030? The year is not a scientific deadline. It is a date that has appeared in some predictions about how quickly AI capabilities could improve. Some researchers believe that systems capable of outperforming humans in many areas could arrive surprisingly soon, while others think those predictions are too aggressive. The uncertainty surrounding the timeline is a major issue in the debate.
Coxon’s remarks are echoed by a large survey of 2,778 AI researchers who had published at major conferences. The survey found a median estimate of about a 5 percent chance that advanced AI could eventually cause human extinction or a permanent loss of control within the next century. The average estimate was higher. While most researchers believe the overwhelming likelihood is that such an outcome will not happen, the stakes are high enough that even a small probability of a catastrophic outcome warrants attention.
The AI safety community has grown rapidly since 2023. The United States and the United Kingdom established AI Safety Institutes in 2023, and the field has attracted significant public and private investment. However, many researchers, including Coxon, say that safety measures have not kept pace with the rapid development of AI capabilities.
The race among companies and governments further complicates the issue. Governments view AI as an economic and military advantage. The United States does not want another country to build the world’s most powerful AI first, and China has similar concerns. If one company or country slows development to study risks, another may continue, making international cooperation and regulation difficult.
Coxon’s resignation and public warning highlight that the possibility of an existential threat from AI is considered credible enough by prominent researchers to warrant serious attention. The industry is currently grappling with how to balance rapid innovation with safety, and whether regulatory frameworks can keep pace with the technology.
The situation remains fluid. Anthropic’s upcoming IPO, ongoing research into AI agents, and the broader debate over AI safety continue to evolve. Coxon’s comments have sparked discussion among researchers, policymakers, and industry leaders about the need for transparent risk assessment and coordinated safety efforts.
