One of the safety leaders of OpenAI has resigned, warning that the culture of Artificial Intelligence firms was “broken” and the tech giants are not “being nearly careful enough” when it comes to developing the technology.
One of the safety leaders of OpenAI has resigned, warning that the culture of Artificial Intelligence firms was “broken” and the tech giants are not “being nearly careful enough” when it comes to developing the technology. David Robinson, who led the writing of safety reports that accompanied the ChatGPT developer’s product releases, wrote an essay on why he left the company, titled: “I quit OpenAI because its culture is broken”.
In his statement, Robinson wrote that a cultural overhaul was needed at cutting-edge AI firms and that incidents such as a “swarm” of OpenAI agents – AI programmes operating autonomously without human oversight – attacking the AI startup Hugging Face were “typical of the industry, given the speed and flexibility with which people operate”. His essay was published in The Atlantic magazine.
“I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture,” Robinson wrote in the letter published on Saturday. “As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed,” he said, while talking about OpenAI’s pace of development.
However, in recent weeks, OpenAI has shown signs of caution following the Hugging Face incident and the revelation that it has notified more than 100 organisations about rogue agent activity. This week, the firm announced that it was scrapping the release of the next-generation AI model after several researchers raised safety concerns during internal testing. In addition, OpenAI has paused training its most advanced models.
Alarms raised by experts
On Saturday, Geoffrey Irvingwho worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings about AI. In his piece in Time magazine, Irving wrote: “Recent warnings about the potential destructive power of AI are understating the severity of the situation.”
“I believe there’s about a 50 per cent chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome,” he said. It is pertinent to note that Robinson’s essay also follows the resignation of Jacob Coxon, a researcher at OpenAI rival Anthropic, who quit the Claude chatbot developer last month.
After resigning from the firm, he warned that AI “could kill us all by the end of the decade”. After Coxon’s warning, Anthropic issued a separate alert that there was more than a 10 per cent chance that AI would wipe out humanity within the next decade. However, critics of such warnings maintained that most of these proclamations are unscientific because they cannot be verified or falsified.
In his essay, Robinson wrote that Silicon Valley lacked an awareness of “how to handle dangerous technology” and “what it means to care for people”. He warned that OpenAI had “unimpeded optimism” about solving problems as they arose; he wrote that this internal culture meant safety failures would only grow as systems become more capable.
“Imagine ‘rogue’ agents that work like teams of hackers (for example, holding hospital computer systems for ransom) but never need to sleep,” Robinson wrote. He went on to call for two safety changes: AI firms rely on safety expertise in other fields such as nuclear and aviation and develop “new science” that ensures powerful systems in the future are capable of being reined in when they are operating autonomously.
“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote. Meanwhile, an OpenAI spokesperson said the company was continuing to “strengthen our safety and security practices to address the risks we see today”, while working on dealing with the risks that might be created by future AI breakthroughs.
“We’re making sure our models don’t become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down,” the spokesperson averred.
Source: www.firstpost.com
