When researchers say AI could kill us all, what should CIOs do?

Leaders under pressure to adopt AI may have to confront a new question: How to balance today's AI goals against concerns about AI's risks?

For the past couple years, business leaders have delivered a consistent message to CIOs: Move faster on AI.

Boards want AI strategies, employees want AI tools and technology vendors are embedding AI into just about every single one of their product offerings.

Then comes a news headline that appears to point in the opposite direction.

This week, a senior researcher at Anthropic said he believes advanced AI has a greater than 10% chance of causing human extinction within the next decade, while another former researcher publicly resigned over concerns about the race toward increasingly powerful AI systems, NBC news reported.

The news ignited fresh debate about the risks associated with AI. However, conversations with tech leaders suggest many organisations view the controversy through a different lens.

Rather than debating hypothetical doomsday scenarios, CIOs and other IT leaders are focused on a more immediate concern: how to govern and manage the AI tools already being deployed across their organisations.

The message isn't 'stop.' It's 'govern'

Among the technology leaders who spoke with MES Computing, none suggested the warnings should cause organisations to abandon AI initiatives.

Instead, they said the comments should reinforce the need for stronger governance and oversight.

"Warnings like these should not change whether we invest in AI, they should change how we invest in it," said Sarfraz Shaikh, CIO/CISO at ONEMESA.

"When the people closest to building these systems are raising concerns, leadership should treat that as meaningful input and apply the same discipline used for any high-impact investment: clear ownership, defined risk tolerance, strong governance and independent oversight."

Shaikh argued that organizations cannot simply defer risk decisions to technology vendors.

"For midsize organisations, the takeaway is we cannot outsource risk judgment to the vendor," he said. "We need to ask hard questions about data handling, model behaviour, security, accountability and where human review remains non-negotiable before AI is embedded into core business processes."

Jim Cusack, CIO at VanEerden, echoed that sentiment. He said the headlines should encourage learning rather than fear.

"I would suggest that IT leaders should interpret these warnings as something to learn more about, especially what prompted the warnings," Cusack said.

"IT leaders must keep learning about AI risks and how to reduce those risks with strong governance being part of any AI investment."

Today's risks versus tomorrow's fears

One consistent theme emerged from nearly every response: enterprise technology leaders are far more concerned about today's AI risks than speculative future scenarios.

"For most of our customers, AI safety concerns in the existential sense aren't the conversation happening in boardrooms right now," said Muhammad Yahya Patel, vCISO and cybersecurity advisor at Huntress.

"The conversation is much more immediate: how do we use AI to do more with less, how do we stop our employees from using AI tools we haven't approved yet and how can we get visibility into where and what AI tools are being used by employees."

Patel said the broader debate around humanity's future may be important ("May"? - Ed.), but it remains separate from the decisions most organisations must make today.

"Customers aren't asking whether AI is safe for humanity," he said. "They're asking whether the AI their employees installed last week is safe for the network. Those are two very different questions and only one of them is keeping security teams busy right now."

Andrew Kon, director of professional services at High Touch Technologies, also highlighted that focus on practical governance concerns.

"From what we're seeing, the pressure to adopt AI is still outweighing businesses' security concerns," he said.

"Most of our clients aren't looking to slow down productivity. Instead, they're trying to get a better handle on how AI is being used, so they can put the right cybersecurity guardrails around it."

Kon said organisations are increasingly shifting from asking whether employees are using AI to determining how to manage AI use responsibly.

Despite the highly publicised warnings, few technology leaders believe organisations will substantially slow their AI adoption efforts.

"The adoption pressure is still winning. Overwhelmingly," Patel said.

"The organisations pumping the brakes on AI entirely are in the minority and facing competitive pressure that makes the position hard to sustain."

Others described similar sentiments among customers.

"We're not seeing customers pull back from AI because of safety concerns," said Vidya Shankaran, Field CTO at Commvault.

"We're seeing them demand stronger governance, security and accountability as adoption accelerates. The pressure to adopt AI remains high because the business value is undeniable."

Shankaran compared AI's current moment to one of humanity's earlier technologies.

"AI is where fire was for early civilisation: enormously powerful, transformative and potentially dangerous if left unmanaged."

Mohammed Aboul-Magd, general manager at SandboxAQ, said safety concerns are less likely to slow adoption than to raise expectations around governance and business value.

"We don't think safety concerns are slowing the pressure to adopt AI as much as they're changing the bar for adoption," Aboul-Magd said.

"Security, governance and guardrails are increasingly table stakes. What's changing now is the focus on value."

Aboul-Magd says organisations are moving beyond questions about deployment speed and focusing on whether AI delivers measurable business outcomes.

"Enterprises are moving from 'How quickly can we deploy AI?' to 'What are we actually getting from it?'" he said.

Meanwhile, companies are attempting to balance long-term concerns with present-day business realities.

"I don't think warnings like this are making customers back away from AI, but they are making people think harder about where this technology is headed," said Ross Filipek, CISO at Corsica Technologies.

"At the same time, there's still enormous pressure to adopt AI."

Governance becomes the common ground

Even among leaders who viewed the extinction-risk debate as largely theoretical, few dismissed the underlying issues altogether.

Jason Frame, CIO and acting director of administration at the Southern Nevada Health District, compared the situation to how public health organisations manage uncertainty.

"These threats may sound alarmist, and they remain theoretical today," Frame said. "From a public health perspective, though, we manage uncertainty by putting guardrails in place and continually adjusting as we learn more."

Similarly, Yuri Kasan, CIO at InfoDefense and a cyber risk and ransomware readiness adviser for small and midsized businesses, argued that firms should neither ignore the warnings nor overreact to them.

"CIOs shouldn't ignore warnings like these, but they shouldn't panic either," Kasan said.

"Companies should start with limited, clearly defined uses, keep people involved in important decisions, and make sure they can monitor, restrict or shut down the system when necessary."

While enterprise leaders may disagree about the likelihood of future AI threats, they largely agree on what organisations should focus on now: establish governance, maintain human oversight, understand where AI is being used and ensure accountability is kept firmly in place.

The question, in other words, is not whether organisations will continue adopting AI; it's whether they can do so responsibly.

This article originally appeared on MES Computing