AI cyber confidence outpaces testing readiness, study finds

SimSpace has published research showing a gap between confidence in AI-driven cyber defences and measured security readiness. The findings suggest many organisations are deploying AI agents in security operations before testing them thoroughly.

The report, based on a survey of global chief information security officers and senior leaders alongside data from SimSpace environments, found that 78% of security leaders reported high confidence in their defences. Yet Defensive Security Readiness scores in exercises were as low as 30%.

AI use in security operations centres is already widespread. The research found that 73% of organisations are using AI agents in their SOC at a moderate to high level.

Testing appears to lag adoption. Only 29% of organisations said they conduct continuous simulation testing, while 44% test twice a year or less, or do not test at all.

This mismatch is becoming more visible as AI tools take on a larger role in day-to-day security work. The report also found that one-off tabletop exercises and certification courses remain among the most common forms of training for security teams.

Testing gap

SimSpace found that the early stages of AI deployment can reduce performance before improvement appears. Introducing AI tools into the SOC initially lowered performance by 10% to 20% until teams refined processes through repeated testing.

That pattern was reflected in readiness data. Each full simulation exercise improved Defensive Security Readiness scores by about three to five percentage points, with the largest gains occurring in the earlier rounds.

Teams running frequent, realistic simulations improved their Defensive Security Readiness scores by 20% to 50% per event and reached higher performance levels within four to six iterations. Teams that tested less often levelled off at materially lower scores.

Lee Rossey, Chief Technology Officer and Co-founder of SimSpace, said current deployment patterns are concentrated on less autonomous systems. "Assistive AI agents are mostly what's being deployed to production today; they're not fully autonomous agents," Rossey said.

He added that formal validation has not kept pace with deployment.

"It's noteworthy, though, that there's not rigorous testing of those agents prior to deployment; enterprise leaders seem to be counting on the humans in the loop to spot and correct any erratic behavior from their AI agents. Said another way, enterprise executives have not yet focused on and/or figured out how to develop trust in agentic AI before deploying it to production," Rossey said.

Operational pressure

The findings come as security teams face pressure to respond to a changing threat environment while incorporating more automation into monitoring and response tasks. In that setting, confidence in tooling is not the same as evidence of readiness under realistic attack conditions.

Instead, the report points to continuous simulation, repeated testing, and measurement of operational outcomes such as detection success, response accuracy, and decision quality as more reliable indicators. It recommends moving away from episodic validation towards ongoing assessment in environments that resemble production conditions.

It also urges organisations to account for an adjustment period when introducing AI into security operations. According to the findings, improvement depends on iteration rather than immediate gains.

Rossey said the next stage of adoption will increase pressure on organisations to prove these systems can operate safely and effectively.

"Autonomous agentic solutions are what's coming next, and enterprise executives are going to want to have complete trust in them to perform appropriately in a wide variety of situations before they get deployed to production," Rossey said.

He said organisations would need places to test both AI systems and human teams under realistic conditions.

"They're going to want to develop that trust through rigorous testing in a production-like environment where it's ok to fail. Eventually, they are going to want to trust those AI systems to act on their behalf in production without depending on human operators to monitor their every action closely. What's interesting in this research is that it shows that they have not yet begun rigorous testing to develop trust in those AI agents as a natural part of their deployment process," Rossey said.