AI Guardrails Impede Cybersecurity Research
AI giants like Anthropic and OpenAI have for months implemented stringent guardrails and vetted access programs for their advanced models, measures initially aimed at preventing misuse by malicious actors. However, these restrictions are now actively impeding the crucial work of legitimate network defenders and offensive cybersecurity researchers globally, challenging the efficacy of current AI infrastructure in compliance security, according to reports from cybersecurity professionals to TechCrunch.
The issue gained prominence after the U.S. government, in June, imposed export control restrictions on Anthropic's highly anticipated AI models, Mythos and Fable. This move was prompted, at least in part, by a report suggesting that the models' guardrails—designed to prevent their use in building and executing malicious cyberattacks—could be bypassed. While the motivations behind the export controls are debated, Anthropic had previously marketed Mythos as a formidable "doomsday cybermachine," intended only for carefully vetted users under strict controls. Although the export controls on Fable 5 and Mythos 5 have since been lifted, with Fable 5 returning to general access on July 1, Mythos 5 has been reintroduced only to vetted U.S. organizations as part of a government review process.
The Double-Edged Sword of AI Access
By Decode Today News
This approach of gatekeeping advanced AI isn't exclusive to Mythos. Both Anthropic, with its broader model portfolio, and OpenAI offer dedicated programs for cybersecurity researchers to gain vetted access to models with fewer cybersecurity restrictions. OpenAI's initiative is known as Trusted Access for Cyber, while Anthropic operates the Cyber Verification Program. Despite these specialized access points, the guardrails themselves have drawn significant criticism, particularly from researchers whose primary objective is to discover unknown system vulnerabilities and develop exploits for them before malicious actors can.
Mark Dowd, a renowned security researcher, voiced his discomfort during a recent cybersecurity podcast. "It's not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what's not," Dowd stated. With decades of experience finding and selling "zero days"—previously unknown software flaws and their corresponding exploits—to Western governments, Dowd's work relies on the very vulnerabilities that AI guardrails aim to obscure. He acknowledged a potential bias given his profession, where governments pay a premium for open vulnerabilities for intelligence operations, but emphasized that his concerns are widely shared within the offensive cybersecurity community.
Understanding AI Guardrails in Cybersecurity
AI guardrails are essentially a set of programmed limitations or rules embedded within artificial intelligence models. Their primary purpose in cybersecurity contexts is to prevent the AI from generating or assisting with malicious content, such as exploit code, phishing emails, or instructions for cyberattacks. These safeguards are designed to ensure responsible AI usage and mitigate potential cybersecurity risk. However, the complexity arises because many legitimate defensive activities, like vulnerability assessment and penetration testing, mirror the actions of an attacker. This dual-use nature of AI tools means that overly restrictive guardrails can inadvertently block critical security research and enterprise integration efforts aimed at proactive defense.
Practical Impediments for Offensive Security Researchers
Several offensive cybersecurity professionals elaborated to TechCrunch on how these AI tools' guardrails impact their daily work. Chris Anley, the chief scientist at security consulting giant NCC Group, highlighted a critical step in vulnerability research: asking an AI model to attempt to exploit a bug to confirm its reality and severity. "If a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders," Anley explained. He further clarified the inherent challenge: "This is where the whole offensive versus defensive and guardrails part comes in, because 'fix this code' as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base." Anley likened AI tools to a hammer: "You can't build a house without a hammer. It's definitely a tool but it's also irreducibly a weapon as well." When faced with such roadblocks, Anley and his team sometimes resort to open-source AI models that lack any guardrails.
Paolo Stagno, CTO at CrowdFense, a company specializing in developing and selling unknown vulnerabilities to government agencies, echoed Dowd's sentiments. He argued that AI companies "essentially treat customers like children who need babysitting" through their vetted programs and guardrails. Stagno noted that while his team uses frontier models for reverse engineering, they deliberately avoid using AI to find vulnerabilities or build exploits directly. This is due to the significant risk of leaking sensitive vulnerability data or having it absorbed into future training runs if fed into cloud-based models. For these sensitive steps, CrowdFense relies on open-source models run locally, ensuring data remains within their control and does not leave their internal AI infrastructure.
Not all researchers find the guardrails equally impeding. Giuseppe Cali, a security researcher focused on zero-days and exploit development, stated that his work remains unaffected because he uses AI primarily for initial reverse engineering, understanding code, and building supporting tools. He emphasized that AI tools accelerate these preparatory phases, allowing him to concentrate on the actual vulnerability discovery. "I still want to own the actual bug discovery and weaponization myself and that wouldn't change if all guardrails were lifted tomorrow," Cali affirmed, adding, "I am jealous of my bugs, and I like this game too much to let models play it for me."
Inconsistency and the Push Towards Foreign AI
The experience of an anonymous researcher at a smartphone-component manufacturer further underscores the impact of strict guardrails. This individual's employer is not part of Anthropic's CVP program, rendering their AI tools nearly useless for vulnerability research due to overly stringent limitations. "If it catches wind we're doing anything security related, it just stops and isn't usable," the researcher disclosed.
Chris Thompson, CEO of cybersecurity firm RemoteThreat and founder of Offensive AI Con, highlighted another critical issue: the inconsistency of guardrails. In his experience with frontier AI models, even within the looser boundaries of vetted programs from Anthropic and OpenAI, guardrails can behave differently day-to-day. "I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," Thompson explained. He added that researchers often find themselves "trying to find why you're getting inconsistent results or why are models over-sanitizing the output," rather than focusing on vulnerability analysis and exploitability reasoning.
This inconsistency, coupled with strict limitations, has an alarming consequence: it pushes responsible researchers toward Chinese open-source models, such as GLM. These models are freely downloadable, can be run locally, and come with no vetting or usage restrictions. "You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems," Thompson warned. He concluded that "it's more harmful than good to have these guardrails in place," asserting that such restrictions prevent network defenders from gaining crucial advantages in the rapidly evolving landscape of cybersecurity risk.
The Looming AI Cybersecurity Race
Rather than tightening restrictions further, Thompson advocates for AI frontier labs to open up their programs, provide responsible access, and hold accountable those who abuse their tools. He stressed the urgency, predicting a "big storm coming" with a "big wave of attacks that are going to happen at speed and scale like never before." In this impending scenario, he argues, legitimate security consulting firms and researchers, who are trying to make a difference in enhancing compliance security, are currently being stifled. The broader implication is that stifling defensive innovation through AI guardrails could leave global enterprises vulnerable as offensive capabilities rapidly advance.
Key Insights from Cybersecurity Researchers on AI Guardrails
- Mark Dowd: Critical of AI companies making "arbitrary decisions" about security safety.
- Chris Anley (NCC Group): Guardrails hinder essential steps in confirming real vulnerabilities; AI is a "hammer" – both tool and weapon.
- Paolo Stagno (CrowdFense): AI companies "treat customers like children"; uses frontier models for reverse engineering but open-source locally for sensitive exploit work to avoid data leaks.
- Giuseppe Cali: Not impeded, uses AI for initial reverse engineering and supporting tools, prefers to "own" bug discovery and weaponization.
- Anonymous Smartphone Researcher: Strict guardrails make AI tools "barely useful" for vulnerability finding without vetted access.
- Chris Thompson (RemoteThreat): Guardrails are inconsistent, forcing researchers to "negotiate with the model" and pushing them towards foreign-owned open-source AI.
