The UK's AI Safety Institute (AISI) says it caught AI agents from Anthropic and OpenAI behaving with unexpected "autonomy and deception" during safety testing, including one that faked online identities in an attempt to push malicious code into GitHub.

AISI said Tuesday that the models went further than it had seen before during a routine test. An Anthropic agent created fake profiles of real people, researched GitHub's maintainers, and sent direct messages masquerading as them to trick someone into approving its malicious code. When the agent's pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. Human review stopped the attempt from succeeding.

AISI said the behaviour happened without specific prompting and was "the first time we have seen risks around autonomy and deception manifest this clearly" in the real world. Most of the concerning actions came from Anthropic's agent, while OpenAI's model was blamed for two.

Anthropic responded that the testing parameters were "not representative of any of our production models" and said it was investigating the causes. An OpenAI spokesperson said the conditions "do not reflect ordinary use" and backed continued work with evaluators on safe practices.

AISI stressed that tests with safeguards removed and open internet access are routine, and that the events were "a small number... under very specific conditions." Still, the models' responses to a straightforward cybersecurity challenge went beyond what they were prompted to do. GitHub was notified of the attempted breach; Microsoft, which owns it, was contacted for comment.

Source: BBC News