BEIJING, China — Chinese AI developer Moonshot is conducting an internal review after security researchers said they were able to bypass safety controls on two of its Kimi models and prompt them to discuss biological weapons and assassination.
Mindgard, a company that tests the security of artificial intelligence systems, said that it discovered the vulnerabilities in July while examining Kimi K2.6 and K3 Swarm.
The researchers used a technique known as jailbreaking, in which users employ carefully crafted prompts or sequences of instructions to persuade an AI system to bypass restrictions imposed by its developers.
Mindgard said the safeguards on the two models should have prevented them from engaging with certain dangerous requests.
Moonshot Reviewing Findings
Moonshot said it welcomed third-party research as an important part of improving AI safety and confirmed that it was discussing the findings with Mindgard.
Mindgard founder Peter Garraghan told the BBC’s Tech Life programme that the findings were concerning because, once the safety controls were bypassed, the models could respond to a broad range of prohibited topics.
He said the models could also generate recommendations beyond the original request, raising concerns about how a successful jailbreak could be exploited.
Mindgard has not established whether the information generated by the models on biological weapons or other harmful subjects would actually work in practice.
The company said, however, that the models should have refused to engage with such requests in the first place.
Researchers Warn Of Wider Risks
The findings highlight a different AI safety challenge from recent incidents involving autonomous AI agents.
AI agents developed by companies including OpenAI, Meta and Anthropic have recently been involved in attempts to interact with or compromise online services, raising concerns about the ability of AI systems to carry out harmful actions with limited human intervention.
Jailbreaking, by contrast, involves manipulating an AI model’s instructions or safeguards to make it provide information that would ordinarily be restricted.
Mindgard said a successfully jailbroken version of Kimi K2.6 could potentially provide access to computing resources and internet connectivity, creating additional cybersecurity risks.
The company did not publicly disclose the specific techniques it used to bypass the model’s safeguards.
Mindgard said it first notified Moonshot about the vulnerability on July 27 and followed up approximately a week later. It subsequently published details about the issue on September 12.
Open-Weight Models Under Scrutiny
The incident has also renewed debate over the security implications of open-weight AI models, which allow users to obtain model parameters and potentially operate the systems on their own computing infrastructure.
Kimi is an open-weight model, meaning developers and other users can potentially deploy it independently rather than accessing it exclusively through Moonshot’s systems.
Professor Alan Woodward of the University of Surrey said that open models could present risks if they fell into the hands of malicious actors, although the same technology could also be used for cybersecurity and defence.
He pointed to an earlier case in which an open-source Chinese AI model was used to help analyse a cyberattack that was later linked to AI agents developed by OpenAI.
AI Safety Race
The episode comes amid growing debate over how governments and technology companies should regulate increasingly capable AI systems.
Experts have warned that regulation could struggle to keep pace with rapid advances in artificial intelligence.
Woodward said greater attention should also be placed on identifying and prosecuting people who deliberately misuse AI rather than focusing solely on restricting the underlying technology.
